SearchOS – A Multi-Agent Search Collaboration Framework Open-Sourced by Ant Group and Others

1. What is SearchOS

SearchOS is a multi-agent search collaboration framework jointly open-sourced by the H瓴 Artificial Intelligence School at Renmin University of China and Ant Group. Drawing inspiration from the design principles of relational databases, this framework abstracts open-domain information retrieval into dynamic relational patterns. It introduces system-level mechanisms such as search-oriented context management (SOCM), pipeline parallel scheduling, and a search tool middleware layer, addressing core challenges in long-range search tasks from an architectural perspective—such as agent memory loss, redundant queries, and disconnection from sources. This enables multi-agent systems to share states efficiently, collaborate effectively, and trace evidence across the entire workflow. The framework aims to provide an industrial-grade multi-agent collaboration infrastructure for complex information collection tasks that require high recall and structured output.

Image source: Official article
Image source: official article

Technical Positioning and Domain: SearchOS belongs to the intersection of multi-agent systems and information retrieval, positioning itself as a search agent operating system tailored for complex information collection tasks. It differs from traditional single-agent search tools or general-purpose agent frameworks based on conversational context. By externalizing search states to the system level and introducing relational abstraction, the information retrieval process becomes structured, traceable, and easily scalable in parallel. This framework is particularly suitable for scenarios requiring systematic enumeration and cross-verification, such as competitive research, academic reviews, and due diligence.

Development Background: Jointly developed by the H瓴 Artificial Intelligence School at Renmin University of China and Ant Group, the team has extensive experience in natural language processing, information retrieval, and multi-agent collaboration. The motivation for development stemmed from systemic issues commonly found in existing agent frameworks during long-range search tasks: agents tend to lose early context after prolonged operation, resulting in redundant queries and omissions; the association between search results and their sources is fragile, making verification and tracing difficult; and there is a lack of effective state-sharing mechanisms among multiple agents. SearchOS was specifically designed to fill this gap.

Core Value: SearchOS fundamentally resolves the problem of information omission and redundant work caused by agent "forgetfulness" in long-range search tasks. By migrating search states from conversational context to system infrastructure, it achieves real-time state synchronization and progress visualization across agents, enabling complex information collection tasks to be completed efficiently and orderly. Additionally, each conclusion is accompanied by traceable original text references, meeting the rigorous requirement of "trying to find everything, with each item having a source," significantly enhancing the credibility and efficiency of information collection.

Technical Features: Key technical features include: an information organization method based on dynamic relational patterns, where table structures evolve in real time as entities are discovered, and experiments have shown that its performance surpasses the best fixed structure; a pipeline parallel scheduling mechanism that reduces task completion time by approximately 24% and LLM invocation counts by about 13% through staggered advancement and continuous dispatch; a three-tier search tool middleware (context, sensor, evidence extraction) that enables intelligent intervention to prevent loops and stagnation; and a preloaded set of approximately 280 hierarchical skills covering search strategies and website access techniques, reducing search time by 36.6% after skill usage.

2. Key Features

• Relation Search Pattern Construction: The system automatically parses user natural language queries into a dynamic relational schema that includes table structures, primary keys, and foreign keys. During the search process, as new entities are discovered, the table structures and attributes in the schema are expanded and completed in real time, ensuring that the information organization method always aligns with the current task requirements. This dynamic evolution mechanism avoids the lack of flexibility inherent in fixed schemas when dealing with diverse queries. Experimental results show that even when the optimal fixed structure (Oracle) is pre-specified, its performance is still lower than that of SearchOS's dynamic schema.

• Multi-Agent Collaborative Search: Utilizes an Orchestrator for unified planning and scheduling. The Explore Agent is responsible for discovering candidate entities and expanding the search boundaries, while the Search Agent performs parallel subtasks to collect information about specific entities. The Writer Agent finally generates a structured report. Each Agent collaborates by sharing state information, and the Orchestrator dynamically adjusts task allocation and priorities based on progress, enabling efficient team collaboration and avoiding task conflicts and resource waste.

• Search-Oriented Context Management (SOCM): Maintains four types of shared states: Frontier Tasks, Evidence Graph, Coverage Map, and Failure Memory. Frontier Tasks record subtasks awaiting exploration, Evidence Graph stores collected evidence and its associations, Coverage Map visualizes the degree of information coverage, and Failure Memory records failed attempts to avoid repetition. This state externalization mechanism fundamentally eliminates context loss and redundant queries in long-range tasks. All Agents retrieve global information by querying these shared states.

• Search Tool Middleware Layer: Inserts a three-tier middleware layer between the model and search tools. The Context Middleware controls the scale of context injected into the model, preventing inference performance degradation caused by excessively long contexts; the Sensor Middleware detects search loops or stagnation and triggers strategy switches (such as changing search terms or adjusting skills); the Evidence Extraction Middleware performs structured extraction, citation anchoring, and data storage of evidence, ensuring that each data point has an accurate source and supporting the ability to trace back from conclusions to specific locations in the original web content.

• Hierarchical Skill Library: Pre-installs approximately 280 skills, categorized into two major types: strategy skills and access skills. Strategy skills include ranking retrieval, multi-hop search, entity disambiguation, and synonym expansion; access skills include handling anti-crawling mechanisms, login wall verification, dynamic page rendering, and JavaScript execution. Agents can automatically select appropriate skill combinations based on task type and current progress. Experimental results show that using these skills reduces search time by 36.6% and decreases API call counts by 39%-43%.

3. How to Use

1. Environment Preparation and Installation: First, clone the project code from the GitHub repository: git clone https://github.com/antins-labs/SearchOS.git. After entering the project directory, install the Python dependencies according to requirements.txt: pip install -r requirements.txt. It is recommended to use Python 3.10 or higher and install within a virtual environment. Then run the configuration wizard (e.g., python setup.py --configure) and select the model service provider you are using (such as OpenAI, Anthropic, or compatible interfaces) one by one, entering the corresponding API Key. After configuration is complete, the system will automatically verify the connection validity.

2. Launching the System Interface: SearchOS provides three interactive interfaces. Execute python searchos.py --cli to launch the lightweight command-line mode, which is suitable for remote servers or environments without a graphical interface; python searchos.py --tui to launch the full-screen terminal interface, offering richer real-time information display; python searchos.py --web to launch the Web research workstation, which listens on the local port 8000 by default. Access it via a browser to obtain a complete visual operation panel. Users can choose based on their use case and preference; specific commands should be referenced in the project documentation.

3. Inputting Query Tasks: Enter a natural language question in the search interface, for example, "List the top 10 global AI companies in terms of 2024 funding, along with their valuations, main business areas, and countries of origin." The system will automatically parse the task type and initialize the relational schema. Users can also predefine part of the table structure as an initial prompt, such as defining the "Company" table to include columns like "Name, Funding Amount, Business," to guide the search direction. Batch input of multiple related queries is supported, and the system will automatically merge the tasks.

4. Real-time Monitoring of Search Progress: After the task is initiated, users can view the completion progress of the Schema, the coverage percentage of the Coverage Map, the current tasks being executed by each Agent, and the number of collected evidence entries through the interface. The Web workstation provides a dynamically updated dashboard, displaying the length of the Frontier Tasks queue and statistics on failed records. If a particular direction is progressing slowly, users can manually adjust the search strategy or add additional prompts; the system will respond immediately and re-plan the tasks.

5. Reviewing and Exporting Results: After the task is completed, the system outputs a structured report, with each data entry accompanied by a citation. Users can click to trace back to the specific location on the original webpage (supporting anchor points at the paragraph level). Results can be exported in CSV, JSON, or Markdown formats, facilitating further analysis or integration with other tools. Users can also edit and annotate results directly within the Web workstation.

6. Interrupting and Resuming: Long-running tasks can be interrupted at any time (e.g., by pressing Ctrl+C), and the system will automatically save the current search state to a local file (default saved in the ./searchos_state/ directory). Next time you start, use the --resume parameter and specify the path to the state file to continue the previous task. All progress, evidence chains, and Agent states are fully preserved, eliminating the need to start over.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

In complex information collection scenarios requiring high recall and structured output (such as competitive research, academic reviews, and due diligence), SearchOS significantly enhances search efficiency and quality through its dynamic relationship model, system-level state management, and rich pre-configured skill set, making it the most professional choice available today. Web2BigTable has a basic foundation for table-based information collection, but its flexibility and performance metrics lag behind those of SearchOS.

For more general multi-agent collaboration tasks, such as conversational question answering, code generation, and content creation, AutoGen and CrewAI offer more flexible frameworks. Developers can customize agent roles, tools, and dialogue processes according to their needs. However, to achieve the same level of performance as SearchOS in search tasks, a significant amount of custom development is required, including implementing state management, scheduling optimization, and building a skill library. Therefore, the choice should be based on the task type: for search-intensive tasks, prioritize SearchOS; for general collaboration tasks, choose AutoGen or CrewAI; and for quickly building a table-based information collection application, Web2BigTable can serve as a lightweight alternative.

6. Editor's Summary

The core innovation of SearchOS lies in externalizing the search state from the Agent context to the system infrastructure. This design fundamentally addresses the issues of information loss and redundant queries in long-horizon Agent tasks, providing a reference architecture model for multi-Agent collaboration systems. The combination of dynamic relationship patterns with pipeline parallel scheduling enables large-scale information collection tasks to be executed efficiently and orderly. Experimental data (WideSearch Item F1 80.3, GISA Set F1 76.5) confirms its significant advantages in recall and precision. Compared to general-purpose multi-Agent frameworks, SearchOS focuses more specifically on the search domain, offering a more specialized skill library and a three-tier middleware mechanism, which reduces the customization cost for developers working on search tasks.

In terms of practical value, SearchOS is particularly suitable for scenarios requiring systematic collection of structured information, such as market research, academic literature reviews, and investment research due diligence. Its full-chain evidence tracing functionality is especially beneficial for users with strict requirements on the source of information, as every conclusion can be traced back to the exact location on the original webpage, thereby enhancing the credibility of results. The framework is currently open-source, allowing the community to build upon it with secondary development and functional extensions, thus promoting the advancement of search agent technology.

Target users include: researchers, analysts, and product managers who need to perform large-scale information collection; developers aiming to build search agents; and technical professionals interested in multi-Agent system design. At present, the framework is primarily optimized for English environments, and support for Chinese needs improvement, which is a key direction for its future development. From a technical perspective, the design of state externalization and dynamic patterns is highly scalable, laying the groundwork for future feature enhancements such as multi-modal search and real-time streaming integration.

7. Application Scenarios

Competitor and Market Research: Systematically collect information on competitors' funding rounds, product features, pricing strategies, market share, and more, generating a large-scale comparative table with sources cited. Users simply need to input the target market and competitor names, and SearchOS will automatically identify relevant entities and complete their attributes, ultimately producing a structured report where each data point can be traced back to its original source. The report supports export to CSV or Markdown formats, making it directly usable for report writing.

Academic Literature Review: For specific research directions, enumerate relevant papers, authors, institutions, and conferences, and complete key conclusions and method summaries. SearchOS's dynamic relationship model can flexibly adapt to the information structure of different academic fields, enabling multi-hop searches to uncover citation networks. The Coverage Map ensures high coverage, making it ideal for early-stage information collection when writing systematic literature reviews.

Investment Research and Due Diligence: Track the shareholder structure, executive changes, litigation records, and financial data of target companies, achieving cross-verification from multiple sources. The system uses multi-level searches and evidence tracing to ensure the accuracy and comprehensiveness of information. The failure memory mechanism avoids revisiting invalid sources, making it suitable for financial scenarios with high demands on timeliness and reliability.

Product Comparison and Selection: Collect technical specifications, user reviews, and price ranges of similar products to generate structured comparison reports that aid in decision-making. SearchOS's skill library includes anti-scraping and dynamic page handling capabilities, allowing it to extract information from e-commerce platforms, review sites, and other sources, while automatically aligning data formats across different sources.

Open-Domain Knowledge Enumeration: For tasks requiring high recall, such as "List all Fields Medal winners since 2000 along with their affiliated institutions and countries." SearchOS's pipeline parallel scheduling and Coverage Map mechanisms can efficiently enumerate and verify results. Real-time progress tracking provides users with a clear understanding of the coverage level, preventing omissions.

8. FAQ

Q: How does SearchOS address Agent memory loss in long-running tasks?
A: SearchOS resolves this by externalizing search state into the system infrastructure through its Search-Oriented Context Management (SOCM). It maintains four types of shared states: Frontier Tasks, Evidence Graph, Coverage Map, and Failure Memory. All Agents retrieve task progress and historical information by querying these shared states, rather than relying on their own conversation context, thereby avoiding memory loss and redundant queries caused by context loss. This design ensures that the state remains intact even if an Agent is restarted or switched.

Q: Which large models does SearchOS support as the underlying inference engine?
A: SearchOS is designed to be model-agnostic and supports integration with various large model APIs through configuration. This includes OpenAI's GPT series (GPT-4o, GPT-4), Anthropic's Claude series, and other models via compatible interfaces (e.g., locally deployed LLaMA). Users need to set the corresponding API Key and model name in the configuration wizard, and the system will automatically adapt to the interface format.

Q: How does the dynamic relationship schema in SearchOS work?
A: After receiving a user query, the system initializes a relationship schema containing table structures, primary keys, and foreign keys. During the search process, the Explore Agent discovers new entities, and the schema automatically expands by adding new tables or attributes. For example, when searching for "AI companies," the initial schema may only contain a "Company" table. As funding information is discovered, it automatically adds a "Funding Round" table and establishes foreign key relationships. This dynamic evolution ensures that the information organization remains aligned with task requirements, offering greater flexibility and accuracy than a pre-defined fixed schema.

Q: What is the specific role of the search tool middleware in SearchOS?
A: The search tool middleware consists of three components: Context Middleware controls the context length injected into the model, preventing excessively long contexts from affecting inference performance and automatically trimming redundant information; Sensor Middleware detects search loops or stagnation (e.g., repeatedly returning the same results) and triggers strategy switches (e.g., changing search terms or adjusting skills); Evidence Extraction Middleware performs structured evidence extraction, citation anchoring, and data storage, ensuring that each piece of data has an accurate source and supports tracing back from conclusions to specific paragraphs in the original web content.

Q: What types of skills are included in SearchOS's pre-built skill library?
A: The skill library contains approximately 280 skills, divided into strategy skills and access skills. Strategy skills include ranking retrieval (e.g., Google Search API), multi-hop search (discovering new sources through citation chains), entity disambiguation (distinguishing between entities with the same name), and synonym expansion; access skills include handling anti-crawling measures (e.g., changing User-Agent, using proxies), bypassing login walls, dynamic page rendering (using Headless browsers), and JavaScript execution. Agents automatically select the optimal skill combination based on task type and current progress.

Q: What hardware requirements does SearchOS deployment have?
A: SearchOS itself is a lightweight framework, with the main resource consumption coming from the inference of the underlying large model. If using cloud APIs, a standard local server (4-core CPU, 8GB memory) is sufficient to run the scheduler and middleware. If using a local model, the required GPU should be configured according to the model size (e.g., at least 16GB VRAM is needed to run a 7B model). It is recommended to use a Linux operating system for the best compatibility.

9. Project Links

• Project Website: https://antins-labs.github.io/SearchOS/
• GitHub Repository: https://github.com/antins-labs/SearchOS
• arXiv Technical Paper: https://arxiv.org/pdf/2607.15257