OpenDS Benchmark Reveals Top Models Hit Only 39% Accuracy in Real-Web Search

OpenDS: A New Deep Search Task Paradigm and OpenDS-World, a Benchmark for Real-World Open Web Environments

In recent years, as the reasoning and tool-use capabilities of Large Language Models (LLMs) have improved, Deep Search Agents have developed a certain degree of autonomous search and information integration capabilities. An increasing number of studies are attempting to enable Agents to autonomously plan search paths and handle complex problems that require traversing multiple information sources.

However, when Deep Search truly enters the real internet, the research process becomes significantly more complex.

Information in the real web is scattered across different websites and resources. It may be hidden in a specific frame of a video, a segment of dialogue in a podcast, charts within academic papers, or data files that require downloading to read. Even after locating relevant websites, Agents may need to continue interacting with dynamic pages and adjusting query parameters to retrieve the specific information required.

Existing Deep Search research has made considerable progress in areas such as multi-turn retrieval, cross-source reasoning, tool calling, and web interaction. Related benchmarks have evaluated these capabilities from various angles. However, facing the real-world open web environment where multimodal information, heterogeneous resource formats, and interactive access requirements intersect, the end-to-end research capabilities of current Agents still lack sufficient evaluation.

This raises a critical question: When the location and acquisition method of evidence are not predefined, can an AI Agent independently determine what to look up next, where to look, how to access and process it, adjust its search direction based on existing information, link evidence from different sources, and ultimately resolve a complex problem?

Addressing this issue, a research team from institutions including The Hong Kong University of Science and Technology (Guangzhou), Fudan University, University of Washington, Imperial College London, and Stanford University proposed a new deep search task paradigm—OpenDS (Open-Web Deep Search)—in their paper Deep Search in Real-World Web Environments. They also constructed OpenDS-World, a benchmark tailored for real-world open web environments.

• Paper: Deep Search in Real-World Web Environments
• Paper Link: https://arxiv.org/pdf/2610.30181v1
• GitHub Link: https://github.com/Yu-xm/OpenDS
• Hugging Face Link: https://huggingface.co/datasets/Yu2020/OpenDS-World

The team constructed 200 tasks covering 4 modalities, 42 resource formats, and 7 types of evidence access requirements. Twelve frontier models—including GPT-6 Astra, Claude Opus 5.5, Gemini 3.8 Flash, Qwen3.8 Max, GLM-5.3 Flash, DeepSeek V4.1-Flash, and Kimi K3—were provided with the same set of search, browser, and video-related tools. Each task was limited to 1 hour of search time and 100 tool calls, requiring the models to autonomously decide the search process, link evidence, and provide final answers.

After completing the 200 tasks, even the best-performing model, GPT-6 Astra, achieved an accuracy of only 39%. Claude Opus 5.5 and Gemini 3.8 Flash scored merely 38.0% and 36.5%, respectively, while other models had accuracies below 35%. Notably, although the average number of source entries manually annotated and reviewed per task was only 3.94, a single search attempt by the models involved dozens, or nearly one hundred, tool calls. Failures in the search process were also highly time-consuming: in GPT-6 Astra's execution logs, failed calls accounted for only 6.34% but consumed 42.57% of the total tool execution time. In the real web, models must invest significant time locating information sources, trying different access methods, and handling failures encountered during the search process.

Figure 1: Performance of 12 models on OpenDS-World. Human scores are derived from independent solving experiments.

What Do We Expect Models to Achieve in Deep Search?

Anyone familiar with research knows this process well: initially, there is only a vague object of interest. After finding an introductory article, one realizes which report needs to be located next; a chart in that report then directs the inquiry toward another dataset. As the search goes deeper, the next steps become clearer.

Deep Search aims to enable agents to autonomously perform such searches. OpenDS focuses on whether, when this process occurs in the real open web environment, an agent can autonomously discover and access information scattered across diverse sources, handle various resource formats and interaction requirements, and use newly acquired clues to determine subsequent search directions.

This autonomous decision-making is concretely reflected in actual information access processes. Even after finding relevant websites, agents often need to perform multiple steps of operations to obtain target information. The paper uses a page as an example: the same page contains text, maps, and charts, while also providing data resources in CSV, JSON, and ZIP formats. To query specific values for a region in a particular year, an agent may need to select the region, adjust the time, read the chart, or retrieve information via data interfaces and file downloads.

Figure 2: A single website can simultaneously contain text and charts, multiple data formats, as well as operations such as parameter selection, time positioning, and data queries.

Different information sources have their own access and processing requirements. Clues in videos require locating specific segments; evidence in papers may be hidden within charts; and data files require finding target records and fields. Even when storing text data, the reading methods for HTML, PDF, and Parquet differ significantly.

The paper categorizes these differences into three dimensions: Diverse Modalities, Heterogeneous Formats, and Interactive Access Requirements. Modality describes how evidence is presented (text, image, audio, or video); format describes the file and data structure in which it is stored; and access requirements describe the operations needed to retrieve it.

These three dimensions combine in different forms during a single search. Objects identified by the model from a video can become keywords for retrieving papers; new clues found in papers may require querying another website. Each step must utilize the evidence already obtained to inform the next action.

OpenDS defines this process as a factual question-answering task with a unique, verifiable answer. After receiving the question and initial clues, the model needs to independently discover, access, process, and link evidence at each step, finally providing the answer. This approach preserves the complexity of the real open web during the search process while maintaining clear verification standards.

How to Turn Real Web Searches into Questions?

To construct questions from the real open web that have reliable factual bases and truly require models to explore step-by-step to solve, two problems must first be addressed: every node in the evidence chain and its associations must have real, verifiable basis; and the final question must not directly expose intermediate entities or answers, preventing the model from reaching the endpoint solely based on the question's clues.

OpenDS-World adopts a "build the evidence chain first, then reverse-generate the question" approach, dividing task construction into three stages:

First, determining the search starting point (Real-World Anchor). The research team selected 500 candidate videos from eight domains: culture, daily life, science, economy, history, nature, society, and technology. Annotators identified initial entities with clear references by combining continuous video frames and audio content, ensuring they uniquely corresponded to valid Wikidata entries. There was a special constraint here: the initial entity had to be determined relying on audiovisual information in the video and could not be extracted directly from text on the video page. This method ensures that the model must truly understand the video content to obtain the starting clues for subsequent tasks. The starting videos used for tasks are archived and provided as benchmark inputs.

After confirming the starting point, the process enters the Open-Web Evidence Expansion stage. Annotators construct queries based on the current entity, search for candidate information sources on the real web via Google Search, check their content one by one, and then select sources with clear semantic associations that are verifiable. New entities extracted from these sources serve as nodes for the next search step. The entire expansion process particularly emphasizes reliable associations between pieces of evidence. Although each new node comes from a specific source, its validity requires cross-validation through multiple independent sources. Relationships between adjacent nodes must also have clear factual basis. Meanwhile, annotators consider the diversity of information modalities, resource formats, and access methods to ensure the evidence path covers different information acquisition needs in the real web. This process repeats until a unique, stable entity suitable as the final answer is found.

Finally, the Question Inversion stage occurs, where the question is written only at this point. The research team sets the endpoint of the evidence chain as the reference answer, then converts the determined intermediate entities into a set of interdependent indirect descriptions and constraints.

For example, the question will not directly give the name of a museum exhibit but instead requires first identifying the organism based on the video, then finding the corresponding collection, and identifying the target fossil from a specified display case. Findings from the previous step provide necessary clues for the next, and various constraints gradually narrow down the candidate scope, ultimately jointly determining the unique answer.

Figure 3: An example from OpenDS-World. This question links audiovisual recognition, virtual tour navigation, document inspection, and structured data retrieval across multiple web sources. Starting from the initial video, it passes through multiple sources including museum virtual tours, podcasts, academic papers, and structured datasets, finally determining the unique ID of a UNESCO Geopark.

This design prevents the model from retrieving the final answer based solely on any directly given entity name in the question. Instead, it requires step-by-step determination of correspondence relationships between intermediate objects and conducting searches in the real web.

After completing question construction, the research team organized 10 independent reviewers to conduct two aspects of validation on candidate tasks:

• Evidence Validity: Check whether the initial entity, intermediate nodes, relationships between nodes, and the final answer have real, verifiable basis, and cross-validate through multiple independent web sources.
• Task Validity: Check whether the question formulation is clear, whether the constraints are accurate and consistent, and whether these constraints can jointly determine a unique factual answer.

Candidate tasks containing factual errors, insufficient evidence, conflicting constraints, or ambiguous answers are filtered out, resulting in a final set of 200 tasks with clear reference answers and verifiable evidence paths.

Looking at individual tasks, excluding the separately provided starting videos, 82.5% involve at least two information modalities, 96.0% involve at least two resource formats, and 95.5% involve at least two types of access operations. Specifically, 67.0% of tasks require page or content navigation, 60.0% involve resource transfer, 47.5% require temporal localization, and 42.0% involve parameterized state updates. Within a single research task, models often need to sequentially process different types of information and access operations.

Figure 4: Distribution of information modalities, resource formats, and access requirements in OpenDS-World. Different categories can overlap within the same task; statistics exclude the separately provided starting videos.

How Long Did It Take Humans? And How Many Did Models Get Right?

To understand the actual difficulty of finding answers from scratch, the team organized 50 participants for independent solving. Each participant was assigned 20 questions, with each question attempted by five different people, resulting in a total of 1,000 attempts.

Participants were only given the original videos and questions. They could not use LLM tools, were restricted to standard browsers and search engines, and were prohibited from communicating with each other. There was no uniform time limit per question for human testing; participants could end their attempt when they believed they could no longer make effective progress.

These 1,000 attempts consumed approximately 646 hours in total, yet only 211 attempts yielded correct answers. Out of the 200 questions, only 115 were solved by at least one person, while 85 remained unsolved across all five independent attempts; only 29 questions were answered correctly by at least three people.

Figure 5: Success rates, time spent, and subjective difficulty of human independent solving. The middle chart aggregates the total time each participant took to complete their assigned 20 questions.

Based on this data, the research team categorized task difficulty: the 54 questions solved by at least two people were labeled "Low," the 61 questions solved by exactly one person were labeled "Medium," and the 85 questions unsolved by all five participants were labeled "High." This stratification was then used to evaluate model performance.

For model evaluation, a unified Harness was employed: 16 tools provided capabilities for Search, GUI-based browser interaction, and video/subtitle reading (Other). These tools supported web clicking, input, scrolling, screenshots, content downloading, and video segment inspection. All 12 models used identical configurations, selecting their own information sources and operation sequences.

Unlike human testing, which had no time limits, models were allowed a maximum of 3,600 seconds for searching and 100 tool calls. Searching ceased once either budget was exhausted. If a complete answer had not been formed, the system entered a wrap-up phase with tools disabled, requiring the model to answer based on existing evidence. After completion, Qwen3.8-27B was used to compare against reference answers, checking for semantic equivalence and completeness. The evaluation framework also recorded task duration and token consumption.

Figure 6: Overall scores of the 12 evaluated models and their performance across different difficulty levels. Numbers in parentheses indicate the count of correctly answered questions; "T" denotes text-only models.

Overall, there remains a significant gap in performance among the 12 models on OpenDS-World. GPT-6 Astra ranked first with an accuracy of 39.0%, followed closely by Claude Opus 5.5 and Gemini 3.8 Flash with accuracies of 38.0% and 36.5%, respectively. The remaining models' accuracies ranged between 12.0% and 33.5%.

As task difficulty increased, overall model performance declined noticeably. The average accuracies for the 12 models across the Low, Medium, and High difficulty groups were 39.66%, 23.50%, and 16.76%, respectively. Relative strengths varied by model: GPT-6 Astra led in the low-difficulty group and tied for first place with Claude Opus 5.5 in the medium-difficulty group. However, in the high-difficulty group—where none of the five human participants succeeded—Gemini 3.8 Flash achieved the highest accuracy at 28.24%.

Short Reference Paths, Long Actual Searches

With the scores established, the next critical question is: What specifically makes these tasks difficult?

As mentioned earlier, the average number of manually annotated and reviewed source entries per question is only 3.94. However, a single search attempt by a model involves dozens, or even nearly a hundred, tool calls. Even when designed with just three annotated sources, the median number of tool calls reached 74.

In retrospect, OpenDS-World tasks exhibit distinct characteristics of Long-Horizon Decision-Making. Models must identify clues in the initial video and gradually locate and correlate evidence scattered across the real web through iterative searching, cross-page navigation, interactive access, and multimodal information processing. A single source may require multiple steps: entering the website, locating the entry point, setting parameters, and reading results—each consuming tool calls. Along the way, irrelevant candidate sources, failed operations, and clues requiring verification appear. Consequently, even if only a few key sources are ultimately involved, discovering and retrieving this evidence can demand extensive exploration and interaction.

The research team's analysis reveals that execution length does not increase monotonically with the size of the evidence chain. For tasks containing 3, 4, 5, and at least 6 annotated sources, the median number of tool calls were 74, 56, 72, and 82.5, respectively. A longer reference evidence chain does not necessarily imply that the actual search requires more operations.

Figure 7: Distribution of tool calls across different evidence chain sizes, along with the average execution length for each model. The dashed line indicates an average of 3.94 annotated source records per question.

While the length of the evidence chain describes which materials support the answer, the search trajectory records the exploration process through which the Agent discovers, accesses, and correlates this information. By combining both perspectives, OpenDS-World uncovers execution complexities that cannot be adequately measured by relying solely on evidence chain length or reasoning hops. This highlights the importance of long-horizon planning, context management, and adaptive tool usage in real-world web research.

Where Are Time and Tools Spent?

Different models exhibit distinct search investments and tool usage strategies within the same environment.

GPT-6 Astra averages 51.05 tool calls per task, taking 18.66 minutes with an accuracy of 39.0%. GPT-5.6 Sol has a similar call count but averages only 8.99 minutes, achieving 28.0% accuracy. Gemini 3.8 Flash consumes approximately 5.22 times the total tokens of GPT-6 Astra, with an accuracy of 36.5%. More tool calls, longer execution times, and higher token consumption do not necessarily lead to better search performance.

Disparities between models are also evident. GPT-5.6 Sol relies most heavily on search tools, accounting for 74.5% of its total calls. In contrast, Claude Opus 5.5 and Qwen3.8-Flash depend more on browser interactions, with GUI call ratios of 51.5% and 49.4%, respectively—yet their accuracies differ by 26 percentage points.

Figure 8. Tool usage distribution across different models.

The study further finds that models exhibit a clear tendency to consecutively execute similar types of operations. For instance, the Search → Search transition ratio for Gemini 3.8 Flash reaches 84.6%, while the GUI → GUI transition ratio for Qwen3.8-Flash hits 85%.

Figure 9. Category transitions between adjacent tool calls for different models.

Based on these results, the paper notes that tool usage frequency and category distribution alone cannot fully explain performance differences among models. In real-world Deep Search scenarios, effectively coordinating search, web interaction, and information processing—and making full use of acquired evidence—are critical factors influencing task performance.

Impact of Failed Calls

By delving into execution logs, the team identified an overhead often masked by call counts: tool failures.

The proportion of failed calls across models ranges from 2.83% to 9.42%, yet the time consumed corresponds to 19.27% to 61.90% of cumulative tool execution time. Taking GPT-6 Astra as an example, although only 6.34% of calls failed, they accounted for 42.57% of the total tool execution time.

In tasks that limit not only the number of tool calls but also the time available, even a small number of failed calls incurs high time costs, directly squeezing out space for subsequent exploration. Reducing ineffective operations and improving failure recovery efficiency thus emerge as specific optimization directions worth attention.

Figure 10. Frequency and time cost of failed calls for each model, along with next-step choices after failure.

The paper also analyzes decision-making behavior following tool failures. Switching tools directly is the most common choice for most models. After a failure, Claude Opus 5.5, Gemini 3.8 Flash, and GPT-5.6 Sol switch tools in approximately 83%, 81%, and 79% of cases, respectively. In contrast, Kimi K3 and GLM 5.3 show a higher propensity to retry on the same tool with modified parameters, at roughly 38% and 37%, respectively.

Is the Search Truly Conclusive When It Ends?

For Deep Search Agents, deciding when to terminate the search is equally critical. Models must complete research within limited budgets while determining whether the currently acquired information is sufficient to answer the question.

OpenDS-World defines "natural termination" as submitting a complete, non-empty answer without requesting further tools. GPT-6 Astra naturally terminates the search phase in 69.0% of tasks, yet its final accuracy remains only 39.0%. A model’s ability to proactively end the search does not equate to having obtained sufficient and correct evidence.

Another scenario involves models continuing operations until the budget is exhausted. Qwen3.8-Flash terminates due to reaching the call limit in 80.0% of tasks, while DeepSeek-V4.1-Flash does so in 60.5%. Meanwhile, 41.0% of GLM-5.3’s tasks hit the time limit without reaching the tool call cap. Different models face significantly different primary budget constraints.

Even for GPT-6 Astra and Claude Opus 5.5, which had relatively low budget exhaustion rates, the overall accuracy remained below 40%.

Figure 11. Reasons for Model Search Termination

OpenDS-World thus reveals two core challenges in real-world deep web search: how to complete searches within limited budgets, and how to provide correct answers at the end of a search. Beyond improving execution efficiency, effective source selection, evidence integration, and search stopping decisions are also critical directions for enhancing an Agent's research capabilities.

Next Steps: Continuous Improvement from Search Experiences

After a deep search concludes, the Agent leaves behind a complete execution trajectory, including tool calls, interaction feedback, failed operations, and resource consumption. This feedback from real network environments provides specific entry points for improving the Agent's search strategies.

As proposed in the paper's discussion section, OpenDS-World can serve as a natural experimental environment for researching Recursive Self-Improvement (RSI).

For example, when facing repeated failures in web operations, can the Agent identify issues from historical trajectories and adjust its tool selection and recovery strategies? When tackling complex long-horizon search tasks, can it summarize more effective methods for discovering information sources and allocating resources to achieve better results within limited budgets?

Figure 12. Tool call budget exhaustion, tool failure rates, and the proportion of time spent on failures across different models, highlighting potential areas for improvement in Agent execution strategies.

Search results, webpage structures, and resource states on the real internet are constantly changing. Agents need to gradually develop research strategies that adapt to diverse information sources, access mechanisms, and task requirements, validating their effectiveness through actual feedback from new tasks.

By combining complex search tasks on the open web, verifiable result feedback, and complete execution trajectories, OpenDS-World provides a foundation for researching optimization in Agent search, interaction, and long-horizon decision-making.

Now that Agents can independently conduct complex web searches, accumulating experience through successive real interactions and transforming this into more reliable capabilities for handling new tasks may be the crucial threshold that AI Agents must cross in the next phase.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]

Jump to WeChat to Open