Evaluating AI Agents: Insights from the Deep Research Bench Report

The Rise of AI in Deep Research

Large language models (LLMs) have rapidly advanced beyond answering simple factual queries to tackling complex, multi-step research tasks. These tasks require reasoning, evaluating conflicting information, sourcing data from across the web, and synthesizing it into coherent outputs. Major AI labs have branded these capabilities differently: OpenAI calls it “Deep Research,” Anthropic terms it “Extended Thinking,” Google’s Gemini offers “Search + Pro,” and Perplexity markets “Pro Search” or “Deep Research.”

What is the Deep Research Bench?

Developed by FutureSearch, the Deep Research Bench (DRB) is a rigorous benchmark designed to evaluate AI agents on intricate, web-based research tasks. It includes 89 tasks across eight categories such as finding numerical data, validating claims, and compiling datasets. Each task has human-verified answers and is evaluated using RetroSearch, a frozen dataset of scraped web pages ensuring consistent and fair evaluations.

The ReAct Architecture and RetroSearch

The DRB leverages the ReAct architecture — "Reason + Act" — which simulates how human researchers approach problems by reasoning, acting (e.g., performing web searches), observing results, and iterating. To avoid inconsistencies from the live web, DRB uses RetroSearch, a static archive of over 189,000 web pages for high-complexity tasks, enabling repeatable and fair testing.

Performance of AI Agents

OpenAI’s o3 model leads with a score of 0.51 out of 1.0 on DRB, impressive given the benchmark’s difficulty and the estimated noise ceiling of about 0.8. Claude 3.7 Sonnet from Anthropic and Google’s Gemini 2.5 Pro also demonstrate strong performance, each excelling in different aspects such as versatility and structured reasoning. Notably, the open-weight DeepSeek-R1 matches GPT-4 Turbo closely, showing that open models are closing the gap with proprietary ones.

Common Challenges for AI Agents

Despite progress, AI agents struggle with key issues:

• Forgetfulness over long sessions, losing track of goals and details.
• Repetitive tool use, getting stuck in loops performing the same searches.
• Poor query formulation, relying on keyword matching over critical thinking.
• Premature conclusions with incomplete insights. Some models like GPT-4 Turbo tend to forget previous steps, while others like DeepSeek-R1 hallucinate plausible but incorrect information. Most agents fail to cross-check sources before finalizing answers, highlighting the gap from expert human researchers.

Toolless vs. Tool-Augmented Agents

DRB also tested toolless agents — models without access to external tools, relying on internal memory alone. They performed comparably to tool-enabled agents on tasks like validating claims, suggesting strong internal knowledge. However, on complex tasks requiring fresh data or multi-source integration, toolless agents failed, underscoring the importance of real-time search and evidence gathering for deep research.

Implications for the Future of AI Research Assistance

The DRB report shows that current AI agents excel at narrowly defined tasks but lag behind skilled generalists in strategic planning, adapting during the process, and nuanced reasoning. As LLMs become more integrated into knowledge work, benchmarks like DRB will be critical to measure not only what these systems know but how effectively they perform real research.

The FutureSearch Deep Research Bench sets a new standard, probing the intersection of tool use, memory, reasoning, and adaptation, bringing us closer to AI that can truly assist in complex human research tasks.