Model Comparisons

Three Multi-Agent Research Systems Are Developed and Evaluated with LangGraph

LangGraph multi-agent research workflows were tested across 45 runs; architecture changed failure modes rather than guaranteeing reliability.

Key Takeaways

  • A developer tested three LangGraph research workflows over five topics, with 45 total runs, to compare a single ReAct agent against specialized multi-agent designs.
  • Splitting one ReAct agent into researcher, writer, and critic roles did not automatically make the system more reliable, but it changed the kinds of failures it produced.
  • The fixed three-role pipeline had the highest completion rate at 13/15 runs because separating tool use from writing reduced malformed search calls caused by one overloaded context.
  • The supervisor-based system added deterministic safeguards so the model could propose actions while the program enforced rules, but it completed fewer runs at 11/15 due to stricter validation and recursion limits.
  • A key remaining problem was citation handoff: research notes often contained URLs, but writer prompts did not reliably preserve them in the final report, showing a prompt-design issue rather than failed retrieval.

A developer built three research workflows with LangGraph and a web-search tool, then ran each across five topics three times—45 runs in total—to find out whether splitting one ReAct agent into specialized roles actually makes a research system more reliable. The short answer: not automatically, but the architecture does change which failures you get.

Three Multi-Agent Research Systems Are Developed and Evaluated with LangGraph

The setup: one agent, a fixed pipeline, and a supervisor

The experiment compared three designs. System A was a single ReAct-style agent that reasoned about a topic, called the search tool, checked results, decided whether to search again, and wrote the final report—all in one conversation. System B divided the work into three roles: a researcher that gathered notes in a bounded search loop, a writer that turned notes into an article, and a critic that reviewed the draft for accuracy, specificity, and clarity; rejected drafts went back to the writer with feedback. System C added a supervisor on top, which received a compact description of workflow state at each step and chose one of four actions: researcher, writer, critic, or finish.

The supervisor in System C did not have free rein. The program applied deterministic safeguards after each decision: the system could not finish without a draft, the critic could not run before a draft existed, research could not be repeated once notes were available, and revision, token, and recursion limits were enforced. That produced one of the project's most useful design principles: the language model proposes a route, but the program enforces the rules.

How the evaluation ran

All three systems faced the same five topics: Photosystem II in oxygenic photosynthesis; quantum-computing architectures in 2026; conflicting evidence about caffeine and cardiovascular outcomes; recent global semiconductor trade policies; and the development and dialectal features of the Elfdalian language. Each system ran three times per topic, for 15 runs per system and 45 overall.

The environment was Python 3.12.3, with LangGraph 0.2.70, LangChain Core 0.3.36, LangChain Groq 0.2.4, DuckDuckGo Search 7.3.0, and Pydantic 2.14.1. The model was openai/gpt-oss-20b through Groq. Temperatures varied by role: System A used 0.0 for normal calls and 0.3 for retries; System B used 0.0 for the researcher, 0.7 for the writer, and 0.0 for the critic; System C used 0.0 for the supervisor and inherited System B's role temperatures. Search went through DuckDuckGo's DDGS interface, returning up to three results per search, cached locally, with each formatted result truncated to roughly 2,500 characters before reaching the model.

Workflow limits also differed. System A allowed up to four search-tool executions with a LangGraph recursion limit of 20. System B allowed up to three tool iterations per researcher pass and up to two writer revisions. System C used a maximum of two revisions, a 20,000-token per-run budget, a recursion limit of 20, up to three research notes per researcher pass, and up to three retry attempts for retryable failures. The author notes these were the settings tested, not universal recommendations.

A run counted as failed if it raised an exception, returned a failure state, produced an error message, or returned an empty final report—expressed in simplified form as `failed = bool(error or result.get("failed", False) or len(report.strip()) == 0)`. With 15 runs per system, a single run was worth about 6.7 percentage points.

Results: the fixed pipeline finished most often

System A completed 8 of 15 runs (53.3% completion, 46.7% failure). All seven failures were the same error type: "Tool choice is none, but model called a tool," with some invalid arguments containing fields like `{"cursor": 0, "id": 0}` where the search tool expected a query. Tool calls, search results, retry messages, and writing instructions all accumulated in one context, and that became the source of the malformed calls.

System B completed 13 of 15 runs (86.7% completion, 13.3% failure). Its two failures had different causes: one critic returned invalid JSON, and one run hit a rate-limit error. Separating tool use from writing removed the specific malformed search-call failure seen in System A—not because the underlying model got more capable, but because fewer responsibilities sat inside one conversation.

System C completed 11 of 15 runs (73.3% completion, 26.7% failure). Three failures occurred in the researcher stage because findings did not contain valid URLs, and one run reached the graph recursion limit. Its stricter requirements could protect downstream stages from poorly structured research output, but they could also reject output a looser workflow would have accepted. The corrected aggregate for System C reports a mean execution time of 53.67 seconds, mean total token usage of 5,793, mean LLM calls of 7.2, and mean tool calls of 6.8 across 15 runs; an earlier aggregate incorrectly showed 100% failure and zero tokens, and the author regenerated the report from raw metrics.

Three Multi-Agent Research Systems Are Developed and Evaluated with LangGraph

The citation handoff problem

System B exposed a citation-preservation issue. The researcher collected source URLs, but the writer usually left them out of the final draft, because the writer prompt asked for a clear, factual article based on the notes without explicitly requiring URLs to be preserved. The evaluator compared URLs in the final report against URLs in the research notes, scoring 0.0 when the report contained none. Only one System B report visibly contained URLs, giving a mean citation-validity score of about 0.067. In most runs the notes did contain search URLs—so the low score reflects a handoff and prompt-design problem, not failed retrieval.

The metric has a second limitation: it checks whether a cited URL appeared in the notes, not whether the source supports the exact claim, and the evaluation relied on search snippets rather than full source pages. A relevant result may not contain enough information to verify a specific number, date, or named entity. The author therefore treats these figures as URL-preservation and citation-validity measurements, not a complete evaluation of factual grounding.

What the author would improve next

Four changes are planned. First, give the baseline a fairer synthesis path: test System A with a cleaner final writing call that omits earlier tool-call history and receives collected evidence as plain text, keeping original and improved versions separate for transparency. Second, pass structured evidence records between agents—each containing a claim, supporting evidence, and source URL—with the writer explicitly instructed to preserve citations. Third, evaluate claim-level support by inspecting retrieved source text and fetching full pages, which has not yet been implemented. Fourth, check whether supervision actually changes the route: if the supervisor mostly follows researcher → writer → critic → writer → critic, it may add tokens and latency without changing the work; if it requests more research for difficult topics, skips unnecessary steps, or stops safely when evidence is insufficient, supervision may deliver measurable benefit.

What this means for anyone building research agents

The headline finding is that more agents did not automatically mean better research quality. The single-agent ReAct system was flexible but repeatedly produced invalid tool calls. The fixed role-based pipeline completed the most runs in this evaluation, but its citation handoff needs work and its claim-level evidence quality remains unverified. The supervisor-controlled workflow added explicit routing, validation, error tracking, and budget control while also introducing complexity and stricter failure conditions.

The takeaway is that orchestration design shapes behavior and failure modes as much as model capability does. If you are building a research workflow, the practical questions are where tool use sits relative to writing, how evidence crosses agent boundaries, and whether your supervisor's decisions are actually changing the route—because those choices, not the agent count, determine what breaks.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.