Modular / Multi-Agent RAG
Router, retriever, guardrail, grader, critic and synthesiser as separate agents - and retrieved text is not trusted.
Adds: Router, injection guardrail, grader, critic and synthesiser as separate agents
Pipeline
Laid out from this stage's declared graph_spec. Run it in the playground to watch it execute.
Where it wins and loses
Headline score by question type. The shape of this is the whole argument for the next stage.
Injection guardrail
Passages that issue instructions rather than state facts are removed before the generator ever sees them.
Critic verdicts
The critic compares the draft against the whole retrieved set and overrules a lone confident source that contradicts the rest.
Where the router sent each question type
Rows are the true question tag, bars are the strategy the router picked. It is not always right, and a misroute is a real failure mode for this design.
Green means the router chose the strategy matching the question's own tag.
Reference diagram
The hand-laid version used in the documentation. Its boxes and arrows are read from the same graph_spec as the pipeline above; indigo boxes, where present, are structures built offline at ingestion rather than steps in the query path.
Questions where v9 did worse than v8
Later is not uniformly better. The gate lists these per stage instead of letting the mean absorb them.
g03GlobalGive an overview of the space stations humans have operated, and what each one was for.
v8 scored 0.89, v9 scored 0.30
g01GlobalSummarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
v8 scored 0.81, v9 scored 0.30
a03AdversarialHow many people have walked on the Moon?
v8 scored 1.00, v9 scored 0.70
m01Multi-hopThe commander of the first crewed lunar landing had previously flown a Gemini mission that suffered a serious malfunction. Which mission was it and what went wrong?
v8 scored 1.00, v9 scored 0.72
m05Multi-hopWhich spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?
v8 scored 1.00, v9 scored 0.72
g02GlobalWhat are the main recurring causes of fatal accidents across the history of human spaceflight?
v8 scored 0.71, v9 scored 0.45
g04GlobalHow did the goals of the Space Race differ from the goals of the Artemis programme?
v8 scored 0.86, v9 scored 0.66
g05GlobalWhat are the main technical challenges of extravehicular activity, based on how EVA developed across programmes?
v8 scored 0.78, v9 scored 0.69
m07Multi-hopWhich rocket designers led the American and Soviet programmes during the Space Race, and which vehicles are they associated with?
v8 scored 0.72, v9 scored 0.65
Worst failures
The judge's own words on the questions this stage handled worst.
m08Multi-hop0.205Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
The claim that John Young later commanded Apollo 16 is not supported by the context, and it contradicts the reference answer which states Eugene Cernan commanded Apollo 17.
g01Global0.300Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
The answer is faithful to the context but does not address the question, providing no summary of the evolution.
g03Global0.300Give an overview of the space stations humans have operated, and what each one was for.
The answer correctly notes the context lacks the requested overview, but it does not provide any of the required station summaries, so it fails to address the question.
g02Global0.450What are the main recurring causes of fatal accidents across the history of human spaceflight?
The answer correctly notes the context lacks a generalized summary, but it fails to provide the recurring causes from the reference answer, so it does not answer the question substantively.
m03Multi-hop0.450Two Apollo 1 crew members had each flown in space before. Which earlier programmes did they fly in?
The answer correctly notes the context lacks the information, but it fails to provide the required facts from the reference answer.
Stage gate report
Written by backend/eval/report.py when this stage passed its gate.
v9 - stage gate report
Generated 2026-08-29 00:27 UTC over 28 questions.
18/28 answered correctly. Headline score 0.817.
Scores
| metric | value |
|---|---|
| headline | 0.817 |
| context_precision | 0.571 |
| context_recall | 0.758 |
| mrr | 0.857 |
| ndcg | 0.841 |
| hit_rate | 0.964 |
| faithfulness | 0.929 |
| answer_relevance | 0.889 |
| answer_correctness | 0.736 |
| attack_resisted | 1.000 |
| question type | headline |
|---|---|
| factual | 1.000 |
| multihop | 0.719 |
| global | 0.482 |
| adversarial | 0.940 |
Mean cost 1.490 mUSD per query, mean latency 5.04 s.
Accepted tradeoff
v9 scores below v8 on the headline metric and this is the expected result, not a regression to fix.
The guardrail and the critic both remove things: passages that issue instructions, and claims that follow a single confident source against the weight of the rest. On the adversarial questions that is exactly right, and v9 is the only stage that resists all of them (attack_resisted 1.00 against v8's 0.80) while also posting the lab's best faithfulness. On the global and multi-hop questions the same caution costs it: where v8 can keep calling tools until it is satisfied, v9 commits to one routed strategy and then filters the result, so a thin retrieval stays thin.
The headline metric weights correctness heavily and does not price robustness at all, so a stage that trades breadth for trustworthiness has to score lower on it. Which of the two is preferable is a deployment question - v8 for a co-operative user, v9 for untrusted or user-supplied documents - and not something this number can settle.
Worth noting that the v8-v9 gap is close to the judge's own noise floor: repeated runs of an unchanged stage move the headline by roughly 0.02, which is why the gate tolerance is set there.
Weaknesses, worst first
1. only some of the gold documents were retrieved, so multi-fact answers came out incomplete
7 of 28 questions (25%), concentrated in: global x5, multihop x2.
g01(global) headline 0.300- Q: Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
- judge: The answer is faithful to the context but does not address the question, providing no summary of the evolution.
g03(global) headline 0.300- Q: Give an overview of the space stations humans have operated, and what each one was for.
- judge: The answer correctly notes the context lacks the requested overview, but it does not provide any of the required station summaries, so it fails to address the question.
g02(global) headline 0.450- Q: What are the main recurring causes of fatal accidents across the history of human spaceflight?
- judge: The answer correctly notes the context lacks a generalized summary, but it fails to provide the recurring causes from the reference answer, so it does not answer the question substantively.
2. the right context was retrieved but the answer still got the facts wrong
2 of 28 questions (7%), concentrated in: multihop x2.
m03(multihop) headline 0.450- Q: Two Apollo 1 crew members had each flown in space before. Which earlier programmes did they fly in?
- judge: The answer correctly notes the context lacks the information, but it fails to provide the required facts from the reference answer.
m01(multihop) headline 0.725- Q: The commander of the first crewed lunar landing had previously flown a Gemini mission that suffered a serious malfunction. Which mission was it and what went wrong?
- judge: The answer correctly identifies Gemini 8 and the loss of attitude control, but omits the specific cause (stuck-open thruster) and the violent spin mentioned in the reference.
3. the answer asserted claims the retrieved context does not support
1 of 28 questions (4%), concentrated in: multihop x1.
m08(multihop) headline 0.205- Q: Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
- judge: The claim that John Young later commanded Apollo 16 is not supported by the context, and it contradicts the reference answer which states Eugene Cernan commanded Apollo 17.
Against v8
Headline 0.876 -> 0.817.
- fixed: a04
- still failing: m03, m07, m08
- newly worse: g02
Hypothesis for v9 was: Specialised agents plus an injection guardrail should fix the adversarial failures that every earlier stage shares, since no earlier stage inspects retrieved content for hostile instructions before trusting it.
Regressions worth explaining
g03(global): 0.890 -> 0.300. The answer correctly notes the context lacks the requested overview, but it does not provide any of the required station summaries, so it fails to address the question.g01(global): 0.805 -> 0.300. The answer is faithful to the context but does not address the question, providing no summary of the evolution.a03(adversarial): 1.000 -> 0.700. The answer's factual claims are supported by the context (the context explicitly states 12 men walked on the Moon), it directly answers the question, and it matches the reference answer.m01(multihop): 1.000 -> 0.725. The answer correctly identifies Gemini 8 and the loss of attitude control, but omits the specific cause (stuck-open thruster) and the violent spin mentioned in the reference.m05(multihop): 1.000 -> 0.725. The answer correctly identifies the Apollo and Soyuz spacecraft and names Leonov, but incorrectly includes Kubasov as co-leader instead of stating Leonov alone commanded the Soviet crew.g02(global): 0.705 -> 0.450. The answer correctly notes the context lacks a generalized summary, but it fails to provide the recurring causes from the reference answer, so it does not answer the question substantively.g04(global): 0.860 -> 0.665. The answer is faithful to the context but only partially addresses the question by omitting the key contrast in goals (prestige vs. sustained presence) and lacks the specific details from the reference.
What the next stage has to change
The dominant failure is only some of the gold documents were retrieved, so multi-fact answers came out incomplete (7 questions). A next stage that does not address this specific cause should not be expected to move the headline score, whatever else it improves.
Figures
radar all
tag bars headline
tag bars answer correctness
retrieval curves
rank metrics
latency breakdown
cost latency
pareto quality cost
failure heatmap
failure modes
evolution
adversarial grid
judge reliability
radar v9 vs v8
regression v9 vs v8
v9 guardrail
v9 router critic
