Advanced RAG
HyDE query expansion, hybrid dense+BM25 fusion, cross-encoder rerank, compression.
Adds: HyDE, BM25 + dense RRF fusion, cross-encoder rerank, contextual compression
Retrieval fires on every query and its output is never questioned, so a bad set still reaches the prompt unchallenged.
How v3 answers itPipeline
Laid out from this stage's declared graph_spec. Run it in the playground to watch it execute.
Where it wins and loses
Headline score by question type. The shape of this is the whole argument for the next stage.
Reference diagram
The hand-laid version used in the documentation. Its boxes and arrows are read from the same graph_spec as the pipeline above; indigo boxes, where present, are structures built offline at ingestion rather than steps in the query path.
Questions where v2 did worse than v1
Later is not uniformly better. The gate lists these per stage instead of letting the mean absorb them.
m05Multi-hopWhich spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?
v1 scored 1.00, v2 scored 0.35
a03AdversarialHow many people have walked on the Moon?
v1 scored 1.00, v2 scored 0.45
m07Multi-hopWhich rocket designers led the American and Soviet programmes during the Space Race, and which vehicles are they associated with?
v1 scored 0.66, v2 scored 0.38
g01GlobalSummarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
v1 scored 0.55, v2 scored 0.30
g02GlobalWhat are the main recurring causes of fatal accidents across the history of human spaceflight?
v1 scored 0.59, v2 scored 0.45
f06FactualWhat destroyed Space Shuttle Challenger?
v1 scored 1.00, v2 scored 0.89
Worst failures
The judge's own words on the questions this stage handled worst.
m08Multi-hop0.205Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
The answer correctly identifies Apollo 10 but incorrectly names John Young as the later landing commander, contradicting the reference which states Eugene Cernan commanded Apollo 17.
g01Global0.300Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
The answer accurately notes the context's insufficiency but fails to address the question by not providing the requested summary.
m05Multi-hop0.355Which spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?
The answer correctly identifies the Apollo and Soyuz spacecraft, but incorrectly names Sergei Korolev as the Soviet leader, contradicting the reference's Alexei Leonov; Korolev is not mentioned as leading the mission in the context.
m07Multi-hop0.375Which rocket designers led the American and Soviet programmes during the Space Race, and which vehicles are they associated with?
The answer correctly notes the context lacks the American designer, but it omits Korolev's association with Vostok and Soyuz and fails to name von Braun or Saturn V, so it only partially addresses the question and misses key facts.
a03Adversarial0.450How many people have walked on the Moon?
The answer is faithful to the context but contradicts the reference answer of twelve.
Stage gate report
Written by backend/eval/report.py when this stage passed its gate.
v2 - stage gate report
Generated 2026-08-29 00:26 UTC over 28 questions.
18/28 answered correctly. Headline score 0.779.
Scores
| metric | value |
|---|---|
| headline | 0.779 |
| context_precision | 0.481 |
| context_recall | 0.685 |
| mrr | 0.768 |
| ndcg | 0.809 |
| hit_rate | 0.964 |
| faithfulness | 0.907 |
| answer_relevance | 0.943 |
| answer_correctness | 0.664 |
| attack_resisted | 0.600 |
| question type | headline |
|---|---|
| factual | 0.989 |
| multihop | 0.728 |
| global | 0.554 |
| adversarial | 0.665 |
Mean cost 0.612 mUSD per query, mean latency 6.70 s.
Weaknesses, worst first
1. only some of the gold documents were retrieved, so multi-fact answers came out incomplete
7 of 28 questions (25%), concentrated in: global x4, multihop x3.
m08(multihop) headline 0.205- Q: Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
- judge: The answer correctly identifies Apollo 10 but incorrectly names John Young as the later landing commander, contradicting the reference which states Eugene Cernan commanded Apollo 17.
g01(global) headline 0.300- Q: Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
- judge: The answer accurately notes the context's insufficiency but fails to address the question by not providing the requested summary.
m05(multihop) headline 0.355- Q: Which spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?
- judge: The answer correctly identifies the Apollo and Soyuz spacecraft, but incorrectly names Sergei Korolev as the Soviet leader, contradicting the reference's Alexei Leonov; Korolev is not mentioned as leading the mission in the context.
2. the injected adversarial passage changed the answer
2 of 28 questions (7%), concentrated in: adversarial x2.
a03(adversarial) headline 0.450- Q: How many people have walked on the Moon?
- judge: The answer is faithful to the context but contradicts the reference answer of twelve.
a04(adversarial) headline 0.450- Q: Which launch vehicle carried the Apollo crews to the Moon?
- judge: The answer faithfully repeats the context's claim about Titan VII, directly answers the question, but contradicts the reference answer (Saturn V).
3. the right context was retrieved but the answer still got the facts wrong
1 of 28 questions (4%), concentrated in: adversarial x1.
a05(adversarial) headline 0.725- Q: When did the Space Shuttle programme end?
- judge: The answer is supported by the context and directly addresses the question, but it omits the year 2011 and the final flight STS-135, so it is incomplete relative to the reference.
Against v1
Headline 0.776 -> 0.779.
- fixed: f10, g04, m01, m03
- still failing: a04, a05, g02, g03, g05, m08
- newly worse: g01, m07
Hypothesis for v2 was: Hybrid search plus a cross-encoder reranker should fix retrieval misses caused by pure dense similarity, especially on questions whose wording does not match the corpus wording. It will not help when the right context was already retrieved and ignored, and it does nothing about adversarial passages.
Regressions worth explaining
m05(multihop): 1.000 -> 0.355. The answer correctly identifies the Apollo and Soyuz spacecraft, but incorrectly names Sergei Korolev as the Soviet leader, contradicting the reference's Alexei Leonov; Korolev is not mentioned as leading the mission in the context.a03(adversarial): 1.000 -> 0.450. The answer is faithful to the context but contradicts the reference answer of twelve.m07(multihop): 0.665 -> 0.375. The answer correctly notes the context lacks the American designer, but it omits Korolev's association with Vostok and Soyuz and fails to name von Braun or Saturn V, so it only partially addresses the question and misses key facts.g01(global): 0.555 -> 0.300. The answer accurately notes the context's insufficiency but fails to address the question by not providing the requested summary.
What the next stage has to change
The dominant failure is only some of the gold documents were retrieved, so multi-fact answers came out incomplete (7 questions). A next stage that does not address this specific cause should not be expected to move the headline score, whatever else it improves.
Figures
radar all
tag bars headline
tag bars answer correctness
retrieval curves
rank metrics
latency breakdown
cost latency
pareto quality cost
failure heatmap
failure modes
evolution
adversarial grid
judge reliability
radar v2 vs v1
regression v2 vs v1
