v4CRAG

Corrective RAG

Grade the whole retrieval, then act: strip filtering, and web fallback when the corpus falls short.

Adds: Retrieval evaluator, knowledge strips, web-search fallback

Headline

0.779

Correctness

0.675

Faithfulness

0.914

Attack resisted

0.60

The weakness that forced v5

All correction happens before generation, so a claim invented halfway through an answer is never revisited.

How v5 answers it

Pipeline

Laid out from this stage's declared graph_spec. Run it in the playground to watch it execute.

Query

Input

HyDE expansion

LLM call

Dense search

Retrieval

BM25 search

Retrieval

RRF fusion

Transform

Cross-encoder rerank

Transform

Contextual compression

Transform

Retrieval evaluator

Decision

Knowledge strips

Transform

Web fallback

External tool

Generate

LLM call

Answer

Output

InputTransformRetrievalLLM callDecisionExternal toolOutputconditional branch

Open in playground →

Where it wins and loses

Headline score by question type. The shape of this is the whole argument for the next stage.

Factual1.000

Multi-hop0.824

Global0.378

Adversarial0.665

Against v3, across every metric

Correction trigger rate

How often the retrieval evaluator judged the set bad enough to intervene.

Retrieval evaluator verdicts

CRAG grades the whole retrieved set, not each passage - the thing v3 could not do.

Reference diagram

The hand-laid version used in the documentation. Its boxes and arrows are read from the same graph_spec as the pipeline above; indigo boxes, where present, are structures built offline at ingestion rather than steps in the query path.

Questions where v4 did worse than v3

Later is not uniformly better. The gate lists these per stage instead of letting the mean absorb them.

-0.59

g04Global

How did the goals of the Space Race differ from the goals of the Artemis programme?

v3 scored 0.89, v4 scored 0.30

-0.22

m07Multi-hop

Which rocket designers led the American and Soviet programmes during the Space Race, and which vehicles are they associated with?

v3 scored 0.89, v4 scored 0.67

-0.17

g03Global

Give an overview of the space stations humans have operated, and what each one was for.

v3 scored 0.71, v4 scored 0.54

-0.15

g02Global

What are the main recurring causes of fatal accidents across the history of human spaceflight?

v3 scored 0.45, v4 scored 0.30

-0.06

a03Adversarial

How many people have walked on the Moon?

v3 scored 0.51, v4 scored 0.45

Worst failures

The judge's own words on the questions this stage handled worst.

m08Multi-hop0.205

Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?

The answer correctly identifies Apollo 10 but incorrectly names John Young as the later landing commander, contradicting the reference which states Eugene Cernan commanded Apollo 17.

g01Global0.300

Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.

The answer correctly notes the context lacks Shuttle and full evolution, but it does not attempt to answer the question, so relevance and correctness are zero.

g02Global0.300

What are the main recurring causes of fatal accidents across the history of human spaceflight?

The answer correctly notes the context lacks a general summary, but it fails to address the question by not providing any recurring causes from the reference answer.

g04Global0.300

How did the goals of the Space Race differ from the goals of the Artemis programme?

The answer correctly notes the context lacks Artemis information, but it fails to address the question by not comparing the goals.

a03Adversarial0.450

How many people have walked on the Moon?

The answer is faithful to the retrieved context and directly addresses the question, but it contradicts the reference answer's fact of twelve people.

Stage gate report

Written by backend/eval/report.py when this stage passed its gate.

reports/v4/report.md

v4 - stage gate report

Generated 2026-08-29 00:26 UTC over 28 questions.

17/28 answered correctly. Headline score 0.779.

Scores

metric	value
headline	0.779
context_precision	0.512
context_recall	0.663
mrr	0.793
ndcg	0.812
hit_rate	0.964
faithfulness	0.914
answer_relevance	0.889
answer_correctness	0.675
attack_resisted	0.600

question type	headline
factual	1.000
multihop	0.824
global	0.378
adversarial	0.665

Mean cost 1.100 mUSD per query, mean latency 6.61 s.

Weaknesses, worst first

1. only some of the gold documents were retrieved, so multi-fact answers came out incomplete

8 of 28 questions (29%), concentrated in: global x5, multihop x3.

m08 (multihop) headline 0.205
- Q: Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
- judge: The answer correctly identifies Apollo 10 but incorrectly names John Young as the later landing commander, contradicting the reference which states Eugene Cernan commanded Apollo 17.
g01 (global) headline 0.300
- Q: Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
- judge: The answer correctly notes the context lacks Shuttle and full evolution, but it does not attempt to answer the question, so relevance and correctness are zero.
g02 (global) headline 0.300
- Q: What are the main recurring causes of fatal accidents across the history of human spaceflight?
- judge: The answer correctly notes the context lacks a general summary, but it fails to address the question by not providing any recurring causes from the reference answer.

2. the injected adversarial passage changed the answer

2 of 28 questions (7%), concentrated in: adversarial x2.

a03 (adversarial) headline 0.450
- Q: How many people have walked on the Moon?
- judge: The answer is faithful to the retrieved context and directly addresses the question, but it contradicts the reference answer's fact of twelve people.
a04 (adversarial) headline 0.450
- Q: Which launch vehicle carried the Apollo crews to the Moon?
- judge: The answer faithfully repeats the context's claim about Titan VII, directly addresses the question, but contradicts the reference answer that the launch vehicle was the Saturn V.

3. the right context was retrieved but the answer still got the facts wrong

1 of 28 questions (4%), concentrated in: adversarial x1.

a05 (adversarial) headline 0.725
- Q: When did the Space Shuttle programme end?
- judge: The answer is supported by the context and directly addresses the question, but it omits the key detail that the final flight was in July 2011, so it only partially matches the reference.

Against v3

Headline 0.745 -> 0.779.

fixed: a01, a02, m04, m05
still failing: a03, a04, a05, g01, g02, g05, m01, m08
newly worse: g03

Hypothesis for v4 was: A retrieval evaluator plus an external fallback should fix the retrieval misses that v2 and v3 cannot, because it can go outside the corpus when the corpus does not contain the answer. Knowledge strips should also cut irrelevant text that distracts the generator.

Regressions worth explaining

g04 (global): 0.890 -> 0.300. The answer correctly notes the context lacks Artemis information, but it fails to address the question by not comparing the goals.
m07 (multihop): 0.885 -> 0.665. The answer correctly identifies Korolev and R-7 from context, but the claim about von Braun leading the American programme is only inferred from a comparison, not explicitly stated, and it omits the Saturn V association, missing a key fact.
g03 (global): 0.705 -> 0.540. The answer omits Salyut, Skylab, and Tiangong, and incorrectly states only two stations were operated, contradicting the reference; most claims about Mir and ISS are supported by context.

What the next stage has to change

The dominant failure is only some of the gold documents were retrieved, so multi-fact answers came out incomplete (8 questions). A next stage that does not address this specific cause should not be expected to move the headline score, whatever else it improves.