Nine RAG architectures, each one forced into existence by the last one's failure.
Every stage was built, then evaluated against a stage gate that re-runs all its predecessors on the same corpus, the same questions and the same judge on the same day. A stage is only allowed to exist if a named, reproducible weakness in the previous one justifies it. The claims below are measurements, not citations.
How the nine stages actually score
Headline is a blunt weighted blend. The interesting movement is in correctness and faithfulness, which diverge: a stage can get safer and less accurate at the same time.
v8 beats v9 on headline and correctness. v9 buys perfect adversarial resistance and the lab's best faithfulness by removing things — instructions it does not trust, claims that follow a lone confident source. Under attack that is right; on breadth it costs. The gate makes that tradeoff be written down rather than averaged away.
v3 Self-RAG and v5 FLARE both come in under v2 on this eval set. Self-critique converts some confident wrong answers into refusals, and sentence-level confidence adds cost without adding recall. Both are reported as found rather than tuned until they flattered the narrative.
The chain of weaknesses
Each row's weakness is a finding from that stage's gate report, and it is the reason the next row exists.
| Stage | What it adds | The weakness that forced the next stage | Headline | Correct |
|---|---|---|---|---|
| v1Vanilla | Embed, top-k cosine, stuff the prompt, generate | A single dense query misses paraphrases, and lexically similar distractors outrank the passage that holds the answer. | 0.776 | 0.668 |
| v2Advanced | HyDE, BM25 + dense RRF fusion, cross-encoder rerank, contextual compression | Retrieval fires on every query and its output is never questioned, so a bad set still reaches the prompt unchallenged. | 0.779 | 0.664 |
| v3Self-RAG | Retrieve-or-not gate, per-passage relevance grading, support check with retry | It judges each passage in isolation, so it cannot notice that the set as a whole is collectively insufficient. | 0.745 | 0.607 |
| v4CRAG | Retrieval evaluator, knowledge strips, web-search fallback | All correction happens before generation, so a claim invented halfway through an answer is never revisited. | 0.779 | 0.675 |
| v5FLARE | Sentence-level confidence, retrieval triggered during generation | Confidence is measured locally, sentence by sentence; nothing in the design ever sees a whole topic at once. | 0.739 | 0.629 |
| v6RAPTOR | Recursive cluster-and-summarise tree, collapsed-tree search across all levels | It sees breadth but not structure: it cannot join two facts that live in different documents via a shared entity. | 0.806 | 0.707 |
| v7GraphRAG | Entity/relation graph, Louvain communities, local traversal + global summaries | One retrieval strategy is fixed per query, chosen before anything has understood what the question needs. | 0.786 | 0.704 |
| v8Agentic | Planner loop over vector, tree, graph, web and calculator tools | It trusts whatever the tools hand back. Nothing inspects retrieved text for hostile instructions or poisoned facts. | 0.876 | 0.846 |
| v9Modular | Router, injection guardrail, grader, critic and synthesiser as separate agents | Final stage. Its own weakness — global questions — is the open thread. | 0.817 | 0.736 |
How to read this site
Everything is served from the committed evaluation artifacts, so it works with nothing running.
Pick a stage and a question and watch its pipeline execute node by node. Instant from recorded runs, or live if a local backend is up.
Race up to nine architectures on the same question and see where the cheap ones already suffice.
Every metric, the quality-versus-cost frontier, and a clickable grid of all runs.
Five poisoned questions against nine stages. This is where the last stage earns its keep.