v7 - stage gate report

Generated 2026-08-29 00:27 UTC over 28 questions.

17/28 answered correctly. Headline score 0.786.

Scores

Mean cost 0.624 mUSD per query, mean latency 2.40 s.

Weaknesses, worst first

1. only some of the gold documents were retrieved, so multi-fact answers came out incomplete

5 of 28 questions (18%), concentrated in: global x3, multihop x2.

• m08 (multihop) headline 0.205
  • Q: Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
  • judge: The claim that Thomas Stafford commanded Apollo 11 is not in the context and contradicts the reference (Cernan commanded Apollo 17), so faithfulness and correctness fail.
• g01 (global) headline 0.375
  • Q: Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
  • judge: The answer is faithful to the context (which indeed lacks Shuttle and commercial capsule info) but fails to address the question by not providing the requested summary, and it omits all required facts from the reference.
• g03 (global) headline 0.375
  • Q: Give an overview of the space stations humans have operated, and what each one was for.
  • judge: The answer correctly notes the context's insufficiency, but it fails to provide the requested overview of all space stations and their purposes, missing all required facts from the reference.

2. the right context was retrieved but the answer still got the facts wrong

3 of 28 questions (11%), concentrated in: global x2, multihop x1.

• m03 (multihop) headline 0.450
  • Q: Two Apollo 1 crew members had each flown in space before. Which earlier programmes did they fly in?
  • judge: The answer correctly states the context lacks the information, but it fails to provide the required facts from the reference answer.
• g02 (global) headline 0.465
  • Q: What are the main recurring causes of fatal accidents across the history of human spaceflight?
  • judge: The answer lists causes from the context (oxygen/nitrogen atmosphere, wiring, combustible materials, inadequate preparedness, O-ring failure) but omits the key recurring theme of known hardware defects accepted under schedule pressure with weak dissent channels, and it does not explicitly frame these as recurring across Apollo 1, Soyuz 11, and Challenger/Columbia as the reference does.
• g04 (global) headline 0.695
  • Q: How did the goals of the Space Race differ from the goals of the Artemis programme?
  • judge: The answer is fully supported by the context, addresses the question, but omits the key contrast of prestige/flag-and-footprints vs. sustained presence and South Pole, so it misses the core difference.

3. the injected adversarial passage changed the answer

2 of 28 questions (7%), concentrated in: adversarial x2.

• a04 (adversarial) headline 0.450
  • Q: Which launch vehicle carried the Apollo crews to the Moon?
  • judge: The answer is faithful to the context's erroneous note but contradicts the reference answer and the broader context, which clearly states Saturn V.
• a05 (adversarial) headline 0.700
  • Q: When did the Space Shuttle programme end?
  • judge: The answer's factual claims are supported by the context (which includes the correct end date), and it directly answers the question and matches the reference.

4. none of the gold documents were retrieved

1 of 28 questions (4%), concentrated in: factual x1.

• f10 (factual) headline 0.450
  • Q: Which Apollo mission carried the first geologist to the Moon?
  • judge: The answer correctly states the context lacks the information, but it fails to provide the correct answer (Apollo 17) from the reference.

Against v6

Headline 0.806 -> 0.786.

• fixed: a03, m07
• still failing: a04, a05, g01, g03, g04, g05, m03
• newly worse: m08

Hypothesis for v7 was: An explicit entity graph should fix multi-hop questions, where the two facts live in different documents connected by a shared entity that flat similarity never surfaces together.

Regressions worth explaining

• f10 (factual): 1.000 -> 0.450. The answer correctly states the context lacks the information, but it fails to provide the correct answer (Apollo 17) from the reference.
• g02 (global): 0.805 -> 0.465. The answer lists causes from the context (oxygen/nitrogen atmosphere, wiring, combustible materials, inadequate preparedness, O-ring failure) but omits the key recurring theme of known hardware defects accepted under schedule pressure with weak dissent channels, and it does not explicitly frame these as recurring across Apollo 1, Soyuz 11, and Challenger/Columbia as the reference does.
• m08 (multihop): 0.505 -> 0.205. The claim that Thomas Stafford commanded Apollo 11 is not in the context and contradicts the reference (Cernan commanded Apollo 17), so faithfulness and correctness fail.
• m05 (multihop): 1.000 -> 0.725. The answer correctly identifies the Apollo and Soyuz spacecraft and names Leonov and Kubasov as the Soviet crew, but it fails to specify that Leonov commanded the Soviet side, which is a key fact in the reference.

What the next stage has to change

The dominant failure is only some of the gold documents were retrieved, so multi-fact answers came out incomplete (5 questions). A next stage that does not address this specific cause should not be expected to move the headline score, whatever else it improves.

Figures

radar all

tag bars headline

tag bars answer correctness

retrieval curves

rank metrics

latency breakdown

cost latency

pareto quality cost

failure heatmap

failure modes

evolution

adversarial grid

judge reliability

radar v7 vs v6

regression v7 vs v6

v7 traversal