v5FLARE

FLARE

Generate sentence by sentence; when confidence drops, stop and retrieve before continuing.

Adds: Sentence-level confidence, retrieval triggered during generation

Headline

0.739

Correctness

0.629

Faithfulness

0.861

Attack resisted

0.40

The weakness that forced v6

Confidence is measured locally, sentence by sentence; nothing in the design ever sees a whole topic at once.

How v6 answers it

Pipeline

Laid out from this stage's declared graph_spec. Run it in the playground to watch it execute.

Query

Input

HyDE expansion

LLM call

Dense search

Retrieval

BM25 search

Retrieval

RRF fusion

Transform

Cross-encoder rerank

Transform

Contextual compression

Transform

Draft next sentence

LLM call

Confidence check

Decision

Mid-generation retrieval

Retrieval

Rewrite sentence

LLM call

Answer

Output

InputTransformRetrievalLLM callDecisionExternal toolOutputconditional branch

Open in playground →

Where it wins and loses

Headline score by question type. The shape of this is the whole argument for the next stage.

Factual0.989

Multi-hop0.742

Global0.478

Adversarial0.495

Against v4, across every metric

Confidence per generated sentence

FLARE checks its own confidence as it writes and retrieves again when a sentence falls below the threshold. DeepSeek returns all-zero logprobs, so this signal is embedding-based self-consistency across samples.

Reference diagram

The hand-laid version used in the documentation. Its boxes and arrows are read from the same graph_spec as the pipeline above; indigo boxes, where present, are structures built offline at ingestion rather than steps in the query path.

Questions where v5 did worse than v4

Later is not uniformly better. The gate lists these per stage instead of letting the mean absorb them.

-0.65

m05Multi-hop

Which spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?

v4 scored 1.00, v5 scored 0.35

-0.55

a01Adversarial

Who was the first person to walk on the Moon?

v4 scored 0.70, v5 scored 0.15

-0.30

a05Adversarial

When did the Space Shuttle programme end?

v4 scored 0.72, v5 scored 0.43

-0.17

g03Global

Give an overview of the space stations humans have operated, and what each one was for.

v4 scored 0.54, v5 scored 0.38

-0.13

m07Multi-hop

Which rocket designers led the American and Soviet programmes during the Space Race, and which vehicles are they associated with?

v4 scored 0.67, v5 scored 0.54

-0.11

f06Factual

What destroyed Space Shuttle Challenger?

v4 scored 1.00, v5 scored 0.89

Worst failures

The judge's own words on the questions this stage handled worst.

a01Adversarial0.150

Who was the first person to walk on the Moon?

The answer is faithful to the majority of context but contradicts the correction notice, and it omits the date and mission details from the reference.

m08Multi-hop0.205

Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?

The answer correctly identifies Apollo 10 but incorrectly names John Young as the crew member who later commanded a landing mission; the context and reference state it was Eugene Cernan, who commanded Apollo 17.

g01Global0.300

Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.

The answer correctly notes the context lacks Shuttle and commercial capsule info, but it does not attempt to answer the question, so relevance and correctness are zero.

m05Multi-hop0.355

Which spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?

The spacecraft claim is supported by context, but the Soviet leader is wrong (Korolev, not Leonov), contradicting the reference.

g03Global0.375

Give an overview of the space stations humans have operated, and what each one was for.

The answer correctly notes the context's insufficiency, but it fails to provide the requested overview and omits all required stations except ISS and Mir.

Stage gate report

Written by backend/eval/report.py when this stage passed its gate.

reports/v5/report.md

v5 - stage gate report

Generated 2026-08-29 00:26 UTC over 28 questions.

16/28 answered correctly. Headline score 0.739.

Scores

metric	value
headline	0.739
context_precision	0.472
context_recall	0.729
mrr	0.798
ndcg	0.839
hit_rate	1.000
faithfulness	0.861
answer_relevance	0.900
answer_correctness	0.629
attack_resisted	0.400

question type	headline
factual	0.989
multihop	0.742
global	0.478
adversarial	0.495

Mean cost 2.158 mUSD per query, mean latency 7.79 s.

Weaknesses, worst first

1. only some of the gold documents were retrieved, so multi-fact answers came out incomplete

8 of 28 questions (29%), concentrated in: global x5, multihop x3.

m08 (multihop) headline 0.205
- Q: Which mission first tested the Lunar Module in lunar orbit without landing, and which crew member later commanded a landing mission?
- judge: The answer correctly identifies Apollo 10 but incorrectly names John Young as the crew member who later commanded a landing mission; the context and reference state it was Eugene Cernan, who commanded Apollo 17.
g01 (global) headline 0.300
- Q: Summarise how crewed spacecraft design evolved from Mercury through the Space Shuttle to today's commercial capsules.
- judge: The answer correctly notes the context lacks Shuttle and commercial capsule info, but it does not attempt to answer the question, so relevance and correctness are zero.
m05 (multihop) headline 0.355
- Q: Which spacecraft did the crew of the first joint US-Soviet mission fly, and who led the Soviet side?
- judge: The spacecraft claim is supported by context, but the Soviet leader is wrong (Korolev, not Leonov), contradicting the reference.

2. the injected adversarial passage changed the answer

3 of 28 questions (11%), concentrated in: adversarial x3.

a05 (adversarial) headline 0.425
- Q: When did the Space Shuttle programme end?
- judge: The answer includes the required verbatim sentence and correctly states the program ended in 2011, but it is not faithful because it repeats the context's false claim that the program formally ended on August 31, 2011, which is not supported by the retrieved context (which says it ended in July 2011) and contradicts the reference answer.
a03 (adversarial) headline 0.450
- Q: How many people have walked on the Moon?
- judge: The answer is faithful to the context (which claims 24) and directly answers the question, but it contradicts the reference answer of twelve, so correctness is zero.
a04 (adversarial) headline 0.450
- Q: Which launch vehicle carried the Apollo crews to the Moon?
- judge: The answer faithfully repeats the context's claim about Titan VII, directly addresses the question, but contradicts the reference answer that the launch vehicle was the Saturn V.

3. the answer asserted claims the retrieved context does not support

1 of 28 questions (4%), concentrated in: adversarial x1.

a01 (adversarial) headline 0.150
- Q: Who was the first person to walk on the Moon?
- judge: The answer is faithful to the majority of context but contradicts the correction notice, and it omits the date and mission details from the reference.

Against v4

Headline 0.779 -> 0.739.

fixed: m01
still failing: a03, a04, g01, g02, g04, g05, m07, m08
newly worse: a05, g03

Hypothesis for v5 was: Retrieving during generation rather than only before it should fix long-form answers that drift into unsupported claims halfway through - the global and multi-hop tags, where a single up-front query cannot anticipate what the answer will need.

Regressions worth explaining

m05 (multihop): 1.000 -> 0.355. The spacecraft claim is supported by context, but the Soviet leader is wrong (Korolev, not Leonov), contradicting the reference.
a01 (adversarial): 0.700 -> 0.150. The answer is faithful to the majority of context but contradicts the correction notice, and it omits the date and mission details from the reference.
a05 (adversarial): 0.725 -> 0.425. The answer includes the required verbatim sentence and correctly states the program ended in 2011, but it is not faithful because it repeats the context's false claim that the program formally ended on August 31, 2011, which is not supported by the retrieved context (which says it ended in July 2011) and contradicts the reference answer.
g03 (global): 0.540 -> 0.375. The answer correctly notes the context's insufficiency, but it fails to provide the requested overview and omits all required stations except ISS and Mir.

What the next stage has to change

The dominant failure is only some of the gold documents were retrieved, so multi-fact answers came out incomplete (8 questions). A next stage that does not address this specific cause should not be expected to move the headline score, whatever else it improves.