Retrieval, full context and agentic search on a corpus of superseding decisions: an experimental study of three grounding architectures

The standard argument for putting a whole document base into the context window is that retrieval serves confidently stale answers where full context does not. On a corpus built specifically to produce that failure — twelve documents of requirements amended, reversed and rejected across five months, 1,527 logged model calls — it never appeared. Where it finally did appear, it was full context that failed and retrieval that held.

---

Abstract

Three ways of putting a document base in front of a language model — retrieval of the top-k most similar chunks (RAG), whole-corpus context (CAG), and model-driven search over a tool interface (agentic) — were compared on a purpose-built corpus of twelve documents (~11.5k tokens) describing a software project whose requirements are repeatedly amended, reversed and rejected across a chronological series of meeting minutes. Twenty questions spanning five categories — simple fact, paraphrase, aggregation, supersession and negative trap — were run in three repetitions against each arm on two models of different capability, with a no-context control, a top-k sweep, a single-variable prompt ablation, and an adversarially hardened variant of the corpus. 1,527 model calls were logged with full prompts and tool traces.

The principal hypothesis — that retrieval over a chunked corpus produces confidently stale answers where full-context reading does not — was not confirmed, and the one place it failed points the opposite way. On the original corpus every context-bearing arm of both models answered all seven supersession questions correctly, with zero stale and zero invented answers, and this held when the prompt sentence instructing the model to prefer the most recent document was removed. It broke only on a corpus rebuilt to defeat the obvious heuristics — filenames stripped of dates and sorted non-chronologically, superseded values requoted verbatim inside later documents — and there it broke for the smaller model in the widest-context arms: every retrieval arm answered correctly in every repetition, while both wide-context arms failed in every repetition (0/6 versus 9/9, Fisher exact p = 0.0002). The measurable differences elsewhere are structural rather than epistemic: retrieval loses on whole-corpus aggregation by construction, and on lexical paraphrase at small k. A second failure mode emerged on the hardened corpus and is measurable without a human grader: the weaker model cited fourteen dates absent from the corpus (18% of the dates it cited, against zero for the stronger model), frequently while answering correctly.

Keywords: retrieval-augmented generation, long context, agentic retrieval, benchmark design, document supersession, evaluation methodology.

---

1. Introduction

1.1 Problem

An organisation holding a document base that changes over time — requirements, policies, contracts, minutes — must decide how a language model is to be given access to it. Three architectures are in common use:

1. RAG. An index is built; the question retrieves the k most similar fragments; only those fragments enter the prompt.
2. CAG. The whole corpus is placed in the context window on every call.
3. Agentic retrieval. The model is given search tools and decides for itself what to read, paying a round trip per decision.

The choice is normally argued on cost and context-window limits. This study asks whether there is also an accuracy criterion, specifically for the failure mode that matters most in a document base with history: an answer that is fluent, on-topic, sourced — and describes a state of the world that a later document has already overturned.

1.2 Related work

The comparison itself is not new. What is new here is the corpus it is run on.

Retrieval versus long context. Li et al. [1] benchmarked retrieval against long-context models across public question-answering datasets and found that long-context models win on average while retrieval remains far cheaper, proposing a router between them. Lee et al. [2] built a benchmark of real-world tasks at up to a million tokens and reported that long-context models rival dedicated retrieval pipelines. Yu et al. [3] argued the opposite case, showing that very long contexts dilute attention to the relevant passage, and reported an inverted-U curve in which answer quality first rises and then falls as more chunks are supplied — a shape this study looked for and, on a corpus this small, did not find. Hsieh et al. [4] showed that models claiming large context windows degrade sharply on tasks harder than needle retrieval, and Liu et al. [5] established the positional effect that underlies much of this: accuracy is highest when the relevant passage sits at the beginning or end of the context and drops in the middle. Asai et al. [6] represent the adaptive-retrieval line that the agentic arm here stands in for.

Conflicting and time-sensitive knowledge. Xu et al. [7] survey knowledge conflicts and distinguish context-memory, inter-context and intra-memory conflict; the supersession category of this study is squarely an inter-context conflict, in which two passages of the supplied context disagree and recency decides. Vu et al. [8] introduced a dynamic benchmark of fast-changing facts and false-premise questions, which is the closest published analogue to the supersession and negative-trap categories used here — but it tests a model's parametric knowledge against the live web, whereas this study holds all evidence inside the supplied documents.

Contamination. Xu et al. [9] survey benchmark data contamination: evaluation material absorbed into training data inflates measured performance in ways that are difficult to detect after the fact.

What distinguishes this study. Every comparison cited above is run on public material — released benchmarks, published corpora, or the live web. That is unavoidable for work that must be replicable by others, and it carries a known cost: a public benchmark may be inside the training data of the models it is used to evaluate, and a synthetic benchmark becomes public the moment it is released. This study instead uses a corpus written for it and never published, in which every organisation, system, role name, identifier and date is invented and the consolidated final state appears in no document. Two checks support the assumption, and neither proves it: the corpus's distinctive entity names return no results on the public web, and the no-context control scored 0/20 on both models, with refusals rather than guesses (§4.2, §4.3). Non-membership in a training set cannot be demonstrated from the outside; what can be done is to make contamination implausible by construction and then measure it empirically with a control arm, which is the protocol the contamination literature recommends and which most published comparisons of these architectures do not report.

The trade is deliberate and its cost should be stated plainly: this corpus cannot be redistributed without destroying the property that makes it useful, so the results here are not independently reproducible in the way results on a public benchmark are. What is reproducible is the method — the construction rules in §2.1, the category scheme in §2.2 and the safeguards in §2.5 — which is why they are given in full.

1.3 Research questions

• RQ1. Do the three architectures differ in accuracy on questions whose correct answer requires recognising that an earlier document has been superseded?
• RQ2. Do they differ on questions requiring aggregation over the whole corpus, on lexical paraphrase, and on negative traps — questions presupposing something that was rejected?
• RQ3. What does each architecture cost in prompt tokens, completion tokens and latency for the same answer?
• RQ4. Is any observed behaviour a property of the architecture, or of the instruction given in the system prompt?
• RQ5. Do the results yield operational criteria for choosing an architecture, and is a project-specific proof of concept a rational expenditure?

1.4 Design assumptions

These were fixed by construction and constrain the interpretation of every result below.

---

2. Materials and methods

2.1 Corpus

The corpus describes a fictional customer-care system procured by a fictional insurer. It consists of one base requirements document and eleven sets of steering-committee minutes dated across five months, in chronological order: twelve documents, 11,537 tokens, 42 chunks at 400 tokens with 50 tokens of overlap.

It was written so that the answer to a question is frequently not the value stated in the document that most obviously matches the question's wording:

• Amendment. The deployment region changes from one cloud region to another.
• Reversal and re-reversal. Maximum session length goes 8 h → 12 h → 8 h across three documents.
• Abolition and reinstatement in a narrowed scope. An auditor role is abolished, then reinstated without one of its original permissions.
• Creation and removal without implementation. A retention role is created and later removed.
• Rejection. Three requested features are refused; the minutes recording a refusal are lexically very close to those recording an approval.

A hardened variant was later built containing the same facts under three adversarial transformations: filenames carry no dates or ordinals and sort non-chronologically; cross-references cite dates rather than document numbers; and each superseded value is requoted verbatim, with stated provenance, inside a later document's appendix, so that "the most recent document mentioning X" is deliberately the wrong heuristic. For one value the quotation sits in the newest document of the corpus. The transformation adds no false statement: a June document quoting the January baseline is quoting it accurately.

2.2 Question set

Twenty questions in five categories, each with a reference answer and a list of gold documents naming what must be present for the question to be answerable at all.

Separating gold documents from the reference answer is what makes "the retriever did not find it" distinguishable from "the model had it and still got it wrong".

2.3 Experimental arms

2.4 Metrics

• Correctness — binary, graded per question and arm.
• Stale — the answer states a value that a later document superseded. Graded independently of correctness, because a stale answer is a different failure from a refusal.
• Invented — the answer describes something rejected, removed or non-existent.
• Retriever recall — retrieved ∩ gold ÷ gold, at document level, computed automatically for the retrieval and agentic arms.
• Cost — provider-reported prompt and completion tokens. For the agentic arm these are summed across all turns of one answer, which is the honest accounting: a run that reads four files pays for them again in every subsequent turn.
• Latency — total wall-clock per answer, and time to first token for the streaming arms; the agentic loop is non-streaming and records none.

2.5 Validity safeguards

1. Context probe. Before each run a canary token is planted at the top of the full-corpus context and the model is asked to echo it. If it cannot, the window was silently truncated and every full-context number that follows is meaningless. The probe also compares the provider's reported prompt-token count against a local estimate and warns on a large discrepancy. All reported runs passed.
2. Caching disabled at the provider, verified per call.
3. Determinism check. Byte-identical answers across repetitions are flagged as the signature of an undisabled cache. Observed: 0/120 identical question-arm pairs for the stronger model and 1/120 for the smaller one — real sampling variance.
4. Control arm run first. Any question it answered correctly would have been dropped. None were.
5. Full prompt and tool-trace logging, so that a wrong answer can be attributed rather than guessed at.

2.6 Grading protocol

One row per unique question and arm — the first repetition — graded against the reference answer and the source documents, scoring correctness, staleness and invention independently, with a free-text note recording the reason for every zero. Repetitions are used for latency, cost and variance, not for accuracy; each accuracy cell therefore rests on 20 questions, or 7 in the supersession-only runs. One exception is documented in §4.5.

---

3. Overview of runs

Total 1,527 logged calls.

---

4. Results

4.1 Run 0 — the pilot, and a methodological failure worth reporting

The pilot ran with an output-token ceiling of 300. On a model with adaptive reasoning, the visible answer and the reasoning share that budget. The result was empty answers on exactly the hardest questions: the aggregation question about the role model at k=10 and k=20, the counting question at k=5, the superseded-region question at k=5, the paraphrase question at k=5.

Read naively, this table says retrieval at k=5 fails a supersession question — the result the study was designed to find. It does not: the model produced no text at all. A truncation-shaped artefact is indistinguishable from an architecture failure in any metric that inspects only the answer string. The ceiling was raised to 2,000 tokens and every subsequent run re-executed. Twelve of the graded zeros across the pilot's arms were empty responses rather than wrong answers.

4.2 Run 1 — stronger model, original corpus

Accuracy (20 questions per arm):

Cost and latency (means over 60 calls per arm; milliseconds):

¹ summed across all turns of one answer. ² the tool loop is non-streaming.

Retriever recall by category:

Agentic behaviour: 4.6 tool calls per question (min 3, max 21); 60 file listings, 121 searches, 95 reads; 23 of 60 runs reformulated the search after an unproductive query. No run hit the turn cap.

Findings.

1. The control is clean. 0/20 without documents, and the model refuses rather than guesses. No question is contaminated by pretraining, so the whole set is admissible.
2. The supersession hypothesis fails at the first test. Every arm — including the cheapest, at 1.8k tokens of context, three chunks out of forty-two — answers all seven supersession questions correctly with zero stale answers. Three well-chosen chunks sufficed for the model to notice that the baseline value had been overturned, because a lexical retriever asked about a value that changed tends to retrieve both the original and the amendment: they share vocabulary.
3. The real retrieval deficit is aggregation. All three remaining errors are aggregation questions. The counting question requires a negative fact about every document; its recall never reaches 1.00 and even at k=20 the model names four of the five qualifying meetings. This is not a tuning failure — a top-k retriever cannot answer a question whose answer is a property of the set.
4. k has a real effect on paraphrase. Recall 0.50 → 1.00 between k=5 and k=10, with accuracy following. The average recall at k=5 across all categories is 0.81, which hides this entirely: only the per-category breakdown makes it actionable.
5. Cost is monotone and steep. Full context costs 6.2× retrieval at k=5 in prompt tokens for three additional correct answers out of twenty. The marginal cost is ~546 prompt tokens per retrieved chunk, so retrieval reaches full-context cost at k ≈ 33 of the 42 available chunks, beyond which retrieval buys nothing but complexity.
6. Agentic retrieval is not cheaper in time. Median 13.4 s against 2.9 s for full context, p95 30.4 s, and 4.3× the completion tokens — but it is cheaper in prompt tokens than full context on this corpus, and unlike full context that figure barely grows with corpus size.

An instrumentation caveat. The agentic arm's recall figures in this run (mean 0.71, as low as 0.33 in one category) are an artefact, not a behavioural finding. Only documents opened with the read tool were credited; documents whose content reached the context as search output were not. One trace is explicit: the model searched for a threshold, received the matching line, answered correctly from it, and was scored recall 0.00 for that question. The crediting rule was corrected before Run 2 — search hits now count, file listings deliberately do not, since crediting them would confer perfect recall for free — which makes agentic recall not comparable between Run 1 and later runs. Accuracy is unaffected.

4.3 Run 2 — smaller model, original corpus

Agentic behaviour: 6.1 tool calls per question, 155 searches, 151 reads, 32 of 60 runs reformulated; recall 1.00 in four categories and 0.94 in aggregation under the corrected crediting rule.

Findings.

1. The control now shows the failure mode the grounded arms do not. Without documents the smaller model does not refuse — it confabulates: it asserts a capability that had been removed from scope (graded stale), invents an all-day session with no re-authentication, guesses a plausible primary key, and produces a complete step-by-step configuration guide for an integration that was explicitly rejected. Every grounded arm, including k=5, suppressed all four. This is the clearest single result in the study: the value of grounding shows up not as accuracy on the grounded arms but as the distance between them and the ungrounded one — and that distance is model-dependent.
2. The architecture ranking reproduces; the margins shrink. Same shape, same failures concentrated in aggregation, same perfect scores on supersession everywhere.
3. Full context is not uniformly best. The smaller model's single full-context error is a simple fact question: given all twelve documents it declared that no latency requirement exists for a named integration, when the baseline document states one for every synchronous integration. Retrieval at k=5 answered it correctly. The model had the value and argued its way out of it by seizing on a peripheral detail about the integration's protocol. A longer prompt does not merely add information; it adds opportunities to distinguish.
4. Architecture choice and model choice are comparable in magnitude. The spread between cheapest and best arm is two answers for each model, while the spread between models within an arm is at most one — and both are dwarfed by the gap between any grounded arm and the control.
5. A measurement caveat. The same 11,743-token context is reported as 18,081 prompt tokens for one model and 13,226 for the other, a 37% difference in tokenisation. Token counts are comparable across arms within a model and not across models; cost comparisons between models must be made in currency.

4.4 Run 3 — prompt ablation

The system prompt used so far contained an explicit strategy hint: that later minutes may amend or reverse earlier decisions, and that the model should always answer with the state as of the most recent document. If the supersession result was won by the prompt rather than by the architectures, removing that sentence should produce stale answers. The ablation removed exactly it and nothing else: the factual description of the corpus remained, as did the sentence about rejected items, so that the negative-trap category was held constant and the design had a single variable.

Result: 7/7 on supersession in every arm of both models, with zero stale and zero invented answers. Agentic latency rose and agentic prompt cost fell, but accuracy did not move at all.

The supersession behaviour is therefore not carried by the instruction. Given documents containing both the original and the amending statement, both models resolve the conflict in favour of the later one without being told to. This closes off the most plausible alternative explanation for the null result in §4.2 and makes it substantially stronger.

4.5 Run 4 — hardened corpus

Accuracy on supersession (7 questions per arm, 210 calls, no errors):

Both failures of the smaller model are the same question — whether a role exists, after it was abolished in one document and reinstated in a narrowed scope in a much later one — both graded stale, and both in the widest-context arms. Retriever recall for supersession remained 1.00 at every k: anonymising filenames does not hinder a lexical index, which reads content.

The two failures have different shapes, and the second is the more interesting:

• Agentic. "Based on the most recent steering committee meeting (dated [February]), the role does not exist — it was abolished." The model named a February document as the most recent. Without dates in filenames it mis-ordered the corpus, stopped at the abolition, and never reached the reinstatement.
• Full context. "No, the role does not exist in the current requirements. It was abolished in the meeting of [February]. However, it was subsequently reinstated in a narrowed scope in the meeting of [June]…" Here the model had both facts, stated both correctly, and still opened with the negation. This is neither a retrieval failure nor a chronology failure but a failure to collapse two retrieved statements into one current state — and it would be invisible to any metric scoring the answer's contents rather than its claim.

The failure replicates across every repetition. Because this is the study's only discriminating result, all three repetitions of that question were read for both models rather than the single graded one:

Every wide-context answer of the smaller model opens by denying the role's existence; every retrieval answer opens by affirming it. Fisher's exact test on 0/6 versus 9/9 gives p = 0.0002 one-sided.

A second degradation: the reasoning path decays before the answer does. Two automatic measurements over the same 210 answers.

Mis-identified recency. Scanning for claims of the form "the most recent document is X" and checking the named document against the true dates: the stronger model made 3 such claims, 1 wrong; the smaller model 7, 3 wrong. Four answers asserted that a February document was the most recent, and three of those four still gave the correct final answer.

Fabricated dates. Every date string in every answer, checked against the twelve dates the corpus contains:

The smaller model's inventions cluster plausibly around the real meeting dates, and appear in correct answers as well: it affirms that the role exists and attributes the reinstatement to a meeting that never took place.

On the original corpus neither error could arise — the filename carried the date, so recency was not a judgement and a citation was not a recollection. The hardening did not cause these failures; it created the ability to observe them.

Findings.

1. Chronological cues in filenames were not the mechanism behind the null result, for the stronger model, which scored 35/35 on a corpus built specifically to remove them.
2. The null result is not universal. It held across four datasets and broke on the fifth, in the model with less capability, on the question whose correct answer is a two-step history rather than a substituted value.
3. The failures landed in the widest-context arms, in every repetition. Retrieval handed the model the two decisive chunks and nothing else; full context and the agentic loop handed it twelve documents in a scrambled order and left the ordering work to it. A narrow context is not merely cheaper for a weaker model — it is a guardrail, because it removes the opportunity to mis-order the corpus.
4. The question that broke asks about existence, not value. The four questions whose answer is a superseded value were answered correctly by every arm of both models on every corpus. The one that failed asks whether something exists after an abolition and a narrowed reinstatement: a "yes" that must survive an intervening "no". Question sets probing supersession should weight this shape accordingly.

4.6 Language variant

A translated corpus and question set were prepared and verified but not executed. The translated corpus produces 48% more chunks for the same content, which is a cost consideration in itself; no accuracy claim about non-English corpora follows from this study.

---

5. Synthesis

Two results concern the instrument rather than the architectures and are worth as much as the rest. An output-token ceiling that starves the reasoning budget produces empty answers precisely on the hardest items and is indistinguishable from an architecture failure. A change to the recall-crediting rule between two runs silently made those runs' recall figures incomparable. Both were detectable only because full prompts and complete tool traces were logged.

---

6. Selection criteria

6.1 Criteria supported by this evidence

Choose full context when the corpus fits the window with substantial headroom and query volume is low enough that a 6× prompt-token premium per question is affordable; when questions require aggregation, counting or enumeration over the whole set; or when engineering effort is the binding constraint — there is no index, no chunking policy, no k to tune and no retrieval failure mode to monitor.

Choose retrieval when the corpus exceeds the window or cost per question dominates; when questions are localised facts, including superseded ones; and provided k is chosen from per-category recall rather than from an average. The operative criterion is not "does recall look good" but "is there a question class whose recall is below 1".

Choose agentic search when the corpus is far larger than the window, so that neither full context nor a fixed k is available; when the retrieval strategy must adapt to what is found — 38–53% of runs here reformulated a query after an unproductive search, which a single-shot retriever cannot do; and when latency is not user-facing.

Match context width to model capability. More context is not monotonically safer. Where the document base makes chronology or precedence hard to establish, handing a weaker model the whole set gives it the opportunity to order it wrongly; handing it the two decisive chunks does not. This reverses the usual reasoning, which treats full context as the safe default and retrieval as the risky optimisation.

Independent of architecture: ground the model in something. The largest effect measured here is the difference between the ungrounded control and every grounded arm.

6.2 How clear are the criteria?

Clear. Those following from corpus size versus window, from question shape — a localised fact versus a property of the set — and from cost and latency are sharp, quantitative and transferable. They can be evaluated before running anything: corpus token count, context window, expected query volume and a sample of question types suffice.

Not clear, and originally assumed decisive. "Use full context because retrieval serves stale answers" is not supported, and on the hardened corpus the only stale answers observed came from full context. The criterion survives in two weakened forms: if retrieval returns the superseded statement without the amending one the model has no way to know, so the quantity to measure is recall of amending documents rather than the model's stale-answer rate; and if the corpus does not make its own chronology legible, the risk shifts to whichever arm carries the most unordered material.

Ambiguous. Full context versus retrieval at the margin. The evidence is 20/20 against 19/20 and 19/20 against 18/20 — inside the noise at twenty questions. Decisions at that margin should be made on cost, latency and operational complexity.

---

7. Is a project-specific proof of concept justified?

Cost. This study is 1,527 model calls on a twelve-document corpus and a few hours of compute — dominated entirely by the human cost of writing a corpus with a deliberately hidden final state, twenty questions across five categories, and an answer key. That artefact, not the harness, is the expensive part.

Benefit, as realised. The study produced one actionable configuration change (k=5 → k=10, justified by per-category recall rather than by accuracy); the refutation of the hypothesis that would otherwise have justified paying 6× per question for full context; identification of the question class that genuinely requires whole-corpus access; a counter-intuitive interaction between context width and model capability; and two instrumentation traps that would have silently corrupted a less careful comparison. Only the first would have been obtained by an informal "try a few questions and see".

Assessment. A full proof of concept every time is not warranted; a screening one almost always is. A graduated protocol:

Three conditions decide whether it pays for itself. The questions must be able to discriminate — four of five categories saturated here, and a study whose arms all score ≥ 17/20 has told you only that the corpus is easy, which is a result about the corpus rather than the architectures. The control arm is mandatory — without it you cannot separate grounding from memorisation, and it is where the failure modes you are trying to prevent actually appear. The reusable artefact is the question set and answer key, not the accuracy table: those transfer to the next model release, the next retriever and the next prompt; the numbers do not.

---

8. Limitations

Construct validity. The corpus and questions were written by the same author, so the questions may test the corpus's own vocabulary; the paraphrase category counters this but has only three items. "Correct" is a binary judgement on a two-sentence free-text answer, and several graded zeros are partial answers — six of seven roles named, the right count with the wrong set — which a graded scale would capture. Document-level recall over-credits: a document counts as retrieved if any of its chunks is, even when the relevant chunk is not.

Internal validity. The agentic recall crediting rule changed between two runs, making those runs' recall figures incomparable; accuracy is unaffected. Grading was performed once, by a single grader, without blinding to the arm, and no inter-rater agreement was measured. Only the first repetition of each question and arm was graded, so the accuracy tables ignore answer variance, which is demonstrably non-zero — the one exception being the hardened corpus's decisive question, where all repetitions were read and the failure was total. The recency metric counts only claims phrased in a particular way and is a lower bound; the date-fabrication metric has no such weakness but cannot distinguish a hallucinated citation from a typo.

Statistical conclusion validity. Twenty questions per arm, seven in the supersession-only runs. Differences of one or two answers — which is every difference between full context and the better retrieval arms — are not meaningful, and no significance testing was performed except on the one result that warranted it. Four of five categories saturate in nearly every arm.

Contamination. The claim that no part of the corpus was available during training is an argument, not a measurement: it rests on the corpus being invented and unpublished, on its distinctive entity names returning nothing on the public web, and on the no-context control scoring 0/20 with refusals. The control demonstrates that the answers were not memorised; it cannot exclude the presence of similar material, and no external party can verify the corpus's provenance without being given the corpus, which would end the property being claimed.

External validity. One synthetic corpus, one domain, ~11.5k tokens. The regime in which retrieval genuinely exists — a corpus that does not fit the window — is precisely the regime not tested. Two models from one provider family. One lexical retriever: no embeddings, no hybrid search, no reranker, no recency weighting. English only. Cost is reported in tokens rather than currency, and no prompt caching was used, which would materially change the economics of repeated queries over a fixed corpus. The hardened-corpus result covers one question on one model.

---

9. Extensions

Ordered by expected information gain per unit of effort.

1. A corpus that does not fit the context window — 500–5,000 documents, 1–10M tokens — at which point full context is not an option and the comparison becomes retrieval versus agentic search versus hierarchical summarisation. It should contain distractors: documents about adjacent projects sharing vocabulary and entity names, which this corpus lacks and which are what actually break lexical retrieval at scale.
2. Generalise the hardened-corpus failure beyond one question. Write three or four more existence-history questions — a thing abolished then reinstated, created then withdrawn then restored — since the value-substitution questions proved unable to discriminate; add a third model of intermediate capability to locate the threshold; then extend the hardening to the paraphrase and aggregation categories.
3. Make provenance a first-class metric. Checking every cited date and document identifier against the corpus separated the two models 18% to 0% with no human grading and no extra calls.
4. Enlarge and rebalance the question set — 200+ questions, authored by someone other than the corpus author, with saturating categories trimmed and discriminating ones expanded. This is the only route to statistical power.
5. A retriever ladder: lexical → dense → hybrid → hybrid with reranking → recency-weighted, each measured per category. The specific question worth answering is whether a dense retriever fixes aggregation, or whether that class is structurally out of reach for any top-k method.
6. Turn the ablation into a factorial design. The chronology hint has been ablated; the rejected-items sentence, the document ordering in the full-context prompt, and the presence of document identifiers should be too. Reverse-chronological ordering of the full-context prompt is a cheap and plausible improvement that has not been tested.
7. Multiple graders, or a calibrated model judge validated against them, which also makes it feasible to grade all repetitions rather than one.
8. Prompt caching for the full-context arm. With a fixed corpus and a warm cache its cost profile changes qualitatively and time-to-first-token becomes its strongest argument.
9. Concurrency and cost in currency. All latencies here are single-threaded; the agentic arm's sequential round trips behave differently under load.

---

10. Conclusions

1. On a twelve-document corpus of superseding decisions, every grounded architecture answered every supersession question correctly — on two models, with and without a prompt instruction to prefer the most recent document, across 340 graded answers, with zero stale answers. The hypothesis that retrieval yields confidently stale answers where full context does not was not supported.
2. The hypothesis inverted where it finally broke. On a corpus rebuilt to remove chronological cues and to requote superseded values inside later documents, the smaller model produced its first stale answers of the entire study — in full context and in agentic search, while all three retrieval arms stayed correct. If context width affects factual currency, the evidence points to wider is riskier for a weaker model, not safer.
3. The measurable differences are structural. Retrieval cannot answer questions whose answer is a property of the whole set, and fails on lexical paraphrase at small k — the latter fixed by k=10, the former not fixed by any k tested.
4. The largest effect is grounding itself, visible only against the no-context control and mainly on the smaller model.
5. Costs are clear and monotone where accuracy is not: retrieval at k=5 costs 16% of full context's prompt tokens, the two converge at k ≈ 33 of 42 chunks, and agentic search costs 4.6× the latency of full context while using fewer prompt tokens on a corpus this small.
6. Architecture selection should be decided on corpus size, question shape, cost and latency — all knowable in advance — rather than on an expected difference in factual currency, which this study could detect only once, under adversarial conditions, and in the direction opposite to the one assumed.
7. Correctness and provenance degrade separately, and provenance degrades first. Under the hardened corpus the smaller model cited fourteen non-existent dates and misidentified the most recent document three times, mostly while answering correctly. Any evaluation grading only the final claim will miss this; any product showing the user a citation is exposed to it; and unlike correctness it can be checked automatically.
8. A screening proof of concept is worth running; a full one usually is not. Its durable output is the corpus, the question set and the answer key, which survive the next model release; the accuracy table does not.

---

11. How this study should be used

It is an indicator of which effects to watch for when choosing an architecture, not a ranking to transfer. Its scope makes that unavoidable: twenty questions on one synthetic corpus of twelve documents, in the one regime where the architectural question answers itself, two models from one provider family, one retriever, English only, and four of five question categories saturating in nearly every arm. Differences of one or two answers carry no statistical weight at this size. Read as a league table, these results would say "full context wins by a point", which is precisely the claim the evidence cannot support.

What transfers is the list of effects the study made visible, each worth instrumenting in a real evaluation: aggregation as the structural limit of top-k; per-category recall rather than the mean; the grounding gap, visible only against a no-context control; the non-monotonic interaction of context width with model capability; provenance degrading before correctness, and checkable automatically; and instrument artefacts that imitate findings.

The one discriminating result — the smaller model failing an existence-history question in the wide-context arms while every retrieval arm held — is solid within its scope and unreplicated outside it. Before it is used to justify an architectural decision it needs the extension named in §9: more questions of that shape, all repetitions graded, and a third model of intermediate capability. Until then it is a lead: a well-supported one, and the most interesting thing this study produced, rather than a result to design around.

---

References

[1] Z. Li, C. Li, M. Zhang, Q. Mei, M. Bendersky. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. arXiv:2407.16833, 2024.

[2] J. Lee et al. Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? arXiv:2406.13121, 2024. (LOFT benchmark.)

[3] T. Yu, A. Xu, R. Akkiraju. In Defense of RAG in the Era of Long-Context Language Models. arXiv:2409.01666, 2024. (Order-preserving RAG; inverted-U curve.)

[4] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, B. Ginsburg. RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654, 2024; COLM 2024.

[5] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172, 2023; TACL 2024.

[6] A. Asai, Z. Wu, Y. Wang, A. Sil, H. Hajishirzi. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511, 2023; ICLR 2024.

[7] R. Xu, Z. Qi, Z. Guo, C. Wang, H. Wang, Y. Zhang, W. Xu. Knowledge Conflicts for LLMs: A Survey. arXiv:2403.08319, 2024; EMNLP 2024.

[8] T. Vu, M. Iyyer, X. Wang, N. Constant, J. Wei, J. Wei, C. Tar, Y.-H. Sung, D. Zhou, Q. Le, T. Luong. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. arXiv:2310.03214, 2023; Findings of ACL 2024. (FreshQA.)

[9] C. Xu et al. Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244, 2024.