What one summarisation pass destroys: compressing a corpus before answering from it, measured against a known answer key

When a document base outgrows every context window, the standard remedy is to summarise it once and answer from the summaries. This ablation measures what that single pass costs against a hidden answer key — and finds that one sentence added to the summariser's instruction moves accuracy further than the entire spread between grounding architectures measured on the same corpus in the companion study.

---

Abstract

When a document base exceeds any available context window, the standard remedy is to compress it — summarise each document once, then answer every question from the summaries. The cost profile is attractive and the risk is rarely quantified: a summary is a lossy encoding chosen without knowing the question. This study measures what a single compression pass destroys, on a twelve-document corpus (~11.5k tokens) of software requirements that are repeatedly amended, reversed and rejected, and for which a hidden answer key exists. Two summarisation styles — question-agnostic and decision-preserving — were run on two models of different capability over twenty questions in three repetitions: 240 answers plus 48 one-off summarisation calls.

Compression to 30–34% of the original text preserved almost every fact on which a reference answer depends, and a decision-preserving summary matched whole-corpus context exactly on the stronger model — 20/20 correct, at 29% of the prompt tokens. One sentence of instruction to the summariser moved accuracy from 15/20 to 20/20, a larger movement than the entire spread between grounding architectures measured on the same corpus in a companion study. Two results run against expectation. Compression inverts the difficulty ordering of the question set: simple verbatim facts, answered correctly by every architecture in the companion study, collapse to 1/3 in three of four summary arms, while supersession — the corpus's designed difficulty — holds at 7/7 under the decision-preserving style. And the degradation is entirely into refusal: zero stale and zero invented answers across 80 graded answers, on a corpus purpose-built to elicit both. A generic summary was found to drop an integration's name and its numeric threshold while faithfully preserving the narrative of what the meeting decided.

Keywords: summarisation, context compression, retrieval-augmented generation, information loss, benchmark design, evaluation methodology.

---

1. Introduction

1.1 Problem

Hierarchical map-reduce summarisation is what remains when a corpus fits neither a context window nor a fixed retrieval budget. Its economics are appealing: pay once per document, then serve every question from a fraction of the tokens. Its risk is obvious in the abstract and seldom measured in practice, because measuring it requires a corpus whose correct answers are known independently of the compressed text.

That is the condition this study creates. The corpus was built so that answers are values changed two or three times and roles abolished and then reinstated in a narrowed scope — precisely the material a summariser flattens into "session parameters were settled" and "the role model was refined". Because a hidden answer key exists, loss is visible against a reference rather than merely suspected.

The study deliberately does not treat summarisation as an architecture candidate. On a corpus occupying 6% of the context window, compressing what already fits can only lose; a defeat under those conditions would say nothing about the regime where summarisation is actually deployed. It is run here as an ablation, to establish what one compression level costs before anyone reasons about several.

1.2 Research questions

• RQ1. Does one summarisation pass preserve the facts on which the reference answers depend?
• RQ2. Does fact retention in the summary predict answer accuracy, or do the two diverge?
• RQ3. How much of any loss is attributable to the summariser's instruction rather than to compression itself?
• RQ4. When compression loses a fact, does the model refuse or fabricate?

1.3 Design assumptions

---

2. Materials and methods

2.1 Corpus and questions

Twelve documents — one baseline requirements specification and eleven sets of steering-committee minutes spanning five months — totalling 11,537 tokens. The consolidated final state of the requirements appears in no single document and must be derived from the chronology.

Twenty questions in five categories, each with a reference answer:

2.2 Summarisation styles

The generic style is the realistic condition — a pipeline summarises before anyone has asked anything. The decision-preserving style is the upper bound available from prompting alone.

2.3 Arms

Each arm supplies the twelve summaries of one style, concatenated, as the entire context. The answering system prompt is the one used for whole-corpus context in the companion study, unchanged. Reference points are taken from that study rather than re-run.

2.4 Metrics

Correctness, staleness, invention, prompt and completion tokens and latency, as in the companion study, plus one metric introduced here:

Fact retention. For each of ten facts on which a reference answer depends, whether the fact appears in the summary text at all. Computed on the summaries, before any question is asked, and therefore independent of the answering model's behaviour. It is the earliest available warning that a compression pipeline has broken, and it costs nothing.

Retrieval recall is undefined for these arms: every document is present in compressed form, so coverage is 1.00 by construction and the quantity of interest is fidelity rather than coverage.

2.5 Validity safeguards

1. Context probe before each run: a canary token planted at the top of the context and echoed back, to rule out silent truncation.
2. Provider caching disabled, verified per call.
3. Summary caching, so that all three repetitions of a question read a byte-identical context.
4. Output budget headroom of 2,000 tokens, after an earlier study on this corpus established that a low ceiling starves the reasoning budget and produces empty answers on the hardest items.
5. A false-negative check on fact retention. The first pass reported two facts as lost that were present under different wording — a parenthetical inside a phrase defeated the pattern. Every reported loss was confirmed by reading the summary before being counted.
6. Determinism check. No question-arm pair produced a byte-identical answer across all three repetitions in any arm, confirming real sampling variance rather than a cache.

2.6 Grading protocol

One row per question and arm — the first repetition — graded against the reference answer and the source documents, scoring correctness, staleness and invention independently. Eighty graded answers.

---

3. Overview of runs

288 calls in total, no errors, no empty answers.

---

4. Results

4.1 Compression achieved and build cost

The build is a one-off cost of twelve calls, amortised over every question thereafter.

4.2 Fact retention, measured on the summaries

Ten facts on which reference answers depend, checked in the summary text before any question was asked:

One decisive fact lost per model under the generic style; none under the decision-preserving style. On this measure, loss is a prompting problem rather than an inherent cost of compression.

Identifiers fare worse than decisions. The same check over the names and numbers by which a reader would query the corpus:

A generic summary retains the narrative of what a meeting decided and discards the name of the thing the decision was about.

4.3 Accuracy

4.4 Failure modes

All seventeen graded failures are refusals or defensible partials. Three trace the causal chain from a missing fact to an answer.

• Fact lost, model refuses. The stronger model's generic summary dropped the naming convention; all three repetitions answer that the documents do not specify one, noting only that creating the groups was assigned as a follow-up action. Correct behaviour on incorrect input.
• Fact lost, precision lost, truth kept. The smaller model's generic summary dropped the primary key's field name; the answer describes it as a universally unique identifier assigned by the upstream data hub — the right concept without the identifier the reference names.
• Name lost, question becomes unanswerable. The stronger model's generic summary contains neither the vendor name nor the threshold; the answer states that no system by that name is mentioned. The threshold survives in the corpus and is unreachable through the compression.

4.5 Cost and latency

Summarisation costs 29–41% of full context's prompt tokens and is slower than full context despite the shorter prompt. The most plausible reading is that a dense, compressed context requires more reasoning per answer than a verbose one, the model having less redundancy to work with; this study cannot separate that explanation from ordinary variance, given a p95 of 19 s against a 5 s median.

---

5. Synthesis

---

6. Selection criteria

Summarise-then-answer is worth considering when the corpus cannot fit the window, query volume is high enough to amortise the build, and questions concern what was decided rather than named entities and numeric thresholds.

It should be avoided, or paired with retrieval, when questions cite identifiers — product names, system identifiers, thresholds — because those are what compression discards first. A hybrid in which summaries provide orientation and a retrieval pass supplies verbatim detail follows directly from the retention results and was not tested here.

Whichever is chosen, the compression prompt is a first-class design decision. It moved five answers out of twenty on the stronger model. It should be version-controlled, reviewed and regression-tested like any other component, and it must name the entity classes that will later be queried.

Instrument fact retention. Checking whether a set of known-critical strings survives compression is automatic, requires no grader and no answering calls, and would have predicted every failure in this study. In a production pipeline it is the cheapest available canary.

---

7. Limitations

• One compression level. Hierarchical summarisation compounds loss across levels; this measures the first level only, and the result should not be extrapolated to a pipeline that summarises summaries.
• Summariser and answerer are the same model. The common production shape — a cheap summariser feeding a stronger answerer — is untested and, on this evidence, would inherit the weaker model's retention rather than the stronger model's reading.
• Ten hand-chosen facts. The retention metric covers the facts the reference answers depend on; it is not a general measure of fidelity, and the accuracy results show failures beyond its coverage.
• Twenty questions, one graded repetition per question and arm. The five-answer gaps are outside plausible noise; the two-answer gap between the smaller model's styles is not.
• Summaries built once. Real pipelines re-summarise as documents change; drift across rebuilds is not measured.
• One synthetic corpus, one domain, English only, and a corpus that comfortably fits the window — the regime this ablation exists to inform is the one it cannot directly test.

---

8. Extensions

1. A hybrid arm: summaries for orientation plus a retrieval pass for verbatim detail. The two failure profiles are complementary, which is the strongest available argument that the combination beats either alone.
2. Cheap summariser, strong answerer — the production shape — to establish whether retention is a property of the summariser alone.
3. Hierarchical summarisation on a corpus that genuinely exceeds the window, measuring retention level by level: the regime this study exists to inform.
4. Retention as a continuous metric over an automatically extracted set of entities and numerals rather than ten hand-chosen facts, so that it can run unsupervised in a pipeline.
5. A compression-ratio sweep. Everything here sits at ~30% of the original; the curve between 10% and 60% is where an operating point would be chosen.

---

9. Conclusions

1. Compression was survivable, and cheaper than expected. A decision-preserving summary at 34% of the original text matched whole-corpus context exactly on the stronger model, for 29% of the prompt tokens.
2. The summarisation prompt is the pipeline's principal design decision. One sentence of instruction moved five answers out of twenty — more than the entire architectural spread measured on the same corpus — and it is the parameter that comparisons of retrieval against full context never vary.
3. Compression destroys identifiers before it destroys decisions. The narrative of what a meeting resolved survives; the name of the system and its threshold do not. Question sets testing only "what was decided" will not detect this.
4. Difficulty is a property of the pipeline, not of the question. The category every architecture found trivial is the one summarisation fails, and the category the corpus was built to make hard is the one it handles. A benchmark that fixes the pipeline cannot see this.
5. The failure mode is benign. Zero stale and zero invented answers in eighty graded answers on a corpus built to elicit both: when the fact is gone, these models say so. That property is what makes a lossy pipeline tolerable in production, and it should be verified rather than assumed for any model placed in this position.