14  Summarization

Chapter status: drafted. Every number on this page was measured on XL-Sum Turkish (Hasan et al. 2021) — 27,176 training and 3,397 test BBC Türkçe articles with journalist-written one-sentence summaries. The extractive systems are scored on all 3,397; the LLM on a 500-article sample, stated where it appears. Section 14.4 reports a tokenizer defect that changes every Turkish ROUGE number computed with the stock tokenizer. So the ROUGE used here is written out rather than imported, and it was validated against rouge-score to ten decimal places on text that tool can tokenize. Software: rouge-score 0.1.2, snowballstemmer 3.1.1, NumPy 2.5.2 on Python 3.14.5; the local LLM through ollama 0.24.0.

Learning objectives. After this chapter you can:

14.1 “Summarize this” is not one task

Before any modelling there is a specification problem, and it sinks more summarization projects than any modelling choice. The word names jobs that differ along at least four axes, and the person asking usually has not decided which:

Four axes, sixteen tasks
axis one end the other end
Audience intent indicative — help me decide whether to read this informative — replace reading it
Scope generic — what is this about? query-focused — what does this say about X?
Input single-document multi-document, where contradiction between sources becomes the hard part
Method extractive — select source sentences abstractive — generate new text

An indicative summary that is 80% accurate is fine; you were only deciding whether to open the document. An informative summary that is 80% accurate is a liability, because the reader is acting on it without checking. Those are different products with different acceptance criteria, and “summarize this” distinguishes neither.

The first question on a summarization project is therefore not which model but who reads the output, and what happens if it is wrong. Everything in this chapter — including which metric to report — follows from that answer.

A useful forcing question when the requirement is vague: “if the summary is wrong, would the reader find out?” If yes, you are building an indicative summary and can tolerate error. If no, you are building an informative one and Section 14.7 is the section that matters, not the leaderboard.

What we will look at. First the corpus, characterised before anything is modelled on it, because one number computed in two seconds decides whether selecting sentences can work here at all. Then the extractive baselines, and the ceiling sitting just above them. Then the metric everyone reports — which in Turkish is broken in a way that mostly goes unnoticed, and which in any language cannot see whether the summary is true. Then a model writing the summaries instead, and documents too long to hand it in one piece. And finally two ways of measuring truth, one crude and one expensive, that agree about the ranking and about almost nothing else.

14.2 Characterising the corpus before modelling it

Chapter 13 made auditing the corpus a habit rather than an accusation. Here the audit answers a question with immediate engineering consequences: can an extractive system work on this data at all?

# XL-Sum Turkish: BBC Türkçe articles, each with a one-sentence summary
# written by the journalist, not crowdsourced.
rows = [json.loads(l) for l in open("turkish_test.jsonl")]
train  27,176 articles   median 303 words, 17 sentences   summary 22 words
val     3,397 articles   median 300 words, 17 sentences   summary 22 words
test    3,397 articles   median 296 words, 17 sentences   summary 21 words

compression 14x        83.4% of test summaries are exactly one sentence

The leakage check that dominated Chapter 13 comes back clean here:

train articles 27,176, distinct texts 27,173      3 internal duplicates
test article also in train                        0    0.00%
test url also in train                            0
test summary also in train                       24    0.71%
duplicate texts inside test                       0

Zero article leakage. Worth saying plainly, because the habit is easily misread as cynicism: you audit so that you can trust the numbers, and sometimes the audit says you can. The 0.71% of repeated summaries are BBC boilerplate attached to different articles, small enough to ignore and large enough to mention.

How much of the summary is in the article?

This is the measurement that decides the architecture. Following Grusky et al. (2018), take each summary and greedily find the longest fragments of it that appear verbatim in the source:

coverage   0.527    share of summary words found anywhere in the article
density    1.05     sum of squared fragment lengths, over summary length
novel     47.3%     summary words that never appear in the article
verbatim   0/500    summaries that are a copied article sentence

Density 1.05 against coverage 0.527 puts the length-weighted mean extractive fragment at \(1.05 / 0.527 \approx\) two words. If every match were a lone word, density would equal coverage exactly; if the summary reused whole clauses it would run into the tens. Two words is the XSum regime: the odd noun phrase survives, the phrasing does not. For contrast, CNN / DailyMail — the corpus most summarization intuitions were formed on — has summaries built from bullet highlights that reuse the article’s own clauses. Running the identical measurement on 400 of its test articles makes the contrast concrete. Three column names come from later sections, so here is what they mean for now: lead-3 uses the article’s first three sentences as the summary; the oracle is the single article sentence that best matches the reference — a ceiling, explained in Section 14.3; and R-1 is ROUGE-1, the word-overlap score of Section 14.4, on a 0–100 scale where higher means more overlap.

The same code on two corpora, each with the tokenizer that is correct for its language, surface forms with no stemming on either side. The reference-length column is there because the single-sentence oracle is not comparable across corpora without it
corpus ref words coverage density lead-3 R-1 oracle R-1
CNN / DailyMail (English) ~56 0.845 — 29.44 40.58
XL-Sum Turkish ~21 0.527 1.05 14.03 24.89

XL-Sum Turkish is in the opposite regime, and the whole chapter follows from that. These are rewritten, in the style Narayan et al. (2018) called extreme summarization: one sentence, composed rather than selected.

The published lead-3 figure for CNN / DailyMail is 40.34 (See et al. 2017), higher than the 29.44 measured here. That number comes from the original Perl ROUGE-1.5.5, with its own options and its own preprocessing of the articles. Stemming alone usually moves English ROUGE-1 by about a point, not eleven, so much of the gap must come from preprocessing — including which three sentences each pipeline’s sentence splitter called “the first three”. The gap is a preview of Section 14.4’s argument: ROUGE is not one number but a family of them, and two papers reporting “ROUGE-1” are often not reporting the same quantity.

Where the summary’s content comes from

If the summary is not copied, is it at least drawn from the top of the article? Matching each summary against every article sentence by unigram F1 and taking the best:

best-matching article sentence, 500 test articles
  sentence  1:   79   15.8%  ###############################
  sentence  2:   54   10.8%  #####################
  sentence  3:   50   10.0%  ####################
  sentence  4:   38    7.6%  ###############
  sentence  5:   40    8.0%  ################
  sentence  6:   35    7.0%  ##############
  sentence  7:   30    6.0%  ############
  sentence  8:   30    6.0%  ############
  sentences 9+: 144   28.8%

Lead bias — the tendency of news writing to put the important content first — is real but weak here: the first sentence wins only 15.8% of the time, and 28.8% of summaries are best matched by something from sentence 9 onward. A system that reads only the opening is throwing away most of the evidence.

A caution about that 47.3%

The novelty figure is inflated by Turkish, and by exactly how much is measurable. Re-run it with the words normalized:

summary words absent from the article
  exact surface form              47.3%
  after Snowball Turkish stemming 37.5%
  after 5-character prefix        32.8%

Between ten and fifteen of those points are inflection, not new content:

summary 'bölgede'      <- article 'bölgedeki'
summary 'parlamento'   <- article 'parlamentonun'
summary 'uzmanlara'    <- article 'uzmanlar'
summary 'işgal'        <- article 'işgali'

Same word, different suffix, counted as different by anything that compares surface strings. Hold on to that: it is a corpus statistic here, and in Section 14.4 it becomes a defect in the field’s standard metric.

14.3 Extractive baselines, and their ceiling

The audit says these summaries are composed rather than copied, which is a prediction about how well selecting sentences can do. Worth checking against the numbers rather than trusting the inference.

The extractive family selects sentences from the source. It cannot invent wording — every word was in the document — and it cannot compress below a sentence. Note the limit of that guarantee: a selected sentence can still misrepresent the source, because the condition, the negation, the attribution or the exception that governs it may live in a sentence you did not select. Extraction rules out fabricated phrasing, not distortion by omission. Full test set, 3,397 articles:

system                    R-1     R-2     R-L    words   time
lead-1                  12.75    3.37   10.69     20.0
lead-2                  14.04    3.61   10.94     37.6    0.2s
lead-3                  14.03    3.74   10.44     54.9
random sentence          8.56    1.73    7.30     17.5
TextRank (1 sentence)   13.66    3.27   10.88     30.3   25.4s
oracle sentence         24.89    9.31   20.39     20.1    1.8s

Three things to read out of that table.

Lead-3 is not strong here. The received wisdom — “always report lead-3, it beats trained models” — is a fact about CNN / DailyMail, where the same code scores it at 29.44 R-1, and not a fact about summarization. On this corpus lead-3 scores 14.03 against lead-2’s 14.04. Do not read that as lead-2 beating it: 0.01 ROUGE-1 over 3,397 articles is a tie, and the table one column over shows the third sentence is adding something — R-2 rises from 3.61 to 3.74. What it adds in recall it pays for in precision, seventeen words’ worth, and the two cancel. Always report the lead baseline; never assume it is strong.

TextRank does not earn its 25 seconds. TextRank (Mihalcea and Tarau 2004) treats each sentence as a node, links sentences that share words, and runs PageRank on that graph: a sentence is important if it is similar to many other important sentences, so the most “central” sentence is taken as the summary. The graph is the kind of object Chapter 11 built over documents; the centrality score is new, but there is no trained model. LexRank (Erkan and Radev, 2004) is the same idea with TF-IDF cosine similarity as the edge weight. TextRank scores 13.66, between lead-1 and lead-2. It is a real method, and on this corpus it buys under one ROUGE-1 point over the first sentence while spending ten more words to do it.

The oracle is the number that decides the chapter. The oracle picks, for each article, the single source sentence maximising ROUGE-1 F1 against the reference — it reads the answer first, so it is not a system but a ceiling. It scores 24.89.

Two cautions about how to read that row. The 9.31 and 20.39 beside it are the R-2 and R-L scores of the sentence R-1 selected, not oracles for those metrics. An R-2 oracle could pick a different sentence, and it would score at least 9.31 on R-2, because by definition it picks the R-2 maximum for every article. And it is a single-sentence ceiling, so it is comparable across corpora only when the reference lengths are: XL-Sum Turkish references run about 21 words in one sentence, CNN / DailyMail’s about 56 across three or four, which structurally caps the recall a single extracted sentence can reach on the second.

No single-sentence extractive system on this corpus can exceed 24.89 ROUGE-1, because that is what perfect one-sentence selection achieves. Keep the scope attached to the number every time you quote it: it bounds single-sentence selection at this length budget, not extraction in general — a multi-sentence selector, or a different length budget, is a different oracle and could land either side of it. Resist reading the gap between lead-2’s 14.04 and that 24.89 as headroom for one-sentence selection: lead-2 takes two sentences, so the two are not on the same budget. For that reading you want a one-sentence baseline — lead-1 is right there — or a two-sentence oracle to put beside lead-2.

That is a corpus property, computed in 1.8 seconds, and it tells you whether to reach for an abstractive model before you train anything.

Run this oracle on any summarization corpus you are handed. If it is high — CNN / DailyMail’s three-sentence oracle is above 50, against the 40.58 the single-sentence oracle used here scores on the same corpus — say which one you computed, because the two are not interchangeable. Either way the reading is the same: high means extraction is viable and cheap, and you should exhaust it before generating. If it is low, as here, the reference summaries are compositions and extraction is structurally capped. Two lines of code, and it decides the architecture.

14.4 What ROUGE measures, and what breaks it

ROUGE (Lin 2004) counts n-gram overlap between a candidate summary and a reference. ROUGE-1 is unigram overlap, ROUGE-2 bigram, ROUGE-L the longest common subsequence — the longest run of words that appears in both texts in the same order, gaps allowed. The name stands for Recall-Oriented Understudy for Gisting Evaluation: the original version reported recall, and today each variant is usually reported as an F1.

A worked example makes the three numbers concrete. Reference: “the minister denied that the bank would be sold” (9 words). Candidate: “the minister said the bank would be sold” (8 words). Seven candidate words also occur in the reference. (A word is matched at most as many times as it occurs in the reference; the occurs twice in each text, so both copies count.) So

\[ P = \tfrac{7}{8} = 0.875, \qquad R = \tfrac{7}{9} = 0.778, \qquad F_1 = \tfrac{2PR}{P+R} = 0.82 . \]

For ROUGE-2 the units are word pairs: 5 of the candidate’s 7 bigrams occur in the reference’s 8, giving \(F_1 = 0.67\). The longest common subsequence is the minister … the bank would be sold, 7 words, so ROUGE-L equals ROUGE-1 here. Papers multiply by 100, so this candidate “scores 82 ROUGE-1”. Keep the example in mind: the candidate says the opposite of the reference, and we come back to it below.

It has been the field’s default metric for twenty years, and it survives for the honest reason that it is cheap, deterministic and correlates with quality well enough to rank obviously-bad systems below obviously-good ones.

ROUGE — n-gram overlap against a reference summary, reported as an F1. Cheap and reproducible; blind to whether the summary is true, and dependent on a tokenizer few people check. ↩︎

This chapter has three complaints, in increasing order of how much they should worry you.

Complaint one: the standard implementation destroys Turkish

Google’s rouge-score is the reference implementation. HuggingFace’s evaluate wraps it, and a large share of papers reporting ROUGE since 2019 have used one or the other. Its tokenizer, reproducing the original Perl script’s behaviour, is one line:

# rouge_score/tokenize.py
text = text.lower()
text = re.sub(r"[^a-z0-9]+", " ", text)   # <- everything else is a separator

[a-z0-9] does not include ç, ğ, ı, ö, ş, ü or İ, so the negated class [^a-z0-9] matches them and treats each as a separator. In Turkish those are not punctuation; they are letters, and common ones — ı alone is among the language’s most frequent letters. So:

görüştü        -> ['g', 'r', 't']
çiğ            -> ['i']
köfte          -> ['k', 'fte']
değişiklik     -> ['de', 'i', 'iklik']
şehrinde       -> ['ehrinde']
Türkiye'nin    -> ['t', 'rkiye', 'nin']
Run the tokenizer yourself — type Turkish, watch it disintegrate

Measured over 300 reference summaries from this corpus:

word tokens, correct Turkish tokenization    6,682
tokens after rouge-score                     9,660   1.45x as many
words left intact                            3,713    55.6%
words split at non-ASCII letters             2,969    44.4%
average pieces per affected word              2.00

Forty-four percent of Turkish words do not survive the tokenizer, and the debris inflates every score. Single-letter fragments such as g, r, t, i turn up in almost any Turkish text, so a candidate and a reference share them by chance, and overlap goes up. ROUGE-1 on the same 3,397 summaries, under two tokenizers:

system                  correct tokenizer    rouge-score default    inflation
lead-1                            12.75                  20.37        +59.8%
lead-2                            14.04                  22.51        +60.3%
lead-3                            14.03                  22.03        +57.0%
random sentence                    8.56                  15.44        +80.4%
TextRank                          13.66                  21.34        +56.2%
oracle sentence                   24.89                  30.28        +21.7%

Be precise about the damage, because overstating it would be its own error: the ranking is unchanged. lead-2 > lead-3 > TextRank > lead-1 under both tokenizers.

What is destroyed is comparability. Every Turkish ROUGE number computed with the stock rouge-score or evaluate default — which is most of them — is roughly 60% larger than the quantity its readers think it is, by an amount that varies with a system’s genuine overlap with the reference: the absolute gain from fragments is similar for every system (5.4 to 8.5 points), so the random sentence, with the least real overlap, gains 80% and the oracle gains 22%.

Not all of them, and the exceptions are worth knowing because they are the fix. XL-Sum’s own authors shipped a fork of rouge-score with Unicode-aware tokenization and per-language stemmers, Turkish among them, and their published Turkish numbers were produced with it; the same gap has been an open issue on HuggingFace’s libraries since 2020 (first on datasets, now on evaluate). So the defect is documented, and the thing that is actually missing is the habit of saying which implementation produced a number. Report the tokenizer beside the score, the way you would report a random seed. The inflation is a function of the system, so the gaps between systems are wrong even though the order survives.

Our own tokenizer needed a fix in the same family, and it is the bug Chapter 11 found in the wild: "İstanbul".lower() gives i plus a combining dot, which \w+ then splits into i and stanbul; and "IŞIK".lower() gives işik rather than ışık. Turkish lowercasing is not str.lower(). Both directions have to be special-cased, and the bug is invisible unless you print the tokens.

To be sure the argument is about the tokenizer and not about our arithmetic, the ROUGE used in this chapter was validated against rouge-score on 300 random ASCII-only pairs, where both tokenizers agree:

max absolute difference vs rouge-score, 900 comparisons:  0.0000000000

The implementations are identical. The entire Turkish gap is tokenization.

The fix, which is one argument

None of the above requires reimplementing ROUGE, and it would be a poor lesson if it did. rouge-score 0.1.2 takes a tokenizer= argument — anything with a tokenize method from string to list of strings — so the defect is one constructor away from fixed:

import re
from rouge_score import rouge_scorer

class TurkishTokenizer:
    def tokenize(self, text):
        # Turkish-aware lowercasing first, then Unicode letters and digits
        text = text.replace("I", "ı").replace("İ", "i").lower()
        return re.findall(r"\w+", text, flags=re.UNICODE)

scorer = rouge_scorer.RougeScorer(
    ["rouge1", "rouge2", "rougeLsum"],
    use_stemmer=False,                  # ignored once tokenizer= is set; see below
    tokenizer=TurkishTokenizer())

Be exact about use_stemmer too, because it is the next trap. Its default is already False, and setting it to True applies a Porter stemmer, which is English. Once you pass your own tokenizer=, the flag is ignored altogether: use_stemmer only configures the default tokenizer. So for Turkish the right answer is not “leave it off” but “replace it”, and the replacement goes inside your tokenize method — a Turkish stemmer applied to each token before it is returned. The next subsection shows how much that choice moves the score. HuggingFace’s evaluate wrapper accepts the same fix: its rouge metric takes a tokenizer= argument (a function from string to tokens) and passes it through.

Two further options: XL-Sum’s own fork of rouge-score, which removes only Unicode punctuation for all its 44 languages and adds stemmers for eighteen of them, Turkish included; and a variant that scores subword tokens from a BPE tokenizer, which sidesteps the language question entirely at the cost of scoring pieces of words. One tool that looks like a fix is not: sacrerouge gives many summarization metrics a common Python interface, but for ROUGE it wraps the original Perl ROUGE-1.5.5, whose tokenizer also throws away every character outside A–Z, a–z, 0–9 and the hyphen — the same defect. If you publish a Turkish ROUGE number, name the tokenizer beside it. That one sentence in a paper is worth more than any of the tooling.

Complaint two: it depends on a normalizer nobody reports

Turkish inflects. bölgede and bölgedeki are two forms built on the same stem; ROUGE sees two unrelated tokens. Applying a stemmer before counting moves every number. ROUGE-1 under three normalizers:

system                  surface   5-char prefix   Snowball tr
lead-1                    12.75           17.85         16.13
lead-2                    14.04           19.46         17.67
lead-3                    14.03           19.13         17.48
TextRank                  13.66           18.24         17.11
oracle sentence           24.89           29.30         27.77

About five points of ROUGE-1 (4.4 to 5.4, depending on the system) sit between “surface form” and “5-character prefix”, which is a larger difference than most papers’ claimed improvements over their baselines. A ROUGE number is not interpretable without the tokenizer and the normalizer that produced it, and those are almost never stated. This is not a Turkish problem; it is worse in Turkish.

Complaint three: the one that actually matters

The first two complaints are about reproducibility, and could be fixed by a convention. The third cannot be fixed.

ROUGE cannot see whether the summary is true. It counts word overlap with a reference, so a summary that copies the right words in a truth-reversing order scores well, and a correct paraphrase using different words scores badly. Return to the worked example: the source says “the minister denied that the bank would be sold”, and that sentence is also the reference:

What overlap cannot distinguish (scores computed with rouge-score)
candidate truth ROUGE-1 F1 vs reference
the minister denied the bank would be sold correct 0.94
the minister said the bank would be sold false 0.82 — one word differs
the sale was ruled out by the ministry correct 0.24 — almost no overlap

The false summary and the true one differ by a single token, and that token carries the entire meaning. Kryściński et al. (2019) made this case at length; Fabbri et al. (2021) then measured it, finding ROUGE’s correlation with human judgements of consistency to be weak enough that it cannot be used to rank systems on faithfulness. Goyal et al. (2022) found human annotators preferring GPT-3 summaries that ROUGE ranked below fine-tuned models.

An embedding-based metric fixes only part of this. BERTScore (Zhang et al. 2020) matches each candidate token to its most similar reference token in the contextual-embedding space of a pretrained encoder, instead of requiring the same string. That rescues the third row — sale and sold, ministry and minister are close in that space — and on Turkish it partly absorbs suffix differences like bölgede / bölgedeki, as long as you use a multilingual or Turkish encoder. It does not reliably rescue the second row: said and denied occur in similar contexts, so their embeddings are close even though they reverse the meaning. BERTScore is still a comparison against a reference, not against the truth.

So: report ROUGE, because it is the common currency and its absence looks evasive. Never conclude from it alone. Section 14.7 and Section 14.8 are what to add.

14.5 Abstractive summarization

Single-sentence extraction is capped at 24.89 on this corpus, so most of the remaining ground belongs to models that write. Two framings have been used, and the history is worth two paragraphs because it explains why the current answer looks so anticlimactic.

Encoder-decoder, trained on the task. The encoder reads the article, the decoder writes the summary, and the whole thing is trained on (article, summary) pairs. BART, T5, mT5 and their descendants. This is what “a summarization model” meant until about 2022, and it still wins on throughput per unit of quality when you have a corpus and a fixed domain.

Decoder-only, instructed. Chapter 8’s next-token objective already teaches a model how text of every genre is shaped, including the shape of a news lede. Chapter 9 is what makes the model follow the instruction to produce one rather than continue the article. Summarization turned out to be one of the tasks that needed no dedicated fine-tuning on a summarization corpus — the capability was latent in the pre-trained model, and general instruction tuning (whose data does include many “summarize this” requests) unlocked it.

That is why this chapter’s abstractive system is a prompt rather than a training run. It is also the background to Goyal et al. (2022), who found human annotators preferring zero-shot GPT-3 summaries over fine-tuned models that scored higher on ROUGE. This chapter has no fine-tuned model to compare against, but it is about to show, in miniature, the mechanism behind that result: references that a faithful summary cannot match.

PROMPT = """Aşağıdaki haber metnini TEK BİR CÜMLEYLE özetle.

Kurallar:
- Yalnızca tek bir cümle yaz, en fazla 30 kelime.
- Sadece metinde geçen bilgileri kullan; metinde olmayan hiçbir şey ekleme.
- Haberin ana olayını ver, ayrıntıları değil.
- Başlık değil, tam bir cümle yaz.
- Sadece özeti yaz, başka hiçbir şey yazma.

Haber metni:
{text}"""

Two details in that prompt earn their place. The length cap exists because ROUGE-1 F1 punishes long summaries through precision, so an uncapped model writes three sentences and scores worse while being no less useful — a metric artefact you have to defend against rather than discover. Expect the cap to be followed only approximately: the model generates tokens, not words, and does not count reliably, so check the length distribution of what comes back (here the mean was 23.7 words) rather than trusting the instruction. And the instruction to use only what is in the text is the faithfulness constraint stated explicitly; Section 14.7 measures whether it worked.

What it scores

All six systems on the same 500 articles, ROUGE-1 with a tokenizer that handles Turkish:

Same 500 articles, surface-form normalizer
system R-1 R-2 R-L words
lead-1 13.24 3.36 10.85 20.1
lead-2 14.69 3.79 11.16 38.0
lead-3 14.52 3.95 10.74 55.4
TextRank (1 sentence) 13.92 3.29 10.93 29.8
LLM, zero-shot 19.29 6.00 16.08 23.7
oracle sentence 25.05 9.21 20.29 19.5

The zero-shot LLM beats the best real extractive system (lead-2) by 4.6 ROUGE-1 points, and the weakest (lead-1) by 6.05, and still sits 5.8 below the extractive oracle. Both halves of that matter: generation is clearly worth it here, and the ceiling on perfect sentence selection is still above what a 9B model writing freely achieves against these references.

Which should already make you suspicious of the metric. To a reader, the single best existing sentence is not obviously a better summary than one written for the purpose; it scores higher partly because it was chosen by looking at the reference. Section 14.7 is where that suspicion is confirmed with a measurement.

14.6 Long documents

The articles above are 300 words. Real summarization requests arrive as 80-page contracts, quarterly filings and year-long ticket histories, and at that point the architecture stops being a model choice and becomes a pipeline design.

Three patterns, in increasing order of what they cost and what they preserve:

Truncate. Feed the first n tokens and ignore the rest. Honest when lead bias is strong; on this corpus, where 28.8% of summaries are best matched by sentence 9 or later, it throws away most of the evidence. The cheapest option and the one most often chosen by accident.

Map-reduce. Chunk, summarize each chunk, then summarize the summaries. Cost is linear in document length, it parallelises perfectly, and it is the default in every framework. What it puts at risk is anything that spans a chunk boundary: a contract clause qualified twelve pages later, a retraction, a narrative arc. Each chunk summarizer sees a window with little idea what came before, so it cannot reliably tell which of its details will matter globally, and the reduce step can only work with what the map step happened to keep. Partial remedies exist and cost tokens: overlap the chunks, carry a running summary into each map call, or run a second pass once the reduce step has told you what the document is about. None of them recovers a dependency the map step threw away.

Refine, or hierarchical rollup. Carry a running summary forward, updating it with each chunk. Preserves cross-chunk continuity, and is strictly sequential — no parallelism — with the running summary drifting as it is repeatedly rewritten. Late chunks get more influence than early ones, which is the opposite failure to truncation.

The three differ in one thing only, and it is not the model. It is what the pipeline can possibly know at the moment it writes. Take a document whose §1 states a clause and whose §5 is the exception to it — the shape every long document has and no chunk boundary respects:

Three pipelines over the same document, and where each one loses the exception
THE DOCUMENT — SIX SECTIONS, AND ONE FACT THAT LIVES IN TWO OF THEM the same clause, forty pages apart §1 clause 4.1 §2 §3 §4 §5 the exception §6 TRUNCATE — READ THE FIRST n TOKENS, IGNORE THE REST §1 §2 dropped dropped dropped dropped §5 is never read, so the summary states clause 4.1 as unconditional — and nothing anywhere in the pipeline can notice. MAP-REDUCE — SUMMARIZE EACH SECTION, THEN SUMMARIZE THE SUMMARIES §1 §2 §3 §4 §5 §6 s1 s2 s3 s4 s5 s6 one final summary Nothing is dropped, and the loss is still total: s1 and s5 were written independently, neither knows the other exists, and the reduce step joins two locally correct sentences into one that is jointly wrong. REFINE — CARRY ONE RUNNING SUMMARY FORWARD, REWRITING IT AT EVERY SECTION §1 §2 §3 §4 §5 §6 r1 r2 r3 r4 r5 r6 r5 still carries clause 4.1, so the exception lands. But §1's wording has been rewritten four times by the time it arrives, no two of these steps can run in parallel, and §6 always gets the last word.

The pipelines differ only in what is on screen when the summary is written. Truncation loses §5 outright; map-reduce reads it and still cannot connect it to §1; refine connects them and pays for it in drift and in serial time.

The arithmetic behind this is Chapter 7’. Attention is quadratic in sequence length, so on hardware you run yourself one pass over 100k tokens costs more than ten times a pass over 10k. Map-reduce trades that quadratic for a linear number of small quadratic passes, which can make it cheaper in compute even where a long-context model would fit the document. Under per-token API pricing the sum runs the other way — map-reduce pays for the intermediate summaries as well — so check which bill you are paying. Either way, “the model has a million-token window” does not settle the design: cost is one question, and whether the model actually uses the middle of a very long input is another.

The rule that survives contact with production: chunk on structure, never on character count. Sections, clauses, chapters, speaker turns, dates. A boundary drawn mid-argument produces two summaries that are each locally plausible and jointly wrong, and that failure is invisible in every automatic metric — it needs a reader who knows the document.

14.7 Faithfulness

A summary can be fluent, on topic, well-sized, high-ROUGE, and assert something the source never said. Faithfulness is whether every claim in the summary is supported by the source, and it is a different question from whether the summary reads well or matches a reference.

faithfulness — whether a summary asserts only what its source supports. Independent of fluency, independent of reference overlap, and the property that decides whether anyone can act on the output. ↩︎

Maynez et al. (2020) split the failure in two, and the distinction is operationally useful because the fixes differ:

  • Intrinsic — the summary misstates something that is in the source. Swapped subject and object, a negation dropped, two entities conflated, a number attached to the wrong thing. Detectable by comparing against the source, and usually a model error.
  • Extrinsic — the summary adds something not in the source at all. Sometimes true (the model knows who the finance minister was) and sometimes invented. Detectable as unsupported; deciding whether it is also false needs the world, not the document.

An extrinsic addition that happens to be true is still a defect in an informative summary, because the reader cannot tell which sentences were grounded and which were supplied by the model.

Counting it without a human

A crude proxy gets you a long way: a number or a proper noun in the summary that appears nowhere in the source is an unsupported claim. It misses intrinsic errors entirely — a swapped subject uses only source words — but it catches extrinsic ones, and it costs nothing to run on every summary you generate.

Crude proxies need a control, or you will believe whatever they say. The control here is free: lead-2 is copied verbatim from the article, so its flag rate is the proxy’s false-positive rate, and it must be zero.

system                nums  unsupported   propers  unsupported   summaries flagged
lead-1  (control)      100            0       521            0       0    ( 0.0%)
lead-2  (control)      173            0       809            0       0    ( 0.0%)

Zero on both — which bounds the false-positive rate rather than eliminating it. Zero out of 150 carries a 95% upper bound of 2.5%, and that matters as soon as the model’s rate comes back at 3.3%: the control does not have the resolution to say how much of that 3.3% is real. Read it as the proxy does not fire on copied text at anything like the rate it fires on the references, which is the comparison the section needs. It is what makes the next table worth reading. Getting there took three attempts; the two failures are in ?sec-summarization-proxy-bugs because they are more instructive than the fix.

Attempt one compared the summary’s capitalised words against the capitalised words of the source — and skipped sentence-initial ones in both, to avoid counting “The” as a proper noun. So a name that appears in the article only at the start of a sentence was recorded as absent. It reported that 56% of BBC journalists’ own summaries were unfaithful, which is the kind of number that should stop you rather than excite you.

Attempt two fixed that by checking the whole source, but built the summary’s token set with str.split() while building the source’s with \w+. So İtalya'ya in the summary never matched italya in the source, neo-İslamcı never matched islamcı, and a source typo like Komisyonu,hemşirenin matched nothing at all. Copied text — which cannot be unfaithful — came back 8% unfaithful.

Both bugs share a shape: the two sides of a comparison were tokenised differently. That is the same defect Section 14.4 finds in rouge-score and the same one Chapter 13 found in an LLM extraction pipeline. When a metric compares text against text, the first thing to check is that both sides went through the same tokenizer, and the cheapest way to check is a control whose answer you already know.

What the proxy found

With both controls at zero, a flag now means something. Here is the same proxy on the model’s summaries and on the human references, over the same 150 articles:

The most surprising number in this chapter is not about a model.

system                nums  unsupported   propers  unsupported   summaries flagged
lead-1  (control)      100            0       521            0       0    ( 0.0%)
lead-2  (control)      173            0       809            0       0    ( 0.0%)
LLM (qwen3.5:9b)        61            0       494            3       5    ( 3.3%)
reference (human)       74           33       644          173      86    (57.3%)

The reference summaries assert unsupported content at about seventeen times the rate the model does — 57.3% against 3.3%, on the same 150 articles.

With \(n = 150\) those are 86 summaries and 5. The 95% Wilson intervals are [49.3, 65.0] and [1.4, 7.6]: non-overlapping by a wide margin, which is the claim. The point estimates are not the claim, and neither is the ratio between them — a ratio of two small proportions is the least stable thing you can quote.

The two right-hand columns of that row do not reconcile, and the honest thing is to say so rather than round it away: five flagged summaries cannot come from three unsupported proper nouns. Five is the figure the rest of this section uses — it is what the judge comparison below reports as 3.3%, and the \(\kappa = 0.08\) reported there is only arithmetically possible with five flags and one shared item. (The 90.7% agreement alone cannot decide it: three flags with no shared item also give 136 agreeing verdicts out of 150, but a \(\kappa\) just below zero.) Treat the per-item counts in this table as needing a recount, and the per-summary rate as the one to quote.

This is not a measurement error — the controls are clean — and it is not BBC journalists inventing news. It is how the corpus was built.

XL-Sum’s “summary” is the bolded intro line that accompanies the article, written to orient a reader arriving cold, so it supplies background the body then takes for granted: the minister’s first name, the date of the referendum, the figure from a report the article only alludes to. Maynez et al. (2020) documented exactly this for XSum, which is built the same way: their annotators found extrinsic content in 73.1% of XSum’s own reference summaries. Here it is again in Turkish.

Some of the 57.3% is the proxy counting an abbreviation mismatch — the reference writes Avrupa Birliği where the article writes AB — so treat it as an upper bound. The direction is not in doubt.

Three consequences follow, and they are the reason this section is not a footnote.

  1. ROUGE against these references rewards unfaithfulness. A model that reproduces the reference’s unsupported background scores higher. A model that refuses scores lower.
  2. Our prompt told the model to use only what is in the text, and it complied — 3.3% against the reference’s 57.3%. It is being penalised by the metric for doing the thing we asked.
  3. Fine-tuning on this corpus would teach a model to hallucinate, because the training signal contains 57.3% unsupported assertions. This is the single strongest argument in this chapter against training on a scraped summarization corpus without auditing it first.

Which is also the answer to “why not just fine-tune mT5 on the 27,176 training pairs?” You can, and it will score better on ROUGE, and part of what it will have learned is to assert things its input does not contain. The metric will not tell you. Chapter 5 is the chapter on noticing this class of problem before it reaches a model.

Between the proxy and the judge

Two families of learned checks sit between a string-matching proxy and a full LLM judge, and you will meet both in papers. This chapter did not run them, but you should know what they do.

  • Entailment (NLI) checks. A natural-language-inference model is a classifier that reads a premise and a hypothesis and says whether the premise entails the hypothesis, contradicts it, or neither. Use the source as premise and each summary sentence as hypothesis; a sentence the source does not entail is flagged. SummaC (Laban et al., 2022) and AlignScore, listed in Section 14.12, work this way.
  • Question-answering checks. QAGS (Wang et al., 2020) generates questions from the summary, answers each one twice — once from the summary, once from the source — and counts the questions whose two answers differ.

Both can catch some intrinsic errors the proxy cannot: a dropped negation changes the entailment label, and a swapped subject changes the answer to “who sold the bank?”. Both are far cheaper to run than an LLM judge. But both inherit the mistakes of the model inside them, and the standard checkpoints were trained on English, so on Turkish they need a multilingual model and the same copied-text control before you trust them.

14.8 LLM-as-judge, and what it costs you

ROUGE cannot see faithfulness; a hand audit can catch failures no overlap metric will, and does not scale. The middle option is to ask a model, which is now the default in practice (Liu et al. 2023) and comes with biases that have to be measured rather than assumed away.

The design used here follows three rules worth stating, because breaking any of them quietly invalidates the result:

  1. A different model judges than generates. gemma3:27b judges qwen3.5:9b. Models prefer their own output, and a self-judged score is not a measurement.
  2. The judge answers one narrow question, under a schema. Not “rate this summary 1-5” — is every claim in this summary supported by the source? with a two-value enum. A rubric with one axis is reproducible; a five-point overall quality score is a mood. One thing this chapter owes you and does not supply: the judge’s prompt in full, the way Section 14.5 prints the generation prompt. A judged rate is a property of a prompt as much as of a model, and 33.3% is not reproducible without it.
  3. The human reference is judged too, under identical conditions. It is the only way to know what the judge’s scores mean, and — as Section 14.7 showed — the reference is not automatically the ceiling.

What the judge said

150 articles, both the model’s summary and the human reference judged under identical conditions:

Share of summaries with at least one unsupported claim, same 150 articles
judged by LLM (qwen3.5:9b) reference (human)
automatic proxy 3.3% 57.3%
gemma3:27b judge 7.3% 33.3%

The judge confirms the direction: the human references are judged unsupported four to five times as often as the model’s summaries. Two methods, one crude and one expensive, disagree about the level and agree about the ordering.

The part worth more than either number

Two methods agreeing on the ordering is a comfortable result, and it is the easy half. The uncomfortable half is whether they agree about the same summaries:

                       proxy flags   judge flags   both   verdicts agreeing
LLM (qwen3.5:9b)                 5            11      1        90.7%
reference (human)               86            50     35        56.0%

On the model’s summaries the two methods flag 5 and 11 items and overlap on exactly one. That 90.7% agreement figure is almost entirely agreement about the clean summaries; where either method sees a problem, they are nearly disjoint.

Which is why raw agreement is the wrong statistic here, and Chapter 5 already said so. Chance-correct it. On the model’s summaries the two methods reach \(\kappa = 0.08\); on the references, where they agree far less often in raw terms, \(\kappa = 0.16\). Both are in the band Chapter 5 calls slight, and the 90.7% is what a high base rate of “clean” does to an uncorrected number. Two methods that agree 90% of the time and \(\kappa = 0.08\) have not corroborated each other; they have both mostly said no.

This is what the theory predicts and it is easy to forget:

  • The proxy can only see extrinsic errors — a name or number that is not in the source. It is structurally blind to intrinsic ones, because a swapped subject or a dropped negation uses only source words.
  • The judge sees both, and adds its own errors: it flags the model at 7.3% where the proxy sees 3.3%, and it clears references the proxy flags, because it correctly resolves AB to Avrupa Birliği and the proxy cannot.

So: the ranking is the trustworthy part; the rate is not. Report both measurements, report the disagreement, and if you need an absolute number — “what fraction of what we ship is wrong?” — neither of these gives it to you. That number comes from reading a random sample, which is Chapter 5’s discipline and Section 14.13’s step 6.

What the judge costs

So the judge earns its place, and the only remaining question is how much of it you can afford. That answer is on the clock:

300 judgements with gemma3:27b     20,482s     68.3s each

Nearly six hours to judge 150 article-summary pairs twice, against 20 minutes to generate 500 summaries with the 9B model. In this setup judging is far more expensive than generating. Input length does not explain it: the generator reads the whole article too. The judge is a model three times the size (27B against 9B), which accounts for only part of the roughly 28-fold gap between 68.3 s per judgement and 2.4 s per summary; the rest most likely comes from the length of the judge’s output and its thinking setting, which this chapter does not break down. And you run the judge once per system. Budget for it as an evaluation cost, not an afterthought, and sample rather than judging everything.

The biases, and which ones this chapter measured

The design rules above exist because LLM judges have known, replicable failure modes. This chapter’s setup avoids two of them by construction and does not measure the third:

Judge biases and what this setup does about them
bias what it is handled here
Self-preference a model scores its own output higher avoided: gemma3:27b judges qwen3.5:9b
Position in a pairwise comparison, the option presented first wins more often (Zheng et al. 2023) not applicable: this judge scores one summary at a time, with no pairwise ordering
Length longer, more fluent answers score higher regardless of content not measured here — the length cap in the prompt holds the model’s summaries near the reference length (23.7 words against 21), which reduces the exposure but does not test for it

Stating the third one plainly rather than waving at it: if you move to a pairwise judge, position bias becomes live, and with a pairwise or 1–5 rubric judge, so does length bias; and Zheng et al. (2023)’s swap test — present both orders, count how often the verdict flips — is the measurement to run before believing any of it. Chapter 22 runs that test, and a padding test for length bias, on these same two models over Turkish news summaries.

14.9 Summarization in production

The engineering questions are the same three every generative feature has, and summarization has unusually clean answers to two of them.

Caching is nearly free money. With deterministic decoding, a summary is, in practice, very nearly a pure function of (document, prompt, model, decoding parameters) — GPU batching and floating-point ordering can occasionally change the output even at temperature 0; with sampling it is not, but once you have accepted one sample there is no reason to pay for another. Key a cache on the hash of those four and the same document is never summarized twice. Documents are re-read far more often than they are written, so hit rates are high, and unlike most LLM features there is no personalisation to defeat the cache. Version the prompt into the key, or a prompt change will silently serve stale output forever.

Cost is dominated by input, not output. Measured over the 500 articles summarized here: 890 prompt tokens per article against 53 generated, a token ratio of nearly 17 to 1 — and these are 300-word articles, the short end of anything you would want summarized. Convert that to money with your own price sheet before quoting it as a cost ratio: output tokens are typically billed at three to five times input, which puts 16.8:1 in tokens somewhere between 3.4:1 and 5.6:1 in currency under those assumed multipliers. The conclusion survives the conversion here; on a task that generates more than it reads, it would not. For long documents under map-reduce the ratio gets worse, because the intermediate summaries are read again.

The arithmetic is Chapter 31’s: what you pay for is reading, so the lever is how much you send, not how much you ask for. Truncating boilerplate — headers, navigation, disclaimers, signature blocks — is usually a larger saving than any model change.

The human in the loop is a design decision, not a fallback. Return to the question from Section 14.1: what happens if the summary is wrong? Three shapes, in order of how much they cost:

Who checks, and what checking costs
pattern when what it costs
Show source alongside the reader can verify in one glance UI work, no per-item cost
Link every claim to its sentence claims are checkable individually the extraction of Chapter 13, plus alignment
Human review before delivery acting on a wrong summary is expensive a person, per item

Showing the source next to the summary is the highest-return feature in most summarization products and it is not a model improvement at all. It converts an informative summary — which must be right — into an indicative one, which only has to be useful.

14.10 Intuition checks

Answer each one before you open it. All six are questions this chapter settled with a measurement rather than an argument.

You cannot be, until you know three things they almost certainly did not state: the tokenizer, the normalizer, and the reference set.

On this corpus the same predictions score 14.03 with a Turkish-correct tokenizer and 22.03 with rouge-score’s default, which treats ç, ğ, ı, ö, ş and ü as punctuation and so splits 44.4% of Turkish words into fragments. Stemming or prefix-truncation moves it another three to five points. So “41” could be a strong system, or a weak system measured generously, and the number alone cannot tell you.

Ask what tokenizer produced it. If the answer is “the standard one”, the number is not comparable to anything computed correctly.

Compute the oracle. For each document, take the single sentence with the highest ROUGE against the reference, and average. That is what perfect one-sentence selection achieves, so no single-sentence extractive system can beat it — say “single-sentence” out loud, because a multi-sentence selector is bounded by a different oracle.

Here it is 24.89, and lead-1, the one-sentence baseline to compare it against, reaches 12.75, both on the full 3,397-article test set. So about twelve points of room are left for better one-sentence selection. (Lead-2’s 14.04 spends two sentences, so it is not the comparison to make.) Anything past 24.89 needs either more sentences or writing. (Quote the pair from one sample: on this chapter’s 500-article subsample the same two numbers are 25.05 and 13.24, and mixing a figure from each is how gaps get invented.) On CNN / DailyMail the same measurement gives 40.58, which is why extractive systems were respectable there for a decade.

Two lines of code, and it decides your architecture before you spend a GPU hour.

Not publish it. A number that surprising is far more likely to be a bug in your measurement than a discovery, and the way to find out is a control whose answer you already know.

Here the control is free: lead-2 is copied verbatim from the article, so it introduces no wording that is not in the source, and any flag a lexical faithfulness proxy raises on it is a false positive. (It could still mislead a reader by omission — but that is not what this proxy claims to detect, which is exactly what makes it a clean control for it.) Running it exposed two bugs in two successive versions of the proxy — both the same mistake, tokenising the two sides of the comparison differently.

Only after the control returned exactly zero was the reference’s rate worth reporting. It was still high (57.3%), for a real reason: the corpus’s summaries are BBC intro lines that supply background the article body does not contain.

Maybe the opposite. On this corpus the reference summaries assert unsupported content in 33–57% of cases depending on how you measure, so ROUGE against them rewards a model for reproducing that. Our prompt told the model to use only what is in the text; it complied, and the metric charged it for the compliance.

Goyal et al. (2022) found the same pattern in a human study — people preferring zero-shot GPT-3 summaries that ROUGE ranked below fine-tuned models. When ROUGE and a faithfulness measurement disagree, the reference is a hypothesis about quality, not the definition of it.

At the chunk boundaries, and invisibly. Each chunk summarizer sees its window with no knowledge of the rest, so it cannot tell which of its details will matter globally; the reduce step can only work with what survived. A clause qualified twelve pages later produces two locally-correct summaries that are jointly wrong.

No automatic metric catches this — both summaries are fluent, both are grounded in their own chunk. The mitigations are to chunk on structure rather than character count, and to carry a running summary forward (refine) when continuity matters more than parallelism.

When being wrong is expensive and being clumsy is not. An extractive summary is quoted text: it invents no wording, though a sentence quoted away from its context can still mislead — which in a regulated or legal setting is worth more than fluency.

Also when the oracle says the answer is in the document — a high oracle score means selection is viable — and when you need determinism, auditability, or a per-document cost of approximately zero. Lead-2 on this corpus takes 0.2 seconds for all 3,397 articles; the LLM took 20 minutes for 500 of them, about 2.4 seconds each.

14.11 Interview questions

On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the task

Q: How would you evaluate a summarization system?

The answer that ends the question: ROUGE for comparability, a faithfulness measurement because ROUGE cannot see truth, and a hand-audited random sample with an interval because neither automatic measure gives you an absolute rate. Then say what you would check first — the tokenizer, the oracle ceiling, and whether the references themselves are faithful. Candidates who answer “ROUGE” and stop have not shipped one.

Q: Why is ROUGE still used if it is this flawed?

Because it is cheap, deterministic, reproducible and comparable across papers, and because the alternatives are expensive, non-deterministic, or model-dependent. It ranks obviously-bad systems below obviously-good ones, which is most of what a leaderboard needs. The honest position is to report it and never to conclude from it alone — and to be able to say precisely what it cannot see: faithfulness, and any paraphrase that uses different words.

Q: Extractive or abstractive?

Ask what the oracle says, then ask what happens if the summary is wrong. High oracle plus expensive errors is an extractive problem. Low oracle plus tolerant readers is an abstractive one. A candidate who answers from preference rather than from those two questions is guessing.

Testing whether you have shipped one

Q: Your summarizer works in the demo and users say it hallucinates. Debug it.

Ordered by cost, and the first two are not the model. (1) Is it actually hallucinating, or is it surfacing something true the user did not expect? Have them point at a sentence. (2) Chunking — a map-reduce boundary produces locally-correct, jointly-wrong output that reads exactly like hallucination. (3) The prompt: is there an explicit grounding instruction? With one, the model measured here was flagged on 3–7% of summaries; this chapter did not run the same model without it, so measure that difference on your own data rather than assume it. (4) Only then, the model and its decoding parameters.

Q: How would you catch it automatically?

An entity-and-number proxy is an afternoon’s work and catches extrinsic errors; validate it on copied text, which must score zero. An entailment model or an LLM judge catches intrinsic errors too. The entailment model is cheap; the LLM judge cost roughly 68 seconds per judgement in this chapter’s setup. Say that the two disagree at the item level even when they agree on the ranking — measured here, an overlap of one item out of the five and eleven each flagged — so neither replaces a sampled human read.

Q: 10 million documents, a budget, and a deadline. What do you build?

Costs first: 890 prompt tokens per 300-word article, and input dominates, so the biggest lever is boilerplate removal, not model choice. Then caching — a summary is a pure function of (document, prompt, model, parameters), so key on all four and version the prompt into the key. Then ask whether the reader needs an informative summary or an indicative one, because showing the source next to the summary converts one into the other and is cheaper than any model improvement.

14.12 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below.

The stack these numbers came from. Python 3.14.5, Apple Silicon
library version used what it is for here
rouge-score 0.1.2 the reference ROUGE, used to validate this chapter’s implementation and to demonstrate its tokenizer
snowballstemmer 3.1.1 Turkish stemming, for the normalizer comparison in Section 14.4
numpy 2.5.2 the graph arithmetic behind TextRank and every average in the chapter
ollama 0.24.0 serving qwen3.5:9b for generation and gemma3:27b for judging

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
bert-score 0.3.13 embedding-based overlap (Zhang et al. 2020) — survives paraphrase where ROUGE does not, and is still blind to faithfulness
summac / AlignScore 0.0.4 / GitHub only entailment-based faithfulness scoring: does the source entail each summary sentence? The principled version of this chapter’s crude proxy
factscore 0.2.0 decomposes a text into atomic claims and checks each one, which is what you want when the summary is long; by default it checks claims against a knowledge source (Wikipedia), not the source document, so for summary faithfulness you swap in the document as the knowledge source
evaluate 0.4.6 HuggingFace’s metric wrapper — note that its rouge metric wraps rouge-score, tokenizer and all
lexrank / sumy 0.1.0 / 0.13.0 classical extractive summarizers packaged, when you want LexRank and LSA baselines without writing them
Trafilatura 2.2.0 stripping boilerplate out of documents before you pay to send them, which Section 14.9 says is the real cost lever
langchain-text-splitters 1.1.2 structure-aware chunking for the map-reduce pipelines of Section 14.6

14.13 Lab: summarize a corpus and find out whether anyone can trust it

Pick a corpus from your own domain with 200+ documents. Reference summaries are optional — the point of this lab is that you can evaluate without them.

Step 1 — decide what kind of summary, in writing

One paragraph naming the four axes from Section 14.1 and who reads the output. If you cannot write it, the rest of the lab is premature.

Step 2 — characterise the corpus before modelling

Compute coverage, density, and the oracle-sentence ROUGE. If the oracle is high, build the extractive system and stop. If it is low, you have just justified a generative model — with a number, to somebody who asked why.

If you have no reference summaries, run the oracle against the document’s own title or first sentence — and remove whatever you used as the reference from the candidate pool, or the oracle selects it and scores a meaningless perfect match. It is a weaker signal even done correctly, and it still tells you whether the answer is in the document or has to be composed.

Step 3 — build lead-k and TextRank

Always. They take an hour and they are the numbers every later claim is measured against.

Step 4 — generate, with the length cap and the grounding instruction

Both matter, and both are in Section 14.5’s prompt for reasons this chapter measured.

Step 5 — build the faithfulness proxy, then break it on purpose

Numbers and proper nouns absent from the source. Then run it on copied text — lead-2 — and confirm it returns zero. It will not, the first time. Fix it until it does, and keep the broken version’s output; the difference between the two is what you would have published.

Step 6 — audit a random sample by hand

Thirty documents, read properly, each summary marked supported or not. Report the rate with a Wilson interval, not as a verdict: 3/30 is 10% with a 95% interval from 3.5% to 25.6%, and shipping decisions differ across that range. Chapter 5 has the sampling discipline.

Step 7 — compare your hand audit against the proxy and the judge

Three measurements of the same property. Where they disagree, read the document. The disagreements are the useful part — they tell you which of your cheap metrics you are allowed to trust unattended.

What to hand in

One table of systems against ROUGE, the faithfulness proxy and your hand audit; the corpus characterisation from step 2; and a paragraph saying which system you would ship, to whom, and what happens when it is wrong.

14.14 Exercises

  1. The oracle first. On any summarization corpus you can reach, compute the oracle-sentence ROUGE-1 before building anything. Say what it implies about extraction, then build lead-3 and see whether the implication held.

  2. Break ROUGE deliberately. Write two summaries of the same article: one true and clumsy, one false and fluent, where the false one differs from a correct summary by a single word (a negation, or a swapped subject). Compute ROUGE-1, -2 and -L for both. Report the ranking.

  3. The tokenizer, in your language. Run rouge_score.tokenize.tokenize over 100 sentences in a language you know that is not English. Count how many words survive intact. If you work in Turkish, predict the answer from Section 14.4 before you run it.

  4. Normalizer sensitivity. Take one system’s output and score it under three normalizers — surface, stemmed, and lemmatized if you have a lemmatizer. Report the spread, and compare it to the improvement claimed by the most recent summarization paper you have read.

  5. Chunk-boundary loss. Take a document long enough to need map-reduce. Summarize it whole, then under map-reduce with chunks that deliberately split a key argument in half. Find a fact present in the first summary and absent from the second, and explain the mechanism.

  6. Audit the references. Run the faithfulness proxy of Section 14.7 against the reference summaries of any corpus you plan to fine-tune on. Validate it on copied text first. Then decide whether you still want to fine-tune on it.

  7. Cost, honestly. For a corpus you actually have, compute the cost of summarizing all of it: prompt tokens, output tokens, wall clock, and the cache hit rate you would expect. Compare against the cost of the extractive baseline, which is approximately zero, and against what the summaries are worth.

14.15 Where to go deeper

Do this before reading a single paper: score a reference summary against itself with your metric. If it does not come back perfect, stop and fix the harness. This check catches the kind of bug this chapter found twice in its faithfulness proxy: the two sides of a comparison processed differently (the chapter found them with the copied-text control, a close relative of this check). It does not catch a tokenizer that damages both sides equally — rouge-score gives a Turkish reference a perfect score against itself — so also print the tokens.

Beyond the papers
resource what it is
Google’s rouge-score read tokenize.py. Its tokenize function is nine lines of code and it explains 44.4% of this chapter
evaluate the metric wrappers most papers use — worth knowing which implementation you are actually calling
Eugene Yan on LLM patterns practical, well-sourced writing on summarization and evaluation in production systems
XL-Sum the multilingual corpus used here, with the collection method documented
Stanford CS224N lectures and slides covering generation and its evaluation
The primary sources
topic source
the metric, as originally proposed Lin (2004)
what it misses, measured against humans Fabbri et al. (2021)
embeddings instead of n-grams Zhang et al. (2020)
an LLM as the scorer Liu et al. (2023); Zheng et al. (2023)
faithfulness, and how often it fails Maynez et al. (2020); Kryściński et al. (2019)
extractive summarization without a model Mihalcea and Tarau (2004)
the corpora, and how extractive each one is Grusky et al. (2018); Narayan et al. (2018); Hasan et al. (2021)
pointer-generator, the pre-LLM standard See et al. (2017)
news summarization after LLMs Goyal et al. (2022)

14.16 What to remember

  • Decide which summary before building one. Indicative or informative, generic or query-focused, single or multi-document, extractive or abstractive. The forcing question is who reads this, and what happens if it is wrong — an 80%-accurate indicative summary is fine and an 80%-accurate informative one is a liability.

  • Compute the oracle before choosing an architecture. Perfect single-sentence selection scores 24.89 ROUGE-1 on this corpus and 40.58 on CNN / DailyMail. That one number, two lines of code, tells you whether the answer is in the document or has to be composed — and therefore whether extraction can work at all.

  • Do not inherit lead-3’s reputation. It scores 40.34 in the literature on CNN / DailyMail and 14.03 here, where lead-2 ties it with one sentence fewer. Always report the lead baseline; never assume it is strong.

  • rouge-score splits 44.4% of Turkish words into fragments. Its tokenizer treats ç ğ ı ö ş ü İ as punctuation, so görüştü becomes ['g','r','t']. The debris matches at random and inflates ROUGE-1 by about 60% — unevenly, from +22% for the oracle to +80% for a random baseline. The rankings survive; comparability does not. Our own implementation needed a Turkish lowercaser for the same class of bug, and agrees with rouge-score to ten decimal places on text that tool can tokenize.

  • A ROUGE number without its tokenizer and normalizer is not a number. Five points of ROUGE-1 separate “surface form” from “5-character prefix” on identical predictions — more than most papers claim over their baselines.

  • ROUGE cannot see whether the summary is true, and on this corpus it actively rewards untruth: the human references assert unsupported content in 33–57% of cases — 95% intervals spanning [26, 65] across the two methods — against 3–7% for a model told to stay in the text. Fine-tuning on such a corpus would be expected to teach a model to hallucinate, and the metric would report that as an improvement.

  • Validate a faithfulness metric on copied text. Lead-2 is quoted verbatim, so under identical preprocessing it should introduce no new words and score zero on this lexical check. That is a tokenizer control, not a guarantee that copied sentences are faithful — lifting a sentence away from the condition or attribution around it can still mislead. That control caught two successive bugs here, both the same mistake: the two sides of the comparison went through different tokenizers. The first version reported BBC journalists as 56% unfaithful.

  • Two faithfulness measurements can agree on the ranking and on almost nothing else. The proxy and the judge flagged 5 and 11 of the model’s summaries and overlapped on one, because one sees only extrinsic errors and the other sees both. Trust the ordering, distrust the rate, and get absolute numbers from a sampled human read with an interval.

  • In this setup, judging costs more than generating. 68.3 seconds per judgement against 2.4 seconds per summary. Sample, do not judge everything, and never let a model judge its own output.

  • Input dominates cost. 890 prompt tokens per article against 53 generated. Stripping boilerplate beats every model change, and a cache keyed on (document, prompt, model, parameters) is close to free money because documents are read far more often than they are written.

  • Showing the source beside the summary is the highest-return feature in most summarization products, and it is not a model improvement. It turns an informative summary, which has to be right, into an indicative one, which only has to be useful.