Uygulamalı Doğal Dil İşleme — Slides – Data, Annotation, and Ground Truth

After this lecture you can

read a sample of your own data and sort what you find into label errors, schema problems and model errors — and act differently on each
write an annotation guideline that two independent people can apply, one that spells out the ambiguous cases and a SKIP option
compute raw inter-annotator agreement and Cohen’s kappa, and explain why a raw agreement as high as 93% is not enough without the class distribution
measure label noise by class and by direction, and assess its effect with a controlled experiment
audit your splits for leaks and, when a leak is found, report how the measurement changes

The chapter’s five learning objectives.

The route of the lecture: first the steps through which labels are produced, and a hand audit of a hundred random reviews. Then annotator agreement, the limits of the schema, label noise and leaks between the splits, each with its own experiment. The last part covers the annotation budget and the alternatives to labelling by hand.

Where the numbers come from: the experiments were run on the Turkish product-review corpus of chapter 4 and on its 40,000/10,000 train/test split. Two results are hand audits: one person, with the criteria written down before the reading began, read 100 randomly selected test reviews and a separate 40 test reviews that the model disputed most strongly. Those counts rest on that reader’s judgement and were not checked by a second human annotator. Say this once at the start; it is the condition on the hand-audit numbers of the lecture.

The code in the chapter is abridged; the full code is in the chapter notebook.

If the chapter is split over two lectures, the natural break is after the slide “Three annotators or more: merging the judgements”.

The number you have been trusting

Chapter 4, round 2: TF-IDF over word 1–2 grams, fed to a balanced logistic regression, scored 92.57% accuracy and 76.21 macro-F1. Bag of words scored 73.51, averaged word2vec 61.39
Every one of those numbers was computed by comparing the model’s output against one column, sentiment
We call the labels in that column the reference labels: the point of reference for evaluation, not a claim to be the one indisputable reading of the text. Gold standard and ground truth name the same function
Who produced the labels, in answer to which question and how, is not documented; chapter 4 did not ask once
This chapter asks. In some experiments the representation, the classifier and the test reviews stay fixed and only the training labels or the test labels change; in others the amount of labelled data, the selection of examples or the split changes too

Each table says which components stay fixed, so a change in score is interpreted only inside the controlled comparison of its own table.

Where A Representation Bake-Off (chapter 4) ended. Its conclusion was that, on that data and with those settings, the sparse representations were well ahead of the from-scratch averaged embeddings, by far more than the sampling uncertainty of the test set. The order of neighbouring rows, though, was sensitive to the tuning budget: raw counts and TF-IDF swapped places once C was chosen on validation. So trying a strong sparse baseline first is a good start; a general claim that one method is better takes new data and a new uncertainty analysis.

Gold standard, as the chapter defines it: the labels treated as correct when a model is scored. They may come from adjudication, from an expert’s decision or from another documented process. Being the reference does not make them free of error; their provenance and the disagreements inside them have to be examined separately.

The corpus is the same one as in chapter 4: 235,165 Turkish product reviews, about 93.7% of them labelled positive. That imbalance strongly shapes many of the results; the schema, the model and the sampling decisions are part of the results too.

Three results, each with its scope

what was observed	the scope it holds in
A 2% error rate in the test labels takes the top entry from 76.21 down to 73.13 — below the 73.51 its bag-of-words rival scored on the original labels	Two conditions with differently labelled test sets, not a rank reversal on a shared one. On the same noisy labels the 2.69-point gap shrinks to 2.50 on average, and the order changes in none of 200 draws
Of the 40 selected test reviews the model disputed most strongly, 20 were classified as texts that a single binary label struggles to describe	One reader, a selected sample: the proportion cannot be carried over to the dataset as a whole
1,000 clean labels tie with 40,000 labels at 10% noise; at 20% noise 500 clean labels are ahead of all 40,000	Artificial noise on one model and one corpus; the tie and the lead were each checked with ten seeds

Not because the model changed, but because the labels the score is computed against did.

Where a label actually comes from

Labels do not arise by themselves. At each step the meaning of the label can drift from what you assume.

step	what happens there	what can break
raw text	scraped or logged	selection — who writes reviews is not who buys (93.7% positive)
schema	what the classes are	schema gap — texts the classes cannot express (20 of 40 read)
guideline	how to decide	ambiguity — the guideline is silent here (human–human κ not measured here)
annotator	reads, presses a key	clear label error — possibly fatigue, misreading, a slip (11 of 40 read)
stored label	one column	collapse — disagreement is not recorded (1 column left)

The 20/40 and 11/40 belong only to the single-reader audit of the 40 test reviews the model disputed most strongly; how often the problems at the other stages occur was not measured in this chapter.

The chapter’s pipeline figure as a table. First the categories and the decision rules are set; then texts are selected, annotators decide, and disagreements are merged. If the steps were not recorded, you cannot tell which decision produced the final label.

The stored label gathers into one column every decision made from the selection of the text to the merging of disagreements; your training run, your leaderboard and your production monitoring see only that column.

The last row is the one to watch. When a dataset is distributed with only a text column and a label column, the history of the annotators’ decisions becomes invisible: disagreements, hard cases and skipped examples are reduced to a single integer. Keeping the disagreements, the hard cases, the skipped examples and the adjudication decisions as well — in extra columns, in a dump of the individual judgements, or in a datasheet — gives a dataset that can be audited far better than a single integer can.

Six questions before you use a dataset

question	what to look at
Who produced the text?	Not “the internet”: which platform, which population, which time window. People who chose to write a review are not the people who bought
Who produced the labels, and how?	Domain experts, crowdworkers, a model, or the platform’s star rating. A sentiment class derived from stars is a proxy labelling decision
What question were they answering?	“Is this review positive?”, “Would you buy this product?” and “Is this customer satisfied?” produce different labels on the same text
What licence, and does it cover your use?	Research-only is a real restriction. So is a licence that permits redistribution of the text but not of derived models
What legal basis and what notice existed?	That anyone can read a review does not show that explicit consent to model training was given
What is missing?	Which texts were deleted, filtered, or never written, and which groups does that affect?

Record what the source does not say as “undocumented”, and assess separately what that uncertainty means for your decision to use the data.

Before any modelling, try to answer these six.

Question two. A proxy label carries the rating scale’s own biases, and how well the stars match the sentiment question you are targeting has to be tested. Crowdsourced labels are not automatically worse than expert ones: Snow et al. (2008) showed that on some tasks the combined judgements of a handful of non-experts can match expert quality. That result is not guaranteed for every task or every way of combining; it depends on the task being one a non-expert can answer at all.

Question five. Examine separately the platform’s terms, the purpose the data was processed for, and the legal basis that could apply.

Question six is the hardest one and the one most worth attempting. Dodge et al. (2021) did it for C4, one of the most-used web corpora in the field, and showed that the blocklist used to “clean” it had disproportionately left out text about and by minority groups. Selection of this kind cannot be seen from the distributed data alone; it takes a record of the collection and filtering decisions.

Exercise 1 asks students to answer the six questions for a public dataset they might use.

Licence and personal data: what an engineer checks

Public does not mean free to use. “I could download it” and “I am permitted to train on it” are different claims; scraped-from-the-web corpora frequently carry no licence at all
What a data licence means for the model is assessed separately. No automatic “the licence travels” rule settles it; it can vary with the jurisdiction and with how much of the data the model memorises. This is unsettled law
Document the provenance: source URL, access and download dates, licence text, data version or file hash, the exact filtering, the splitting decisions
Personal data is a separate regime. Consent is not the only legal basis: KVKK Article 5(2)(d) allows processing data the person has made public themselves — read narrowly, and limited to the purpose it was made public for
Erasure requests — KVKK Articles 7 and 11, GDPR Article 17: “I already trained on it” is not, by itself, an answer. For a high-risk system under the EU AI Act, check Article 10 as well

Longpre et al. 2024

A question that can stop a project after it ships rather than before it starts: are you allowed to use this data at all? Being able to access content does not mean permission to redistribute it or to train a model on it has been documented. Examine the source’s licence, the platform’s terms and the rights in the content separately, for your own intended use.

On licences such as CC BY-NC-SA: examine the licence text, your form of use and the relevant jurisdiction for your concrete case, and be conservative in anything you ship.

Provenance: Longpre et al. (2024) audited 1,800+ popular text datasets and found licence information missing or wrong for a large fraction of them — including datasets used in production. What you did not record at download time becomes hard to reconstruct later.

Personal data: if a review makes its writer identifiable, directly or indirectly, a personal-data regime applies alongside copyright. A review posted to inform other shoppers does not automatically become model-training material. Erasure requests are assessed within the conditions and exceptions of those articles. Meeting a request takes the row-id-to-source mapping; because that mapping can itself carry personal data, keep it for as long as needed and with security measures.

EU AI Act, Article 10, on the governance of training, validation and test data: documented provenance, examination for bias, and a stated data-governance process. The obligations and their dates are in the governance chapter.

Read a hundred rows. Yourself.

100 of the 10,000 test rows, drawn with seed 42 and read with the label visible. Criteria fixed before reading: error, undecidable, ok.

### random 100 test rows, read by hand by one reader

errors (gold contradicts the text)          0
undecidable (binary label not determined)   7
   ... of which "not yet received"          3
arguable (post-hoc; defensible either way)  9
ok (the remaining rows)                    84
                                          ---
                                          100
positive gold labels in the sample         98

arguable was split off from ok during the reading — a deviation from the scheme fixed in advance, so a post-hoc category that carries less weight

Early in a data project, reading a randomly selected sample by hand is a useful audit that needs no code beyond drawing the sample. Set time aside for it: draw 100 rows from your test set, read them with the label visible, and record the ambiguities and the outright label contradictions you see separately.

The draw: np.random.default_rng(42).choice(10000, 100, replace=False).

The criteria. Error: the gold label contradicts the text. Undecidable: a single binary label is not determined by the text. Ok: everything else. During the reading ok was split in two: arguable for rows where the gold label is defensible but the reader would have chosen the other one, and plain ok for the rest. The chapter records the deviation for exactly that reason.

Exercise 3 asks students to reproduce this audit on a dataset of their own, with criteria written down before they start reading.

Zero errors in a hundred: the rule of three

With zero hits, the rule of three gives an approximate one-sided 95% upper confidence bound of \(3/n\): the data are consistent with an error rate up to about 3%, and rule nothing below that out
It is not a measurement that the rate is under 3%: 100 draws cannot distinguish 0.1% from 2%
With a uniform prior, Beta(1,1), the posterior mean is \(1/(n+2) = 1/102 \approx 0.98\%\) — a value that depends on the prior and on one reader’s definition of an “outright error”
Later calculations use about 1% only as an example scenario
98 of the 100 rows were positive, against a corpus rate of 93.7%: to audit the minority class, sample it separately

The seven undecidable rows are not errors

Test row 4745, in full: sipariş verdim umarım tez ulaşır :) — “I placed the order, hope it arrives fast :)”. Gold label: positive.

No product verdict in the text, only a smiley and an expression of hope. Two of the other six are the same shape
One arrived used and scratched, but the reviewer says the product itself is good: the verdict on the delivery points the opposite way from the verdict on the product
Which question the original sentiment label answered is not documented, so we do not call the source label of this row an error

The problem may lie in the definition of the target question rather than in a wrong decision by the annotator. Hold onto this distinction between a label error and a schema problem.

The seven undecidable rows are the interesting ones, because they are not errors. Under the schema proposed in this chapter, which targets the verdict on the product, row 4745 is undecidable: the reviewer has not received the product.

The other six: two are the same shape (“I just ordered it, judging by the reviews…”, “it will probably arrive tomorrow”). One arrived used and scratched. Of the remaining three, one is a proverb about doors, one offers only a suggestion (“it would be better in liquid form”), and one is the text seat leon st 1.6tdi and nothing else.

When the target is a binary verdict on the product, these seven rows fit a single label poorly. Confirming that the problem is in the target definition would take the original schema and a second reader’s decisions. The rest of the chapter keeps returning to this distinction.

The schema can do as much damage as the annotators

A screening rule: mark ama, fakat, ancak and lakin — all roughly “but”. It matched in 2,003 of the 10,000 test reviews (20.03%).

	contrast rule matches	no match
share of test set	20.03%	79.97%
gold negative rate	11.48%	5.03%
model accuracy	84.07	94.70
model macro-F1	72.11	77.72
model F1 (negative)	53.84	58.27

A screen, not a parser: its precision is unmeasured; 20.03% is only the match rate of the rule
5.6 points lower macro-F1 on the matching fifth (bootstrap interval 2.5 to 8.8); schema, model and length are not separated — aspect-level annotation is what decides

The contrastive-connective test does not diagnose a schema problem; it selects reviews to re-read. Whether a matching review really contains two opposed evaluations cannot be known without checking by hand.

The caveat the section rests on. Ancak also means “only” or “barely” — ancak 3 günde geldi is a statement about delivery time, not a contrast — and the rule misses gerçi, ne var ki and buna rağmen entirely. So the 20.03% is neither an upper nor a lower bound; it is only the match rate of the rule. The part that can be checked mechanically was checked: the ambiguous ancak occurs in 351 reviews, 287 of which match through ancak alone, and with ancak taken out of the rule the gap between the two groups stays in place (macro-F1 71.29 against 77.80).

Reading the table: this fifth of the test set has more than twice the negative rate, and the model scores 5.6 points lower macro-F1 and 4.4 points lower negative-class F1 on it. The ten-point accuracy gap overstates this: the two groups have different base rates, so part of any accuracy difference comes from the class mix alone.

The possible explanations: the label schema may be being asked to compress two verdicts into one bit; the model may be making more mistakes on these sentences; and the matching reviews are clearly longer (median 30 tokens against 16). A sample in which the product, delivery and price verdicts are labelled separately is what it takes to test which error comes from which source: if the model recovers both aspects correctly and only the single bit is wrong, the problem is in the schema. If the disagreement comes from the target definition, the fix is a richer schema or an explicit rule in the guideline telling every annotator which clause wins. If it comes from the model reading these sentences badly, a better representation is exactly what helps — and nothing here rules that out.

Annotation artifacts

Annotation artifacts: the labelling process creates a cue between text and label that lies outside the target task
When a guideline pushes annotators toward a habit, models can learn the habit instead of the task, and the score is usually pushed up
Gururangan et al. (2018): a well-known inference benchmark could be partly solved by reading the hypothesis sentence alone and ignoring the premise entirely
The connective match is something else: it defines a candidate group; by itself it is not evidence of an artifact or of a schema flaw in this data

What the two situations share: a label can reflect the labelling process as well as the text.

A related problem, with a name. In the inference benchmark the annotators who wrote the hypothesis sentences had tells, and the model found them.

Geva, Goldberg and Berant (2019) showed, in a data setup where the annotators wrote the examples themselves, that models can identify which annotator wrote an item, and that scores drop when the test set comes from annotators not seen in training. A setup in which ready-made product reviews are only handed out for labelling is different. This corpus has no author or annotator identifiers, so the size of any such effect cannot be measured here.

Keep the two things apart for the students: an artifact pushes a score up through a cue outside the task; the connective rule only marks a group of reviews in which schema problems and model errors are candidates.

A usable guideline: the task and two decided cases

### annotation guideline v1 — product review sentiment

TASK
  Label each review positive (1) or negative (0) according to the
  reviewer's verdict ON THE PRODUCT.

DECIDED CASES  (these make the definition above concrete)
  1. Delivery, packaging, seller, and returns are NOT the product.
     "Fast shipping, product is useless"          -> 0
     "Shipping was late, the product is great"    -> 1
  2. Price complaints are NOT negative unless the reviewer says the
     product is not worth it.
     "Expensive but excellent"                    -> 1
     "Not worth this money"                       -> 0

Class definitions are only the start: decided examples, priority rules and a SKIP option, kept together with a version number

A usable guideline: mixed verdicts, SKIP, negation

  3. Mixed verdicts on the product: the FINAL clause wins.
     "Cuts well but the handle broke"             -> 0
     "The handle is cheap but it cuts very well"  -> 1
  4. Not yet received / no verdict given          -> SKIP (do not label)
  5. Truncated text: if the visible text contains a verdict, label it;
     otherwise SKIP.
  6. Judge negation in context. "fena değil" (not bad) is positive
     for the product. "fena değil ama çabuk bozuldu" (not bad but it
     broke quickly) is 0 by rule 3, because the final verdict on the
     product is negative.

Every decided case has a worked example on both sides, so an annotator matching by pattern gets it right
SKIP exists: forcing a binary choice on undecidable items can manufacture disagreement — or an agreement that looks high and means little
“The final clause wins” is arbitrary. Good agreement does not make a shared answer to the wrong target correct

Two properties make this usable. First, every decided case has a worked example on both sides, so an annotator matching by pattern gets it right without understanding the principle. Second, SKIP exists. Forcing a binary choice on undecidable items can manufacture disagreement you will later blame on the annotators; and if the annotators guess by the same habit, it can also manufacture an agreement that looks high and means little.

The guideline’s last section, “when to skip”: if no decision about the product can be made, choose SKIP. Keep the skipped examples and the reason; they help improve the guideline and the schema. A forced binary label can measure the wrong target even when it looks consistent.

On rule 3: the choice is arbitrary. It can give consistency; whether it really represents satisfaction with the product has to be tested separately. Settle the target question first, then write a decidable and consistent rule for the hard cases: an annotator must always be able to tell which rule applies, and to apply it the same way every time. An arbitrary rule applied consistently produces learnable data, and a principled rule applied inconsistently does not.

Agreement: what does 93% tell you by itself?

No second human annotator was available. A is the reference sentiment column; B is a lexicon rule — 36 positive and 37 negative phrases — applied to all 10,000 test rows.

### A (gold, rows) vs B (lexicon rule, columns), n = 10,000

                B: negative   B: positive
A: negative           141           491
A: positive           167          9201

Raw agreement is \((141 + 9201)/10000 = \mathbf{93.42\%}\) — but 9,201 of that total are shared positive decisions: the majority class pulls the overall agreement up
B is not an independent second opinion: the rule was written by someone who had already seen the reference labels
\(\kappa(\text{A}, \text{B})\) summarises the overlap between the reference labels and the lexicon rule; it is not a measure of human–human agreement or of whether the schema is well defined

Inter-annotator agreement: the extent to which the decisions of people labelling the same items independently coincide. It can be summarised by the raw rate, by class-specific agreement or by chance-corrected coefficients. High agreement does not by itself prove that the labels represent the target concept correctly.

You cannot know your labels are good by looking at one annotator’s output. You need a second opinion on the same items, and a number for how often the two opinions match.

State the condition before any number, because it changes what the coefficient means. For brevity the chapter calls both A and B “annotators”, but neither is a human reading the text independently. The same formula and the same arithmetic — prevalence, the expected agreement, skew — apply to a table of two humans’ judgements; but the sampling and independence conditions of that experiment have to be set up separately.

About the rule: it looks for the phrases as substrings; it does not check word boundaries, and it says positive when a review contains both a positive and a negative phrase. The phrase lists are not printed in the chapter, so the 2×2 table cannot be reproduced from the code shown, while every coefficient derived from the table can.

To see the disagreement on the negative labels you have to look at the first row and the first column of the table separately.

Cohen’s kappa: observed against expected

\[\kappa = \frac{p_o - p_e}{1 - p_e} = \frac{0.9342 - 0.9099}{1 - 0.9099} = \mathbf{0.2698}\]

\(p_e\) is the agreement expected if both labelled independently at their own base rates. A calls 6.32% of items negative, B 3.08%: \(p_e = (0.0632)(0.0308) + (0.9368)(0.9692) = 0.9099\)
The useful reading is a comparison: 93.42 points of observed agreement against 90.99 under independence, leaving 2.43 points above that baseline
Resist turning that into “90.99 points were chance”: \(p_e\) belongs to an independence null, not to a measurable portion of what actually happened
A low kappa on a corpus this skewed is consistent with careless labelling and with careful labelling alike; the coefficient does not tell you which one you have

Cohen 1960

Cohen’s kappa: a coefficient that compares the observed agreement of two label sequences with the agreement expected if each decided independently at its own label rates. A low kappa is not, by itself, evidence of label errors or of carelessness. Nor does a low kappa against a lexicon show that the reference labels are wrong.

The arithmetic, spelled out. A calls 632/10000 of the items negative, B 308/10000. The two products are 0.00195 and 0.90795; almost all of the expected agreement comes from the second term. Do the arithmetic with the real marginals rather than a round number: two labellers at 90% would reach 0.9² + 0.1² = 0.82, not 0.90 — the expected agreement is not the marginal.

Kappa’s denominator, 1 − p_e = 0.09, is the room left for genuine agreement, and the numerator, 0.0243, is how much of that room was actually filled. This is why kappa comes out low on skewed tasks, and why it is volatile there: if the marginals are assumed fixed, a small change in the observed agreement is multiplied by about eleven on its way into kappa. When a cell really changes, the marginals and p_e move as well.

Krippendorff’s alpha gives 0.2655 on the same table. Its closeness to kappa is not an independent confirmation: the two coefficients are computed from the same label pairs.

The number 0.2698 belongs only to this reference-versus-rule table and should not be quoted as a human–human agreement figure.

The same reference and rule, a different sample

All 632 reviews A calls negative, plus a random 632 of the reviews A calls positive: a 1,264-row artificially balanced subsample.

                    full test set (n=10,000)   balanced (n=1,264)
raw agreement                       93.42%              60.28%
p_e                                 90.99%              50.00%
Cohen's kappa                        0.2698              0.2057
Krippendorff's alpha                 0.2655              0.0722

The decision rules of A and B are the same; the texts and the class mix differ. Raw agreement is not a fixed property of the annotators; it changes with the sample
Kappa computes \(p_e\) from each annotator’s own marginals, alpha from a single pooled marginal: on the balanced subsample kappa is nearly three times alpha, from the same data
Report kappa and alpha together with the marginals and the sampling design

Feinstein & Cicchetti 1990 · Cicchetti & Feinstein 1990

The balanced subsample does not estimate agreement on the full test distribution; it shows how the same A and B behave on differently selected texts.

Why the two chance-corrected coefficients part company. On the balanced subsample A is 50% negative by construction and B is roughly 12% — a large divergence in how often the two of them reach for the minority label. Divergent marginals lower Cohen’s chance baseline (here to 0.50, against 0.57 for the pooled marginal), so the same observed agreement counts as further above chance; alpha’s pooled baseline is not lowered by the divergence, so it does not give that credit.

The table illustrates two known situations that affect how kappa is read. On the full set a high raw agreement comes with a low kappa (the prevalence effect); on the balanced subsample the different marginals of A and B widen the gap between kappa and alpha (the bias effect). These are not malfunctions of the coefficient; they are examples of the coefficients using different null models and being sensitive to the composition of the sample.

One number in the table is exact rather than measured: on the balanced subsample p_e is 0.5000 exactly, because when one annotator is a perfect 50/50 split the expected agreement is 0.5 whatever the other annotator’s rate is.

Exercise 5 asks students to build a second annotator of their own and compute kappa on both kinds of sample.

One table, five coefficients

### the identical 10,000-row table, five coefficients
raw agreement  p_o                    0.9342
Gwet's AC1                            0.9277
PABAK  (2*p_o - 1)                    0.8684
Cohen's kappa                         0.2698
Krippendorff's alpha                  0.2655

Not one of these is wrong: kappa and alpha charge you for the skew; PABAK and AC1 do not (Gwet 2008)
The five summaries of this table range from 0.27 to 0.93: settle which coefficient you will use, and why, before you look at the results
Specific agreement shows where the divergence is: 96.55% on positive and 30.00% on negative
B’s 93.42% accuracy against gold is below the majority-class baseline’s 93.68%; on macro-F1 it is 63.27 for B against 48.37 for the constant

Choosing the coefficient after looking at the table can shift the interpretation. The five summaries of the same 2×2 table answer different questions. They differ because they make different assumptions about what “chance” means when one class dominates: kappa and alpha assume annotators guess at their observed rates; PABAK and AC1 assume they do not. Which is right depends on which null you think is the fair comparison for your task, not on diagnosing from the table whether anybody guessed. The coefficients differ in what they assume chance agreement would look like; none of them can tell you what the annotators were doing. A good report gives the confusion table, class-specific agreement and a coefficient justified in advance, together.

Specific agreement, for one class: twice the number of examples on which both chose that class, divided by the total number of times A and B chose it. In this binary table the calculation is numerically identical to B’s class F1 against A. The equality comes from the definition of the metric; it does not imply that B is a human or that A is free of error. These two label sequences overlap almost completely on praise and very little on complaint.

The second lesson in B’s scores: a hand-built lexicon of 73 phrases of careful Turkish performs worse than a constant on accuracy. On macro-F1 the picture is different, which is the whole reason chapter 4 refused to report accuracy alone.

More than two annotators, and the verbal scale

Cohen’s kappa is defined for exactly two annotators
Fleiss’ kappa, for three or more: the generalisation of Scott’s pi rather than of Cohen’s — it computes \(p_e\) from a single marginal pooled across all annotators
Krippendorff’s alpha: \(\alpha = 1 - D_o/D_e\), observed disagreement over expected disagreement, with a distance function you choose; it covers nominal, ordinal and interval labels and tolerates missing judgements
Slight, fair, moderate, substantial, almost perfect: Landis and Koch’s benchmark scale, proposed as “clearly arbitrary” divisions for discussion, not as measured thresholds. A kappa of 0.27 is “fair” on it

Report the number and the interval; treat the adjective as a label for a conversation, never as a passing grade.

Fleiss 1971 · Krippendorff 2018 · Landis & Koch 1977

Three or more annotators is the normal case once a crowd platform is involved. Fleiss’ kappa does not require the same annotators to have labelled every item; it does assume every item received the same number of judgements. The interpretation, and the skew problem, are unchanged for all three coefficients.

The difference between “each annotator’s own rates” and “one pooled rate” is not bookkeeping: it is what separated kappa and alpha on the balanced subsample of the previous slides.

On the verbal labels: they are worth naming precisely because everyone uses them and almost nobody says where they come from.

The chapter has an interactive widget here: sliders for raw agreement, A’s minority rate and the ratio of B’s rate to A’s. The same 93.4% agreement is worth kappa ≈ 0.27 at the rates of this table; it would be worth less at more skewed rates and more at balanced ones. Not every setting of the three sliders describes a 2×2 table that can exist. Show it from the book page if there is time.

Adjudication: a loop you can follow

Two annotators label the same 200 items independently of each other — the same items, not a sample of each
Compute the confusion table and an uncertainty interval for kappa together
Read every disagreement together — not to decide who was right, but to find out why they differed
Add each situation to the guideline as a decided case with examples on both sides. This is the deliverable
Annotate a fresh 200 independently and re-measure

Agreement measurement is diagnostic; adjudication is the step that follows it. Do not treat step 3 as a vote.

Adjudication: an additional review that examines why judgements disagree and records either the decision or the remaining uncertainty. Some disagreements improve the guideline; others show a lasting difference of interpretation.

Step 1. To be able to see minority cases you can stratify the pilot toward the minority class: at this corpus’s 6.3% minority share a plain random 200 contains about thirteen minority items, and most of kappa’s numerator comes from those cells. The kappa of such a pilot belongs to the class ratio you chose. If you want to report agreement on the natural distribution, use a separate, representative sample, or state the sampling design explicitly.

Step 3. Disagreements often cluster into a few recurring situations. Resolving disagreements one by one improves the pilot data. Adding the recurring kinds of disagreement to the guideline can also pay off in the next round of annotation; measure its effect on new, independent items.

Step 5. If agreement did not rise, examine the disagreements again: the cause may be a gap in the guideline, the guideline being applied differently, a change in the sample, or a limit of the schema.

This loop is the chapter’s lab, in small: two balanced rounds of 100, guideline v0, guideline v1.

The kappa interval is not optional at this size

rng = np.random.default_rng(42)
idx = (rng.integers(0, len(a), len(a)) for _ in range(2000))
boot = [cohen_kappa_score(a[i], b[i]) for i in idx]
lo, hi = np.percentile(boot, [2.5, 97.5])

sample	standard error of kappa	what follows
10,000 items	about 0.02	—
a plain random 200	about 0.14	the 95% interval around 0.27 is roughly \([-0.02, 0.54]\)
stratified: 100 gold-negative plus 100 gold-positive	about 0.04	the point estimate moves to around 0.21

End the pilot loop when the lower bound of the interval clears the target; an interval that contains the target is not evidence in either direction

These numbers come from resampling the reference-versus-lexicon table; in your own pilot with two humans, compute the interval separately.

Three annotators or more: merging the judgements

Sending every item to a third person is expensive at scale; most projects combine a majority decision, a reliability model and expert adjudication of selected disagreements
Majority vote: a simple starting method. If error rates differ markedly, or errors depend on one another, equal weights can lose information
Latent-variable aggregation: estimates each annotator’s error pattern and the hidden class with a joint probability model, by EM; reference labels are not required
Keep the vote distribution and train against a soft target: an item two of three annotators called positive gets 2/3 instead of a hard 1 — not, by itself, a calibrated probability

How you reduce several judgements to one label is a choice, and keeping the raw judgements lets you change that choice later.

Dawid & Skene 1979 · Uma et al. 2021 · Plank 2022

The pipeline table at the start named collapse — the record of disagreement not being kept when the judgements are merged into one label — as a problem of its own, without saying what to do about it. With a crowd you have several judgements per item.

EM, in one breath: expectation–maximisation — guess the true labels, re-estimate each annotator’s error rates from the guesses, re-guess, and repeat until nothing changes. The result depends on the annotators’ judgements overlapping enough and on the model’s assumptions. Snorkel-style weak supervision also models the reliability of its sources; there the dependence between the labelling functions and the model family have to be specified separately.

On the soft target: whether the disagreement has to do with the task has to be read separately. In this chapter half of the 40 test reviews the model disputed most strongly were classified as texts that fit a single binary label poorly; on such items a 2–1 split can carry information that a hard label deletes.

If the chapter is taught over two lectures, this is the place to stop.

Auditing labels at scale: out-of-fold flags

Reading all 40,000 rows by hand is expensive. Set up a procedure that chooses which rows to read first.

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof = np.zeros(len(y))
for train_idx, val_idx in skf.split(Xtr, y):
    m = clf().fit(Xtr[train_idx], y[train_idx])
    oof[val_idx] = m.predict_proba(Xtr[val_idx])[:, 1]
# the score the model gives to the opposite class;
# not the probability that the label is wrong
wrong_conf = np.where(y == 1, 1 - oof, oof)

Fit on four folds, predict the fifth, repeat: every training row gets a prediction from a model that never saw that row’s label
A simple threshold rule inspired by confident learning, not the method itself: the full method thresholds each class separately

Northcutt et al. 2021

Out-of-fold prediction is used here to produce a review priority. Rows where the model gives a high score to the other class are not automatic verdicts of error; they are candidates to be read by hand.

Confident learning uses the model’s own confidence to find candidate label errors. The full method does not use a single global threshold: it thresholds each class at the average probability the model assigns to that class over the examples labelled with it, and builds a count matrix (the confident joint) from those thresholds. The single 0.90 threshold on the next slide is a shortcut, and per-class thresholds exist to correct exactly this when classes are so imbalanced.

The code on the slide is abridged from the chapter. One thing to say out loud: in the chapter’s run Xtr comes from a vectoriser fitted on all 40,000 rows, before the folds. For a fully out-of-fold prediction, fit the vectoriser inside each training fold too; a Pipeline does that by itself. The next slide shows what that changes.

What the thresholds flag

threshold	rows flagged	share	gold positive	gold negative
> 0.50	3,026	7.57%	2,232	794
> 0.90	92	0.23%	49	43
> 0.95	34	0.09%	18	16
> 0.99	2	0.005%	2	0

The flags are class-balanced, the corpus is not. At 0.90, 43 of 92 flags (47%) are on gold-negative rows in a corpus that is 6.3% negative: 1.70% of negative labels against 0.13% of positive ones. A difference in flag rates, not error rates
The high threshold flags very few rows, and the rule is blind to errors on texts the model itself finds hard: rows that are not flagged cannot be declared clean
A flag is a request to read, not a verdict

With the vectoriser also fitted inside each fold, the 92 rows above 0.90 become 123 and the 34 above 0.95 become 47; 89 of the first 92 candidates are in the new list too. What follows belongs to the first list of 92.

The 40 test reviews the model disputed most

A separate audit, on the test set: the 40 reviews on which the model disputed the stored label with the highest score.

### 40 highest-confidence gold/model disagreements, test set, read by hand

error   (gold label is wrong)                        11    27.5%
schema  (binary label cannot represent the text)     20    50.0%
model   (label defensible, classifier wrong)          9    22.5%
                                                     --
data-side problems                                   31 / 40

schema breakdown:  mixed 12 · truncated 4 · lukewarm 3 · no verdict 1

A selected sample, read by a single unblinded reader: not an estimate of a rate for the corpus or for the 92 training candidates
95% Wilson intervals: schema from about 35% to 65%, error from 16% to 43% — counting uncertainty only; they do not correct for purposive selection

The 92 candidates of the previous slide are in the training set. These 40 are in the test set, where a wrong label costs you a wrong leaderboard rather than a slightly lower-scoring model. The two selections have different sampling frames; the proportions of one cannot be carried over to the other.

The three categories: error — the gold label clearly contradicts the text; schema — a single binary label cannot represent the text; model — the label is defensible and the classifier is wrong.

Of the 40 selected candidates, 31 show a problem on the data side according to the single reader. The two Wilson intervals overlap. The reading was done by the author, who had written the hypothesis. The same forty items should be read by a second, blinded reader and the agreement reported; that step tests the subjective classification, and does not by itself solve the problem of generalising. The chapter says so in the first person: I did not do that, and you should before you quote the number.

The Wilson interval is a confidence interval for a proportion that stays sensible at small n.

Four of the forty

row	gold	what the text says	how it was read
6874	negative	clear praise, no visible reservation; model p(positive) = 0.984	plainly contradicts the negative label
8356	positive	“yastıklar ince yeterli değil” — “the pillows are thin, not enough”; p = 0.059	another clear contradiction
8006	positive	praises the returns process and reports “I did not like it” as the reason for the return; p = 0.034	counted as a model error; under guideline v1 the label would be 0, or the row `SKIP`
2544	positive	“paketleme başarısız. ampüller fena değil” — “packaging failed. the bulbs are not bad”	two verdicts of opposite polarity about two different aspects

The category depends on the definition of the target: read the counts as a direction rather than a rate.

Row 6874: at which stage the error arose is not known.

Row 8006, in full: “without giving any reason, following the return rules, I said I did not like it and returned it. my refund was made at once. the shipping fee too.” The review gives no positive verdict on the product. The largest negative contributions to the model’s decision score come from the features iade (“return”, −1.78), beğenmedim (“I did not like it”, −1.30) and iade ettim (“I returned it”, −0.65); about half of the negative contributions come from features containing iade. Under a “was this a good experience” target the positive label is defensible, which is why the row was counted as a model error. Under the product-targeted guideline v1 the returns process is not the product (case 1): if “I did not like it” counts as a product verdict the label would be 0, and if it does not the row is SKIP (case 4).

Row 2544: which label fits cannot be decided before the target and the rule for mixed reviews are set.

On negation: five of the nine selected model errors involve negation — fena değil (“not bad”), özelliklerinin altında bir ürün değil (“not a product below its specifications”), and one triple negation. A model that uses word unigrams and bigrams can learn frequent patterns such as fena değil as a separate feature; but it does not structurally represent the scope of a long or nested negation. On row 2544 the contribution of the bigram fena değil to the decision score is only −0.06, while the unigrams değil (−1.77) and fena (−0.55) dominate. These examples show the limit of this feature setup, not that every n-gram model will get every negation wrong. Models that represent word order and scope are the subject of later chapters.

What cleaning actually bought

### repairs, judged on the test split used throughout the chapter

                          acc    macro-F1   F1(neg)
baseline (no repair)     92.57      76.21     56.47
drop 92  (p > 0.90)      92.37      75.72     55.61
flip 92  (p > 0.90)      92.47      75.94     55.99
drop 34  (p > 0.95)      92.39      75.83     55.83
flip 34  (p > 0.95)      92.52      76.03     56.15
drop  2  (p > 0.99)      92.54      76.15     56.37
flip  2  (p > 0.99)      92.52      76.11     56.31

None of these six repair conditions beat the baseline on the test split used. The largest difference is about half a macro-F1 point (dropping the 92 rows: −0.49).

Given 92 suspect rows, the obvious next move is to remove them or flip them and retrain. Both conditions were compared on the same test split used for reporting throughout the chapter. This is an exploratory comparison; it should not be read as a final result on a never-seen test set.

These are the differences observed for the fixed sets of rows that were tried; they are not a confidence interval for the benefit of the method, nor an upper bound that rules out a large gain. Nor does the seed deviation of other experiments stand in for the uncertainty of these deterministic repairs. The right comparison is to apply the same operation to the same number of random rows — the next slide.

Flipping (which keeps the row) edged out dropping (which removes it) at the 92- and 34-row thresholds, and the order reversed at 2 rows; such small differences are not evidence about why keeping the rows would be better. Read the rows before either operation.

The control: the same operation on 92 random rows

92 random rows with the same class mix as the 92 flags (49 positive, 43 negative), 20 separate draws.

                            macro-F1     sd      min      max
drop 92 random rows  (mean)    76.15   0.13    75.91    76.45
flip 92 random rows  (mean)    75.87   0.36    75.01    76.47
drop the 92 flagged rows       75.72
flip the 92 flagged rows       75.94

Dropping the flagged rows (75.72) is below even the lowest of the 20 random draws (75.91). Flipping them (75.94) cannot be distinguished from the mean of flipping 92 random labels (75.87)

Producing a review list and raising the score by dropping or flipping that list wholesale are separate goals. A flag is a reading list, not a decision.

So the rows selected by the model’s disagreement do behave differently from random rows, but not in the expected direction of raising the score. 92 rows out of 40,000 is 0.23% of the training data, and editing that few rows is not expected to move a model by much: in the control, dropping 92 random rows moves the score by 0.06 points on average. That the flagged rows move it more (0.49 points) shows that rows near the decision boundary can matter more than their count suggests.

The flagged rows may not all be of one kind. Of the 40 selected disputes in the test set, 20 were schema problems according to the single reader; for the 92 training candidates that proportion is unknown, and those rows were not read separately.

The concrete return of this audit is not the retrained model; it is that schema problems became visible in the 40 selected disputes. That gives a reason to re-examine the guideline; it does not show that half of the confusion in the whole corpus comes from the schema. The automated tool finds the candidates; it is the human reading them who revisits the definition of the target.

In separate work, Northcutt, Athalye and Mueller (2021) found label errors in the test sets of ten of the most-used ML benchmarks and showed that model rankings can change when the test labels are corrected: scored against corrected labels, on ImageNet a ResNet-18 overtakes a ResNet-50 once the originally mislabelled examples make up just 6% more of the test set. No such change of ranking was measured in this chapter.

What label noise costs, and in which direction

Training labels corrupted at controlled rates, the model retrained, scored on the original test labels. Rates are shares of the source class; “symmetric” flips that share of each class.

### macro-F1, clean test set, model retrained at each rate

flip rate      0%     1%     2%     5%    10%    20%    30%    40%
symmetric   76.21  74.18  72.43  68.58  63.34  55.43  49.86  43.71
neg -> pos  76.21  76.14  76.29  76.27  76.42  76.85  77.12  76.47
pos -> neg  76.21  74.17  72.87  68.63  63.63  56.69  52.01  47.99

The middle row barely moves: with 40% of the negative labels flipped to positive, macro-F1 stays close to the baseline. The bottom row loses 28 points over the same range
The same percentage is not the same number of labels: 40% of the negative class is about 1,000 rows, 40% of the positive class about 15,000

The effect of a label error depends jointly on its direction, on the sizes of the classes and on how many rows change. Three seeds each; the standard deviation never exceeded 0.50 macro-F1.

Where the small rise in the middle row comes from: at 30% it is 0.91 above the baseline. In that condition the model calls fewer reviews negative (about 845 instead of 1,075); precision on the negative class rises from 44.8% to about 50%, and recall falls from 76.3% to 67%. So as negative training labels get scarcer the decision boundary shifts, and on this test split macro-F1 rises a little. That is not a recommendation to corrupt labels.

The chapter draws these three rows as a line chart; the slide keeps the numbers. The 93.7/6.3 class imbalance is an important factor in this asymmetry, not its only explanation.

Count the rows, not only the percentage

condition	rows flipped	the class afterwards	macro-F1
neg → pos at 10%	253 of 2,529 negatives	the positive class grows to 37,724, of which 253 are wrong: 0.67% contamination; the negative class shrinks to 2,276, every member correct	76.42
pos → neg at 10%	3,747 of 37,471 positives	the negative class is 2,529 correct plus 3,747 wrong: 59.7% mislabelled	63.63 (−12.58)
pos → neg at 1%	375 rows, 0.94% of all training rows	12.9% of everything now labelled negative (375 of 2,904)	74.17 (−2.04)

Do not close uncertainty with a guess: send an uncertain item to SKIP and to adjudication

Measure your annotation error rate per class. Name the source class and the target class every time, and give the number of rows along with the percentage.

The same source-class percentage changes very different numbers of labels when the classes are imbalanced.

Row one: a smaller, pure minority class is still learnable.

Row two: the model is fitting a class whose majority membership is noise. The true negatives still form a coherent cluster, so the model still learns something (63.63, against 48.37 for the majority baseline), but the signal is diluted in the noise.

Row three: one percent of the positive class is a rate most teams would call excellent. It costs 2.04 macro-F1 points — more than the entire gap between bag of words and TF-IDF unigrams in chapter 4, which was 1.56.

An overall 1% is not informative by itself: errors that turn minority rows into majority ones barely registered here, while errors that turn majority rows into minority ones cost 12.58 points at the same 10% class-conditional rate.

The same number of flipped labels, both directions

labels flipped	neg → pos	pos → neg	share of the negative class	share of the positive class
250	76.35 (sd 0.17)	74.98 (sd 0.25)	9.9%	0.67%
500	76.80 (sd 0.20)	73.64 (sd 0.20)	19.8%	1.33%
1,000	76.95 (sd 0.68)	71.11 (sd 0.27)	39.5%	2.67%
2,000	68.60 (sd 1.02)	67.91 (sd 0.15)	79.1%	5.34%

The direction still matters: flipping 1,000 positive labels to negative loses 5.1 points, while flipping 1,000 negative labels to positive does not lower the score
The asymmetry has a limit: with 79% of the negative class flipped, that direction loses 7.6 points too

The flips were chosen independently of the text. Real errors can concentrate on hard items, and the effect of the same nominal rate then differs (Wei et al. 2022): do not read these numbers as a bound.

Mean macro-F1 over three seeds. Because the percentages mean different numbers of rows in the two directions, the chapter also ran the comparison with the same number of labels.

On the second of the chapter’s two suggestions, not closing uncertainty with a guess. In this experiment the examples wrongly moved from the majority to the minority were the expensive ones; but it does not follow that an annotator should systematically avoid the minority decision — that raises the number of missed minority cases. Where a missed minority case is the expensive error — screening, safety, fraud — the calculation comes out differently. Which direction does more harm in spam, fraud, medical screening or content moderation does not follow from this table.

On the small note. Frénay and Verleysen (2014) separate noise independent of everything, noise depending only on the class, and noise depending on the instance. This experiment’s symmetric regime is an example of the first and its two one-directional regimes of the second; the cost of the third was not tested here. The hard reviews the model flagged in the audit do not show the distribution of verified label errors; they only suggest that such concentration is possible.

Exercise 4 repeats the noise-direction experiment on a balanced dataset.

Test labels: one fixed model, corrupted labels

Now the reverse: one model, trained once on the original data, scored against test labels flipped in a controlled way.

### one fixed model, corrupted test labels, mean of 5 seeds

test noise      acc   macro-F1   F1(neg)   sd(macro-F1)
### noise here = share of ALL test rows, flipped uniformly at random
    0%        92.57      76.21     56.47          0.00
    1%        91.73      74.68     53.90          0.23
    2%        90.85      73.13     51.31          0.12
    5%        88.31      69.41     45.36          0.17
   10%        84.19      64.79     38.66          0.24
   20%        75.48      57.00     28.82          0.36

A “clean” or “original” test label means no artificial noise was added; it does not mean a label independently verified as true

Scores against different versions of the labels

system	macro-F1
TF-IDF 1–2 gram, original test labels	76.21
TF-IDF 1–2 gram, 1% test-label noise	74.68
bag of words, original test labels	73.51
TF-IDF 1–2 gram, 2% test-label noise	73.13
TF-IDF unigram, original test labels	71.95
TF-IDF char_wb 3–5, original test labels	70.39

Two percent artificial noise takes the same model from 76.21 to 73.13, below the 73.51 of bag of words on the original labels. Do not conclude that the ranking has reversed: the two are not compared on a common version of the labels.

Does the order survive on the same noisy labels?

In expectation, for independent symmetric flips at rate \(\eta < 0.5\), it does: a classifier’s noisy accuracy is \(\eta + (1 - 2\eta)\,a_{\text{clean}}\), which is increasing in \(a_{\text{clean}}\). On one realised test set it need not be
Measured for this chapter’s pair: bigram TF-IDF and bag-of-words scored 200 times against the same noisy test labels
The 2.69-point gap between them shrinks to 2.50 on average at 2% noise and to 1.36 at 20%; the order reverses in none of the 200 draws
That result holds for uniform, independent flips; not for real errors concentrated on hard rows

“Which test set, with which labels” belongs in every comparison.

What a small gap means

The test labels determine how the score is read. If each true label is flipped independently with probability 2%, the expected accuracy of a model that is perfect against the truth is 98% — not a hard upper bound for a single run
The 1.53 points at 1% noise are a systematic shift of one model’s score (seed standard deviation only 0.23); on the same test set that shift applies to both models
What makes a small gap fragile is how many label errors sit on the rows where the two models disagree
The worst case, measured: chapter 4’s 1.2-point gap is erased by changing the 18 most damaging labels among the 231 rows the two models disagree on; for the 2.69-point gap the number is 41. With randomly scattered errors the gap survives
Read the disagreement rows, blind if you can, and test the gap on a common version of the labels with a paired bootstrap (McNemar for accuracy)

Set aside a separate quality budget for the test labels.

Recht et al. 2019 · Beyer et al. 2020

Three consequences for evaluation design.

First: a model that has learned to reproduce the errors your annotators made can report a higher score. Once your model’s error rate approaches your labels’, a high score does not by itself show closeness to the real target, and further improvements can become invisible.

Second: small differences on a noisy benchmark are not results until you have read the rows the two models disagree on. Get the reasoning right. Under independent symmetric noise the expected accuracy gap shrinks only by the factor (1 − 2η), that is, to 0.98 of itself at 1% noise; for macro-F1 the relation is not linear, but the shift still moves the level, not the gap. Real label errors can cluster on the hard rows where models disagree. On this chapter’s test set, changing the label of one row moves one model’s macro-F1 by 0.02–0.045 points; the effect on the gap between two models depends on which rows they disagree on and on their predictions, and there is no fixed threshold. The 1.2-point gap is the one between tuned TF-IDF (74.71) and raw counts (73.51). The 18 and the 41 are extreme cases in which the errors fall in the worst places. Recht et al. (2019) and Beyer et al. (2020) document what happens when this is taken seriously on a famous benchmark.

Third: independently examining the rows on which models disagree, and the rows of the minority class, is especially worthwhile. Plan quality and sample size for training, validation and the final test from the costs of your own task; a single 10,000/40,000 allocation is not a general rule.

Leakage: the errors that make you look good

An evaluation risk separate from wrong labels: information that should not be seen during training, or the test examples themselves, reach the model, and the reported score may not represent the intended conditions of use
Label leakage: target information the model will not have at the time of use gets into the input. The same text sitting in both training and test is contamination between the splits, under the wider term data leakage
Noise does not always lower the score, and leakage does not raise it under every condition: the direction has to be measured with a controlled comparison
This corpus: 235,165 rows. Exact de-duplication on the text removes 1,657 extra rows (0.70%); 1,055 distinct texts occur more than once (2,712 rows in all), and 77 of those groups carry both labels

Splitting before de-duplicating

### split-before-dedup, the ordinary careless mistake

rows removed by exact de-duplication           1,657   (0.70%)
test rows also present in train                   45   (0.45% of test)
near-duplicates after normalisation               34   (0.34% of test)

same model, same run:
  accuracy on the 45 leaked rows                93.33
  accuracy on the 9,955 clean rows              91.90
  macro-F1 on the leaked rows                   76.76
  macro-F1 on the clean rows                    74.08

The 2.68-point gap cannot be read as the causal gain of memorisation: 42 of the 45 rows are positive and their median length is 2 tokens (19 for the rest)
A 0.45% contamination is far too little to move the headline number
The 34 near-duplicates score 67.62 macro-F1 against 76.23 on the rest: on thirty-four rows, close to meaningless

Chapter 4 de-duplicated before splitting. Suppose it had not: this is that run.

At n = 45 macro-F1 is volatile. In 5 of the 45 rows the copy in training carries the other label. Nor can the 74.08 here be compared directly with the 76.21 baseline used elsewhere in the chapter: this run deliberately did not de-duplicate before splitting, so it is a different split. The presence of contamination is shown by matching rows; its effect on the score, by a shared evaluation design.

The near-duplicates: on the de-duplicated split of chapter 4, normalising away case, punctuation and whitespace finds 34 test rows whose normalised form also appears in training. On such a small, selected subset the reason for the gap cannot be determined without examining the texts one by one. In the chapter’s recount (32 rows, depending on the details of the normalisation) 31 of these rows are positive and their median length is 3 tokens (“güzel ürün”, “tavsiye ederim”, “teşekkürler hb”); so the F1 of the negative class is decided by a single row.

What leakage is worth, at controlled doses

test rows copied into train	accuracy	macro-F1	F1 (negative)
0%	92.57	76.21	56.47
1%	92.57	76.27	56.60
2%	92.62	76.41	56.86
5%	92.65	76.53	57.08
10%	92.82	77.14	58.20
25%	93.33	78.59	60.82
50%	94.00	80.89	65.07
100%	95.62	85.89	74.17

The 10% dose gives a difference of the size you would expect from a genuine modelling improvement. It is specific to this model, this dataset and this copying setup: not a fixed gain per 10%, nor an estimate of the effect of a natural leak.

To see how the effect changes with the dose, a controlled fraction of the test rows is copied into the training data; the vectoriser is refitted on that enlarged training set as well. This is an artificial contamination: it need not produce the same distribution, or the same effect on the score, as natural splitting mistakes.

The 10% condition takes macro-F1 from 76.21 to 77.14, and the 100% condition to 85.89 (9.68 points — more than twice the 4.3 points that moving from unigram to bigram TF-IDF bought in chapter 4). Which rows are copied moves the result too: re-running the 10% dose with three different draws, macro-F1 ranged from 76.81 to 77.13.

Controlled copying can produce an increase in score that looks plausible, and it cannot be told apart by looking at the score alone. So verify an unexpectedly good result with checks on the split, on duplicates and on preprocessing.

Six leaks worth checking

risk	the check
Exact duplicates across splits	Before splitting, group by the text; examine same-text, different-label conflicts first (77 groups in this corpus), then de-duplicate by a rule you have documented
Near-duplicates	Normalise case, punctuation and whitespace, then compare; for longer texts, Jaccard similarity over word n-gram “shingles”, or MinHash
Group leakage	The same user, the same document, the same patient: split on the group, not the row. This corpus has no reviewer identifiers; report that limitation
Temporal leakage	If your model will run on future data, your test set must be from the future
Preprocessing leakage	Fit the vectoriser, the scaler and the feature selector on the relevant training part only — inside every fold too (92 → 123 in this chapter’s audit)
Benchmark contamination	Public test texts may have entered the pretraining data of the pretrained model you are fine-tuning

Kapoor & Narayanan 2023 · Magar & Schwartz 2022 · Oren et al. 2024

Check these six risks separately, against your own data and your own setup of use.

Exact duplicates: df.drop_duplicates(subset="text") keeps the first row of each text; or make sure each group stays in a single split.

Near-duplicates: Jaccard similarity is shared shingles divided by all shingles; MinHash is a fast approximation of it that scales to millions of documents.

Group leakage is easy to miss because the rows genuinely differ. Temporal leakage: a random split lets the model see next month while predicting last month.

Preprocessing: on the test set, only transform — chapter 4’s pipeline had vec.fit_transform(train) and then vec.transform(test).

Benchmark contamination: Kapoor and Narayanan (2023) catalogue kinds of leakage across 17 scientific fields; Magar and Schwartz (2022) measure, with controlled pretraining, the effect of contamination on scores; Oren et al. (2024) propose a statistical test of whether a specific dataset is in a black-box model’s pretraining data. Report the assumptions and error probabilities of such a test; do not read its result as a definitive provenance record for individual texts. In LLM evaluation this becomes the central problem.

Exercise 7: take a benchmark result you believe and audit it for all six.

Where to spend the next thousand labels

On a fixed annotation budget you choose between the number of examples and the cost of producing and checking each label: careful labels cost more, so you get fewer
A cost example: at 200 short reviews an hour, 40,000 labels take about 200 person-hours. Measure the real speed in a small pilot before planning
Pay and time targets set the working conditions and can affect an annotator’s attention
Task design is a working condition: a SKIP option and decided examples can keep an annotator from being forced into a binary decision on an ambiguous case
On some tasks — moderation, safety, abuse detection — exposure is a real cost: budget rotation and support, not only annotator hours

Record in your datasheet who labelled, how they were paid and at what rate, stating the unknowns explicitly as well.

Gray & Suri 2019 · Sambasivan et al. 2021 · Gebru et al. 2021

The budget grid: number of labels against noise

The same model at seven budget sizes and six noise levels, three seeds each; all 42 cells are scored against the same original test labels.

### macro-F1 by budget and label noise (clean test set, 3 seeds)

                 0%      5%     10%     20%     30%     40%
    500       59.44   58.56   55.57   51.18   47.40   42.12
  1,000       63.71   60.91   57.77   52.32   46.99   42.78
  2,500       68.76   64.17   60.03   53.36   48.23   42.93
  5,000       71.69   66.00   60.21   53.17   47.86   42.42
 10,000       73.72   66.47   61.81   54.46   48.53   42.88
 20,000       74.89   66.99   62.04   55.21   49.09   43.81
 40,000       76.21   68.18   63.20   55.74   49.34   44.16

Read across a row for what noise costs, down a column for what volume buys, and diagonally for the finding

In each cell the training rows are drawn at random, the vectoriser is fitted on those rows only, and the labels of each class are flipped at the same rate. In chapter 4’s learning curve the vectoriser had been fitted on all 40,000 reviews, so the two curves cannot be compared directly — there bigram TF-IDF averaged 51.7 at 1,000 labels.

The grid does not separate the direction of the noise: it is a separate experiment for noise that corrupts each class at the same rate. The last row is close to the symmetric row of the earlier table (68.18 against 68.58 at 5%, 63.20 against 63.34 at 10%), but it comes from separate runs. Re-running the same cells with ten seeds each gave a standard deviation of about half a point (68.38 ± 0.48 at 5%; 62.66 ± 0.46 at 10%), so these differences are at the level of seed-to-seed variation. If you know the direction of your noise, use the earlier table.

The book has an interactive version of this grid: choose a number of labels and an error rate, and it shows whether a cell with fewer labels reaches at least the same score. Monetary cost is not part of that calculation. On each curve the high-noise lines flatten very early.

Reading the grid diagonally, with ten seeds

pair	in the grid (3 seeds)	re-run with ten seeds each	reading
1,000 clean labels against 40,000 labels at 10% noise	63.71 against 63.20: a difference of 0.51 points	62.24 (sd 2.24; 59.29–65.44) against 62.66 (sd 0.46)	a tie: the small clean set is ahead in 37 of the hundred seed pairs
500 clean labels against 40,000 labels at 20% noise	59.44 against 55.74	a difference of 3.9 points	in the same direction in all hundred seed pairs

The score of the small set depends heavily on which 1,000 rows were drawn
What volume buys depends on the label noise: going from 500 to 40,000 labels brings 16.77 macro-F1 points at 0% noise and 2.04 at 40%

The grid shows the balance of quality and quantity. The return per unit of money takes the costs of annotation and quality control, which have to be known separately.

Not all labels are worth the same

m0 = clf().fit(X_seed, y_seed)                       # 200 random labels
unc = np.abs(m0.predict_proba(X_pool)[:, 1] - 0.5)   # distance from 0.5
picks = pool[np.argsort(unc)[:800]]                  # the 800 closest to 0.5

### 1,000 labels, spent two ways

                              negatives found    acc   macro-F1   F1(neg)
1,000 drawn at random                      62  93.50      65.49     34.40
200 random + 800 uncertain                248  93.59      68.80     40.99

Uncertainty sampling (Settles 2009): fit a model on a small seed set and choose the next labels from the examples that model is least decided about
248 negatives against 62, four times as many; 68.80 against 65.49 macro-F1, an observed difference of 3.31 points (mean of three seeds)
Do not generalise it as a definite effect size: a re-run with our own protocol gave 5.3 and 5.2 points in two groups of three seeds. The direction is the same; the size is sensitive to the protocol and to the seed

The grid assumed labels are drawn at random. They need not be. Uncertainty sampling is one of the common methods of active learning.

The code is schematic: X_seed, y_seed, X_pool and pool have to be prepared beforehand. It is not a runnable experiment by itself; it summarises the workflow. The lowest unc selects the examples whose model score is closest to 0.5; the calibration of the scores was not checked.

On the negatives: finding four times as many may have improved coverage of the minority class; but the selection also changes how hard the texts are and how close they lie to the decision boundary. How much of the 6.59-point difference in negative-class F1 comes from the number of negative examples cannot be said without a separate comparison.

On the re-run: the seed model used the vectoriser of the 40,000 texts, the final model a vectoriser fitted on the 1,000 selected rows. The two groups of three seeds gave 63.13 → 68.46 and 61.96 → 67.19; the number of negatives found averaged between 188 and 290 depending on the group. Paired differences per seed are not reported for the first table.

Active learning: three limits

“1,000 uncertainty-sampled (68.80) matches 2,500 random (68.76)” is the more quotable claim and the weaker one: the two tables do not share a baseline — random 1,000 is 63.71 in the grid and 65.49 here
A training set chosen by uncertainty does not represent the natural test distribution: never select the test set this way
How small a seed set is enough depends on the task and the class ratio; 20 and 200 seed labels were not compared here
The selected set can be tied to the model that selected it: a set chosen for a linear TF-IDF classifier is not necessarily a good set for the transformer you fine-tune later

Lowell et al. 2019

Labels you do not write by hand

method	what you do	what to watch
Weak supervision	write partial, noisy labelling functions and combine them; the lexicon rule in this chapter is one such source	dependence: five variations on the same keyword list should not count as five independent pieces of evidence
Distant supervision	use an existing signal as a proxy label: star ratings, a knowledge base, user clicks	the proxy is not the target: a 3-star review is not “neutral”
LLM labelling	prompt a large model to label your data	the model’s systematic biases; labels that may not be independent of a model you then evaluate

These methods can reduce the need for human labels; they do not by themselves take the place of quality control. You still need a human-labelled test set to know whether any of it worked.

Ratner et al. 2017 · Gilardi et al. 2023 · Ziems et al. 2024 · Panickssery et al. 2024 · Pangakis et al. 2023

Hand annotation is not the only source of labels, and on a 235,000-row corpus it takes a great deal of labour; choose its scope from your budget and your goal.

Weak supervision: frameworks like Snorkel fit a generative model over many functions; under the chosen model assumptions it estimates the accuracy of the sources and the dependencies between them from the functions’ agreements and conflicts, and it needs no reference labels. An aggregation model that assumes independence trusts a “consensus” of near-identical functions too much. Test the dependence and the coverage of the functions on a small sample with verified labels.

Distant supervision is cheap and abundant. A 3-star review is whatever that platform’s users mean by three stars, and the mapping shifts across platforms and across years.

LLM labelling: on some well-specified tasks, labelling by prompt has matched or beaten non-expert crowdworkers (Gilardi et al. 2023); on subjective or culturally loaded tasks the margin shrinks and sometimes reverses (Ziems et al. 2024). In the cost comparison, count the labour of checking and correcting as well as the model calls. Three things to hold on to. First, LLM labels inherit the model’s systematic biases; those do not disappear by collecting more rows, and their size and direction have to be measured on your task. Second, if the same model family labelled your test set, your evaluation partly measures agreement with that family, and models have been shown to prefer their own output (Panickssery et al. 2024). Third, treat LLM labels as an unvalidated instrument and report accuracy and disagreement on a held-out sample that humans have examined independently (Pangakis et al. 2023).

One more caution for Turkish: the measurements above were made on English data; on user text with misspellings and missing Turkish characters the result may differ, so make this measurement on your own Turkish data.

Synthetic data and the quality filter

Generating training text with an LLM (Self-Instruct and its descendants) is a common option; its benefit depends on selection, filtering and comparison with real data as well as on the generation method
Lack of diversity: you can ask for 10,000 negative reviews and get variations on a small number of templates
Distributional mismatch: generated text is fluent, well-punctuated and correctly spelled; real Turkish product reviews are none of those things
Recursive contamination: training on the output of models trained on model output degrades over generations, and the tails of the distribution disappear first. That result replaces the real data at each generation; when real data is accumulated alongside the synthetic, the collapse largely disappears

The main test set rests on real texts selected independently from the distribution of the target use. Synthetic examples can be used in additional stress tests; they do not replace the main estimate of performance.

Wang et al. 2023 · Shumailov et al. 2024 · Gerstgrasser et al. 2024

Three kinds of problem worth knowing by name.

Lack of diversity: examine examples by hand; measures such as the ratio of distinct n-grams and embedding-space coverage (with the tools of chapter 3) are not sufficient by themselves, but they make repetition visible. This is a separate problem from the recursive model collapse.

Distributional mismatch: much of the signal lives in exactly the messiness a generator smooths away. A model trained on clean synthetic text and deployed on real text is being asked to generalise across a shift you introduced yourself.

Recursive contamination: Shumailov et al. (2024) call it model collapse. The rare cases go first, which means exactly the cases you needed the model for. When real data is accumulated alongside the synthetic instead of swapped out — which is what actually happens in practice — the test error stays bounded (Gerstgrasser et al. 2024). The cautious working rule is narrower than “synthetic data is dangerous”: watch the risk of new synthetic generations taking the place of the real data. Using them together with real data, and keeping a fixed, independent evaluation set outside the loop, makes that risk easier to measure. Which ratio and which filtering help is something to try on your own task.

Write it down: datasheets and model cards

The audits produced observations that need documenting. Unless you write them down, with how they were sampled and what their limits are, they disappear
Datasheets for datasets: a standard document covering motivation, composition, collection, preprocessing, uses, distribution and maintenance
Model cards: the same idea for trained models, with performance disaggregated across the groups that matter
Both are conventions, not standards, and both are frequently treated as compliance paperwork
Tie the document to a stage that the decision-makers review before release

Gebru et al. 2021 · Mitchell et al. 2019 · Sambasivan et al. 2021

The observations of this chapter that need documenting: 20.03% of the test reviews contain a contrastive connective; 20 of the 40 test reviews the model disputed with its highest scores are schema problems according to a single reader; in training, the flag rate of negative-labelled rows is thirteen times that of positive ones.

Do not leave the datasheet as a form that merely gets filled in. Tie the provenance of the data, the conditions of use and the known limits to a stage that the decision-makers review before release.

Sambasivan et al. (2021) found, across 53 practitioner interviews, that data problems were pervasive, expensive, and consistently under-valued relative to modelling — “everyone wants to do the model work, not the data work”. Actually using the document can reduce that invisibility.

The next two slides show the chapter’s own extract for this corpus.

A datasheet extract: composition, licence, labels

### datasheet extract — Turkish product reviews

COMPOSITION
  235,165 reviews; 233,508 after exact de-duplication (1,657 extra rows,
  0.70%). 93.7% positive. 77 texts carry both labels. Product category,
  date, and reviewer are not available.

LICENCE AND PROVENANCE
  The licence file of the original GitHub repository and the licensing
  section of the card say CC BY-SA 4.0; the licence field at the top of
  the Hugging Face page says "unknown" (checked on 10 October 2026).
  The site and the date of collection are not documented.

LABELS
  Provenance unknown; how the labels were derived is not documented
  (possibly from star ratings). No guideline document exists. Whether
  inter-annotator agreement was measured is not reported.

A datasheet extract: known limitations

KNOWN LIMITATIONS
  - 20.03% of the test reviews contain one of the connectives ama/fakat/
    ancak/lakin; that alone does not show mixed sentiment or a problem
    with the binary schema. Model macro-F1 on this subset is 72.11,
    against 77.72 on the rest.
  - The 40 test reviews the model disputed with its highest scores were
    examined by one reader: 11 clear label contradictions, 20 schema
    problems. These proportions cannot be generalised to the corpus.
  - In 100 randomly selected test rows the same reader found 0 clear
    contradictions and 7 undecidable rows. The "rule of three" upper
    bound of about 3% for zero observations holds under independent
    sampling and the same definition of an error.
  - Selection bias: reviewers are self-selected. Non-reviewers absent.
    (Not measured.)

Not recommended without independent, task-specific validation: deployments where errors on the negative class are costly. A good datasheet also records, explicitly, the limitations it cannot put a number on.

Where the effort goes: an example order

activity	the information it can give	when to do it
read 100 random rows by hand	candidates for clear contradictions and ambiguity; a wide uncertainty interval for the rate	when the data arrives, before modelling
write a guideline with examples	how clear the decision rules are	before bulk annotation
measure two humans’ agreement on a shared sample	a disagreement table and a kappa that depends on the sampling	in the guideline pilot
out-of-fold flagging + hand reading	candidate rows to examine	once training data exists
leakage audit	risks in the split and the preprocessing	before reporting a result

Strengths — reading random rows, writing a guideline and running an agreement pilot can bring out, early, problems that would be invisible later
Weaknesses — out-of-fold flagging produces a list of candidates and feels like automation: put the labour of review and adjudication into the plan

What to remember

Read your data. A hundred rows, by hand, before modelling
Audit the schema too. Of the 40 model-selected test disagreements examined by a single reader, 20 were counted as schema problems and 11 as clear label contradictions: a reason to review the guideline, not an estimate of rates in the corpus
Report the source class together with the number of rows affected. Flipping 40% of the negatives to positive kept macro-F1 close to the baseline; flipping 1% of the positives to negative lowered it by 2.04 points
Test quality and volume together on your own budget. 500 clean labels came out ahead of 40,000 labels with 20% artificial noise; 1,000 clean labels and 40,000 labels at 10% noise could not be told apart
Audit for leakage before you report a result, especially when the number is unexpectedly good

The quality of the evaluation labels has to be reported along with the model’s score.

Chapter 4 compared representations; this chapter asked what that comparison is measured against. With that chapter’s highest-scoring model held fixed and 2% of the test labels flipped artificially, macro-F1 fell from 76.21 to 73.13. That number is below the 73.51 another model scored on the original labels; but the two numbers belong to different versions of the labels. So this experiment does not show a ranking reversing on a shared test set — when the two models were scored against the same noisy labels, the order changed in none of 200 draws. What it shows is the risk of comparing across studies without reporting which version of the labels was used.

On the first line: most of the observations in this chapter came from reading rows.

On the third line: the two percentages correspond to different numbers of rows; the direction also mattered when the same number of labels was flipped, but the result does not carry over directly to other tasks.

On the fourth line: the choice also takes the real cost of labels, the target distribution and the quality of an independent test set.

These constraints do not relax as the models get larger. The later chapters introduce systems with orders of magnitude more capacity, and every one of them is trained against labels somebody manufactured and evaluated against a test set somebody could have contaminated. The models change; the problem of the quality of the evaluation labels does not — it only gets harder to see, which is why the chapters on LLM evaluation and on evaluation in production have to return to it.

Intuition check 1 of 5

Two independent sources of labels agree on 93% of items. Is that enough?

Unanswerable without the marginals of the two sources
The 93.42% raw agreement and the kappa of about 0.27 here were computed between the reference labels and the lexicon rule; they are not a human–human agreement
The agreement expected under independence, \(p_e\), was 90.99%
On the balanced subsample raw agreement fell to 60.28%; the texts sampled there were different too
Raw agreement is largely a property of the sample: report kappa, the marginals and the confusion table together

Intuition check 2 of 5

How many macro-F1 points did mislabelling 10% of the source class cost?

Flipping 10% of the negatives to positive took macro-F1 from 76.21 to 76.42: +0.21 — not “zero”, but inside the seed deviation
Flipping 10% of the positives to negative took it to 63.63: −12.58
The class-conditional rate is the same, the number of rows flipped is not: about 253 against 3,747, fifteen times as many
With the same number of labels flipped the direction still matters (76.95 against 71.11 at 1,000 labels), but part of the gulf comes from the number of rows
On your own task, measure direction, quantity and class composition separately

Intuition check 3 of 5

Model A beats model B by 1.2 macro-F1 on your test set. Ship A?

Not on the strength of 1.2 macro-F1 points alone
On this corpus 1% artificial test-label noise shifted one model’s score by 1.53 points; that shift in level does not by itself erase the gap between two models on a common version of the labels
How sensitive the gap is to label errors depends on the rows where the two models disagree: chapter 4’s 1.2-point gap is erased by 18 label changes placed in the worst possible way
Read the rows where A and B disagree, blind if you can, fix their label errors, and test the gap with a paired bootstrap

Intuition check 4 of 5

An automated audit flags 92 of your 40,000 training labels. Delete them?

No; read them first
In the condition that dropped these 92 rows, macro-F1 fell from 76.21 to 75.72 — below even the lowest of 20 draws of dropping 92 random rows (75.91); flipping the labels did not beat the baseline either
In a separate audit of the test set, half of the 40 selected candidates were schema problems according to a single reader; the distribution of the 92 training candidates is unknown
Use the flags as an order of review, not as an automatic decision to delete

Intuition check 5 of 5

You have budget for 5,000 careful labels or 40,000 cheap ones. Which?

In this grid 5,000 clean labels scored 71.69; the 40,000 scored 76.21 at 0% noise, 68.18 at 5% and 63.20 at 10%
Linear interpolation between the two measured cells puts the break-even rate at about 2.8%: a value read off a line, not a measured threshold
A real decision also takes the cost of the labels, the mechanism and direction of the noise, the uncertainty, and the quality of the test set
Whichever you choose, set aside a separate quality budget for the test labels

Interview-style questions for discussion

Three of the chapter’s seven questions. When you answer, give the direction and which split rather than a single rate.

Does the annotation budget go to training or to test?
How do you find the candidates for review among 40,000 rows?
The human annotators disagree. What do you do?

The chapter’s questions test whether you can read its numbers within their scope and set apart the claims that have not been measured yet. Some of the answers were observed in this chapter’s experiments; others need verification on other data. The other four questions repeat the intuition checks and the leakage checklist.

What a strong answer names.

Question 1: give each a share according to the costs of the task, and audit the label quality of the final test set independently. Then the 2% result with its condition: a comparison across two versions of the labels; when the two models were scored against the same noisy labels, the order did not change.

Question 2: out-of-fold prediction, with every transformation (the vectoriser included) fitted on the training part of each fold only, and the score the model gives to the opposite class used as an order of review. Then what the measurement taught: the 0.90 threshold flagged 92 candidates in training (123 with the vectoriser fitted inside the fold), and 92 is not a count of real errors; the thirteen-times-higher flag rate of negative-labelled rows is not a difference in error rates; dropping the flagged rows lowered the score more than dropping random rows did (75.72 against a mean of 76.15).

Question 3: first read the shared examples and separate the reasons for disagreement: a gap in the guideline, the guideline being applied differently, ambiguity in the text itself, or a limit of the target definition. This chapter did not measure the disagreement of two humans; on the 40 selected test reviews a single reader recorded 20 schema problems — mixed reviews, truncated texts, reviews about delivery instead of the product. Adding a class such as mixed or not applicable can be an option; but it requires a new target definition, a new guideline, a new class distribution and a re-evaluation. When agreement is low the question is whether the annotators are inconsistent or the label set is under-specified; only reading can separate the two.

Suggested format: pairs, one question per pair, two minutes.

Lab and exercises

Lab: a small annotation project, end to end

Work in pairs: two balanced rounds of 100 items, 50 gold-negative and 50 gold-positive each
Guideline v0 (class definitions only), then round one, independently, with SKIP allowed
Raw agreement, the confusion table, and Cohen’s kappa with a bootstrap interval
Read every disagreement together, write guideline v1, annotate round two, compare to the gold labels

Exercises: nine in the chapter

The six questions for a public dataset; kappa by hand at two sets of marginals
The random-100 audit and the noise-direction experiment on data of your own
Out-of-fold flags with the preprocessing fitted inside each fold; a leakage audit of a result you believe
The break-even rate from the budget grid, marked as an interpolation

Do not compare your kappa to this chapter’s 0.2698: that one is gold-versus-a-lexicon, not human-versus-human. Full text: chapter 5 in the book.

The lab’s deliverable: guideline v1, the two confusion tables, both kappa values with their intervals, the list of disagreement groups, and a one-paragraph statement of what your schema cannot represent.

Details worth reading out. Why balanced rounds: a plain random 200 would contain about thirteen negatives. Shuffling a balanced 200 and cutting it in half is not enough, because the halves need not come out 50/50 and the two rounds’ kappas would then differ by composition as well. This purposive balance does not represent the natural class ratio of the corpus; the kappa you get belongs to the ratio you chose. Say how you counted the SKIP decisions: as a category of their own, or left out.

Report the change between the rounds, but do not credit it to the guideline alone: the texts and your own practice changed along with the guideline. Separating the guideline’s contribution takes a comparative design, for example a second pair labelling the same items under v0.

When comparing to the gold labels: two people agreeing does not by itself make the gold label wrong; do not force a single “right” label without a third independent judgement or an adjudication step.

The exercises are in the chapter, and the code for the experiments is in the chapter notebook.