5  Data, Annotation, and Ground Truth

Chapter status: drafted. The experiments were run on the Turkish product-review corpus of chapter 4 and on its 40,000/10,000 train/test split (Chapter 4); in some experiments the size or the labels of the training subset were changed. The computations used scikit-learn 1.9.0, NumPy 2.5.2 and pandas 3.0.5 on Python 3.12.13. Two results are hand audits: one person, with the criteria written down before the reading began, read 100 randomly selected test reviews and a separate 40 test reviews that the model disputed most strongly. Those counts rest on that reader’s judgement and were not checked by a second human annotator.

Learning objectives. After this chapter you can:

5.1 The number you have been trusting

Round 2 of Chapter 4 ranked seven representations in one table. Tf-idf over word 1–2 grams, fed to a balanced logistic regression, scored 92.57% accuracy and 76.21 macro-F1. Bag of words scored 73.51. Averaged word2vec scored 61.39. The chapter’s conclusion was that, on that data and with those settings, the sparse representations were well ahead of the from-scratch averaged embeddings, by far more than the sampling uncertainty of the test set. The order of neighbouring rows, though, was sensitive to the tuning budget: raw counts and TF-IDF swapped places once C was chosen on validation. So trying a strong sparse baseline first is a good start; a general claim that one method is better takes new data and a new uncertainty analysis.

gold standard — the labels treated as correct when a model is scored. They may come from adjudication, from an expert’s decision or from another documented process. Being the reference does not make them free of error; their provenance and the disagreements inside them have to be examined separately. ↩︎

Every one of those numbers was computed by comparing the model’s output against a column called sentiment. Throughout this chapter we call the labels in that column the reference labels: they are the point of reference for evaluation, not a claim to be the one indisputable reading of the text. The terms gold standard and ground truth name the same function. Who produced the labels, in answer to which question and how, is not documented; in the previous chapter we did not ask once.

This chapter asks. In some of the experiments we hold chapter 4’s highest-scoring representation, the classifier and the test reviews fixed and change only the training labels or the test labels. In others the amount of labelled training data, the selection of examples or the split itself changes too; each table says which components stay fixed. That way a change in score is interpreted only inside the controlled comparison of its own table.

What comes out is uncomfortable. A 2% error rate in the test labels drags the leaderboard’s top entry below the score its bag-of-words rival earned on the original labels — not because the model changed, but because the labels the score is computed against did. Note the shape of that comparison, because it is the honest one: two conditions with differently labelled test sets, not a rank reversal on a shared one. Scoring both models on the same noisy labels is a different question, and we measured that too: the 2.69-point gap between them shrinks to 2.50 on average at 2% noise, and the order does not change in any of 200 draws. Of the 40 selected test reviews the model disputed most strongly, 20 were classified as texts that a single binary label struggles to describe; that proportion cannot be carried over to the dataset as a whole. And 1,000 clean labels tie with 40,000 labels at 10% noise; at 20% noise, 500 clean labels beat all 40,000.

The corpus is the same one as in chapter 4 (Chapter 4): 235,165 Turkish product reviews, about 93.7% of them labelled positive. That imbalance strongly shapes many of the results; the schema, the model and the sampling decisions are part of the results too.

What we will look at. First the steps through which labels are produced, and a hand audit of a hundred random reviews. Then annotator agreement, the limits of the schema, label noise and leaks between the splits, each with its own experiment. The last sections cover the annotation budget and the alternatives to labelling by hand.

5.2 Where a label actually comes from

Labels do not arise by themselves. First the categories and the decision rules are set; then texts are selected, annotators decide, and disagreements are merged. At each step the meaning of the label can drift from what you assume; and if the steps were not recorded, you cannot tell which decision produced the final label.

The steps a label goes through, and where it can break
the pipeline raw text scraped or logged schema what the classes are guideline how to decide annotator reads, presses a key stored label one column what can break here selection who writes reviews is not who buys 93.7% positive schema gap texts the classes cannot express 20 of 40 read ambiguity the guideline is silent here human–human κ not measured here clear label error possibly fatigue, misreading, a slip 11 of 40 read collapse disagreement is not recorded 1 column left

The stored label gathers into one column every decision made from the selection of the text to the merging of disagreements; your training run, your leaderboard and your production monitoring see only that column. The 20/40 and 11/40 in the figure belong only to the single-reader audit of the 40 test reviews the model disputed most strongly; how often the problems at the other stages occur was not measured in this chapter.

The last box is the one to watch. When a dataset is distributed with only a text column and a label column, the history of the annotators’ decisions becomes invisible: disagreements, hard cases and skipped examples are reduced to a single integer. Keeping the disagreements, the hard cases, the skipped examples and the adjudication decisions as well — in extra columns, in a dump of the individual judgements, or in a datasheet — gives a dataset that can be audited far better than a single integer can.

The questions to ask before you use a dataset

Before any modelling, try to answer the six questions below. Record what the source does not say as “undocumented”, and assess separately what that uncertainty means for your decision to use the data.

  1. Who produced the text? Not “the internet”. Which platform, which population, which time window. The review corpus is people who chose to write a review after a purchase, which is not the same population as people who bought.
  2. Who produced the labels, and how? It may be domain experts, crowdworkers, a model, or the platform’s star rating. Deriving a sentiment class from a star rating is a proxy labelling decision: it carries the rating scale’s own biases, and how well the stars match the sentiment question you are targeting has to be tested. Crowdsourced labels are not automatically worse than expert ones: Snow et al. (2008) showed that on some tasks the combined judgements of a handful of non-experts can match expert quality. That result is not guaranteed for every task or every way of combining; it depends on the task being one a non-expert can answer at all.
  3. What question were they answering? “Is this review positive?” and “Would you buy this product?” and “Is this customer satisfied?” produce different labels on the same text.
  4. What licence, and does it cover your use? Research-only is a real restriction. So is a licence that permits redistribution of the text but not of derived models.
  5. What legal basis and what notice existed for the use? That anyone can read a review does not show that explicit consent to model training was given. Examine separately the platform’s terms, the purpose the data was processed for, and the legal basis that could apply.
  6. What is missing? Which texts were deleted, filtered, or never written, and which groups does that affect? This is the hardest one and the one most worth attempting. Dodge et al. (2021) did it for C4, one of the most-used web corpora in the field, and showed that the blocklist used to “clean” it had disproportionately left out text about and by minority groups. Selection of this kind cannot be seen from the distributed data alone; it takes a record of the collection and filtering decisions.

Before any of that, a question that can stop a project after it ships rather than before it starts: are you allowed to use this data at all?

Public does not mean free to use. “I could download it” and “I am permitted to train on it” are different claims. Being able to access content does not mean permission to redistribute it or to train a model on it has been documented. Scraped-from-the-web corpora frequently carry no licence at all. Examine the source’s licence, the platform’s terms and the rights in the content separately, for your own intended use.

Assess separately what the data licence means for the model. Which obligations a licence such as CC BY-NC-SA creates for training, for distributing a model or for the outputs it generates is not settled by an automatic “the licence travels” rule; it can vary with the jurisdiction and with how much of the data the model memorises. This is unsettled law. Examine the licence text, your form of use and the relevant jurisdiction for your concrete case, and be conservative in anything you ship.

Document the provenance. Record, at minimum: the source URL, the access and download dates, the licence text, the data version or a file hash, the exact filtering you applied, and your splitting decisions. Longpre et al. (2024) audited 1,800+ popular text datasets and found licence information missing or wrong for a large fraction of them — including datasets used in production. What you did not record at download time becomes hard to reconstruct later.

Assess personal data and data governance separately. If a review makes its writer identifiable, directly or indirectly, a personal-data regime applies alongside copyright. Consent is not the only legal basis: KVKK Article 5(2)(d) allows processing data the person has made public themselves. But that basis is read narrowly and is limited to the purpose for which the person made the data public; a review posted to inform other shoppers does not automatically become model-training material. Erasure requests — KVKK Articles 7 and 11 in Turkey, GDPR Article 17 in the EU — are assessed within the conditions and exceptions of those articles; “I already trained on it” is not, by itself, an answer. Meeting a request takes the row-id-to-source mapping; because that mapping can itself carry personal data, keep it for as long as needed and with security measures. And if the system you build is high-risk under the EU AI Act, check the obligations of its Article 10 on the governance of training, validation and test data as well: documented provenance, examination for bias, and a stated data-governance process. For the obligations and their dates see Section 38.1.

5.3 Read a hundred rows. Yourself.

Early in a data project, reading a randomly selected sample by hand is a useful audit that needs no code beyond drawing the sample. Set time aside for it: draw 100 rows from your test set, read them with the label visible, and record the ambiguities and the outright label contradictions you see separately.

I drew 100 rows from the 10,000-row test split with seed 42 (np.random.default_rng(42).choice(10000, 100, replace=False)) and read all of them. The criteria were fixed before reading:

  • error — the gold label contradicts the text;
  • undecidable — a single binary label is not determined by the text;
  • ok — everything else.

I split ok in two during the reading, which is a deviation from the pre-registered scheme and is recorded here for exactly that reason: arguable for rows where the gold label is defensible but I would have chosen the other one, and plain ok for the rest. The arguable count is therefore a post-hoc category and carries less weight than the other three.

The result:

### random 100 test rows, read by hand by one reader

errors (gold contradicts the text)          0
undecidable (binary label not determined)   7
   ... of which "not yet received"          3
arguable (post-hoc; defensible either way)  9
ok (the remaining rows)                    84
                                          ---
                                          100
positive gold labels in the sample         98

Zero outright errors in 100 draws. With zero hits, the rule of three gives an approximate one-sided 95% upper confidence bound of 3/n: the data are consistent with an error rate up to about 3%, and rule nothing below that out. It is not a measurement that the rate is under 3%, and 100 draws cannot distinguish 0.1% from 2%. If a uniform prior (Beta(1,1)) is chosen, the posterior mean for zero outright errors in 100 rows is \(1/(n+2) = 1/102 \approx 0.98\%\). That value does not mean the error rate of the corpus has been measured as 1%: it depends on the choice of prior and on one reader’s definition of an “outright error”. In later calculations we use about 1% only as an example scenario. The seven undecidable rows and the nine arguable ones are not part of this count of outright errors.

The seven undecidable rows are the interesting ones, because they are not errors. Test row 4745 reads, in full:

sipariş verdim umarım tez ulaşır :) — “I placed the order, hope it arrives fast :)”

Gold label: positive. The reviewer has not received the product; there is no product verdict in the text, only a smiley and an expression of hope. Under the schema proposed in this chapter, which targets the verdict on the product, this review is undecidable. Because it is not documented which question the original sentiment label answered, we do not call the source label of this row an error. Two of the other six are the same shape (“I just ordered it, judging by the reviews…”, “it will probably arrive tomorrow”). One arrived used and scratched, but the reviewer says the product itself is good — a verdict on the delivery pointing the opposite way from the verdict on the product. Of the remaining three, one is a proverb about doors, one offers only a suggestion (“it would be better in liquid form”), and one is the string seat leon st 1.6tdi and nothing else.

When the target is a binary verdict on the product, these seven rows fit a single label poorly. The problem may lie in the definition of the target question rather than in a wrong decision by the source annotator; confirming that would take the original schema and a second reader’s decisions. Hold onto this distinction between a label error and a schema problem; the rest of the chapter keeps returning to it.

Ninety-eight of the hundred were positive, against a corpus rate of 93.7%. That is borderline but still explainable by sampling noise for n = 100 (at a 6.3% negative rate, the chance of at most 2 negatives in 100 rows is about 4.5%), but it is a reminder that a 100-row sample of a skewed corpus tells you almost nothing about the minority class. To audit the minority class, sample it separately.

5.4 The schema can do as much damage as the annotators

To look into whether the limits of the schema show up on a larger body of data, we can set up a screening rule. The contrastive-connective test below does not diagnose a schema problem; it selects reviews to re-read.

To search 10,000 reviews for candidate contrasts, mark the words ama, fakat, ancak and lakin — all roughly “but”. Whether a matching review really contains two opposed evaluations cannot be known without checking by hand.

A caveat the rest of this section rests on: this is a screen, not a parser, and its precision is unmeasured. Ancak also means “only” or “barely” — ancak 3 günde geldi is a statement about delivery time, not a contrast — and the rule misses gerçi, ne var ki and buna rağmen entirely. So the 20.03% found below is neither an upper nor a lower bound; it is only the match rate of the rule. Reading randomly selected matches would measure the rule’s precision; for the contrasts it misses, the non-matching reviews have to be examined too. We checked the part that can be checked mechanically: the ambiguous ancak occurs in 351 reviews, 287 of which match through ancak alone, and with ancak taken out of the rule the gap between the two groups of the table below stays in place (macro-F1 71.29 against 77.80). Attributing the score gap between the two groups to the schema takes a more detailed annotation.

import re
# tr_lower is the Turkish-aware lowercaser from chapter 4 (İ->i, I->ı)
CONTRAST = re.compile(r"\b(ama|fakat|ancak|lakin)\b")
has_but = np.array([bool(CONTRAST.search(tr_lower(t))) for t in test.text])

In 2,003 of the 10,000 test reviews (20.03%) at least one of the four connectives matched. The class distribution and the model scores of this group are compared with the non-matching group below:

The same model, the same test set, split by the four-connective screening rule
contrast rule matches no match
share of test set 20.03% 79.97%
gold negative rate 11.48% 5.03%
model accuracy 84.07 94.70
model macro-F1 72.11 77.72
model F1 (negative) 53.84 58.27

This fifth of the test set has more than twice the negative rate (11.48% against 5.03%), and the model scores clearly lower on it: 5.6 points lower macro-F1 and 4.4 points lower negative-class F1 (the bootstrap interval of the macro-F1 gap over the test rows runs from 2.5 to 8.8 points). The ten-point accuracy gap overstates this: the two groups have different base rates, so part of any accuracy difference comes from the class mix alone. This measurement does not separate the possible explanations. The label schema may be being asked to compress two verdicts into one bit; the model may be making more mistakes on these sentences; and the two groups differ in other respects too — the matching reviews are clearly longer (median 30 tokens against 16). Because no aspect-level re-annotation was done (product, delivery, price and so on labelled separately), we cannot choose between the schema explanation and the model explanation. A sample in which the product, delivery and price verdicts are labelled separately is what it takes to test which error comes from which source: if the model recovers both aspects correctly and only the single bit is wrong, the problem is in the schema.

Which fix applies follows from which explanation holds, so the aspect-level annotation above is what decides it. If the disagreement comes from the target definition, the fix is a richer schema (aspect-level labels: product / delivery / price / packaging, each with its own polarity) or an explicit rule in the guideline telling every annotator which clause wins. If it comes from the model reading these sentences badly, a better representation is exactly what helps — and nothing here rules that out.

A related problem has a name: annotation artifacts — the labelling process creating a cue between text and label that lies outside the target task. When a guideline pushes annotators toward a habit, models can learn the habit instead of the task, and the score is usually pushed up. Gururangan et al. (2018) showed that a well-known inference benchmark could be partly solved by reading the hypothesis sentence alone and ignoring the premise entirely: the annotators who wrote those sentences had tells, and the model found them. The connective match here is something else: it defines a candidate group for schema problems and model errors; by itself it is not evidence of an artifact or of a schema flaw in this data. What the two situations share is that a label can reflect the labelling process as well as the text.

Geva et al. (2019) showed, in a data setup where the annotators wrote the examples themselves, that models can identify which annotator wrote an item, and that scores drop when the test set comes from annotators not seen in training. A setup in which ready-made product reviews are only handed out for labelling is different. This corpus has no author or annotator identifiers, so the size of any such effect cannot be measured here.

What a usable guideline looks like

annotation guideline — the document that sets out in writing the task, the classes, the decided examples and how ambiguous cases are to be handled. It makes the decisions of independent annotators more consistent and easier to audit; it does not guarantee full agreement. ↩︎

In an annotation guideline the class definitions are only the start. For independent annotators to decide consistently on borderline cases it also needs decided examples, priority rules and a SKIP option. Keep the guideline together with the version number of those decisions.

### annotation guideline v1 — product review sentiment

TASK
  Label each review positive (1) or negative (0) according to the
  reviewer's verdict ON THE PRODUCT.

DECIDED CASES  (these make the definition above concrete)
  1. Delivery, packaging, seller, and returns are NOT the product.
     "Fast shipping, product is useless"          -> 0
     "Shipping was late, the product is great"    -> 1
  2. Price complaints are NOT negative unless the reviewer says the
     product is not worth it.
     "Expensive but excellent"                    -> 1
     "Not worth this money"                       -> 0
  3. Mixed verdicts on the product: the FINAL clause wins.
     "Cuts well but the handle broke"             -> 0
     "The handle is cheap but it cuts very well"  -> 1
  4. Not yet received / no verdict given          -> SKIP (do not label)
  5. Truncated text: if the visible text contains a verdict, label it;
     otherwise SKIP.
    6. Judge negation in context. "fena değil" (not bad) is positive
     for the product. "fena değil ama çabuk bozuldu" (not bad but it
     broke quickly) is 0 by rule 3, because the final verdict on the
     product is negative.

WHEN TO SKIP
    If no decision about the product can be made, choose SKIP. Keep the
  skipped examples and the reason; they help improve the guideline and
  the schema. A forced binary label can measure the wrong target even
  when it looks consistent.

Two properties make this usable. First, every decided case has a worked example on both sides, so an annotator matching by pattern gets it right without understanding the principle. Second, SKIP exists. Forcing a binary choice on undecidable items can manufacture disagreement you will later blame on the annotators; and if the annotators guess by the same habit, it can also manufacture an agreement that looks high and means little.

The choice in rule 3 — “the final clause wins” — is arbitrary. It can give consistency; whether it really represents satisfaction with the product has to be tested separately. Settle the target question first, then write a decidable and consistent rule for the hard cases: an annotator must always be able to tell which rule applies, and to apply it the same way every time. An arbitrary rule applied consistently produces learnable data, and a principled rule applied inconsistently does not. But good agreement does not make a shared answer to the wrong target correct.

5.5 Agreement: what does 93% tell you by itself?

The hundred-row audit showed one reader’s decisions. Having an independent second person label the same reviews shows how hard it is to agree on the borderline cases. The agreement figure, though, has to be read together with the class distribution and the table of disagreements; this section explains why.

inter-annotator agreement (IAA) — the extent to which the decisions of people labelling the same items independently coincide. It can be summarised by the raw rate, by class-specific agreement or by chance-corrected coefficients. High agreement does not by itself prove that the labels represent the target concept correctly. ↩︎

You cannot know your labels are good by looking at one annotator’s output. You need a second opinion on the same items, and a number for how often the two opinions match — the inter-annotator agreement.

I had no second human annotator, so to have something to compare the reference column with I used a lexicon rule: 36 positive and 37 negative phrases, applied to all 10,000 test rows. The phrase lists are not printed on this page; so the 2×2 table below cannot be reproduced from the code shown, while every coefficient derived from the table can.

One caveat before the numbers, because it changes what the coefficient means. B is not an independent second opinion in the sense inter-annotator agreement requires: the rule was written by someone who had already seen the reference labels. In this table \(\kappa(\text{A}, \text{B})\) summarises the overlap between the reference labels and the lexicon rule; it is not a measure of human–human agreement or of whether the schema is well defined. The same formula and the same arithmetic — prevalence, \(p_e\), skew — apply to a table of two humans’ judgements; but the sampling and independence conditions of that experiment have to be set up separately. The number 0.2698 belongs only to this reference-versus-rule table and should not be quoted as a human–human agreement figure.

def annotator_b(text):
    t = tr_lower(text)
    neg = any(p in t for p in NEG)
    pos = any(p in t for p in POS)
    if neg and not pos:
        return 0
    return 1          # no match, or matches of both kinds: positive

The rule looks for the phrases as substrings; it does not check word boundaries, and it says positive when a review contains both a positive and a negative phrase. Switching to a rule with word boundaries would mean recomputing the table below.

A is the reference sentiment column of the dataset; B is the output of the lexicon rule above. For brevity we call both “annotators”, but neither is a human reading the text independently. The table comparing these two label sequences on the 10,000 test reviews:

### A (gold, rows) vs B (lexicon rule, columns), n = 10,000

                B: negative   B: positive
A: negative           141           491
A: positive           167          9201

Raw agreement is \((141 + 9201)/10000 = \mathbf{93.42\%}\). But 9,201 of that total are shared positive decisions: the majority class pulls the overall agreement up. To see the disagreement on the negative labels you have to look at the first row and the first column of the table separately.

Cohen’s kappa — a coefficient that compares the observed agreement of two label sequences with the agreement expected if each decided independently at its own label rates. A low kappa is not, by itself, evidence of label errors or of carelessness. ↩︎

The 93.42% raw agreement is not strong evidence by itself. If the decisions are assumed to be made independently of each other under the marginals of these two label sequences, the expected agreement is about 90.99%. That is a mathematical reference value; it does not mean the rule, or real annotators, did not read the texts. Nor does a low kappa against a lexicon show that the reference labels are wrong. Cohen’s kappa (Cohen 1960) compares these two rates:

\[\kappa = \frac{p_o - p_e}{1 - p_e}\]

where \(p_o\) is the observed agreement and \(p_e\) is the agreement expected if both annotators labelled independently at their own base rates. Here \(p_e = 0.9099\), and:

\[\kappa = \frac{0.9342 - 0.9099}{1 - 0.9099} = \mathbf{0.2698}\]

Krippendorff’s alpha (Krippendorff 2018), which handles more than two annotators and missing values, gives 0.2655 on the same table. Its closeness to kappa is not an independent confirmation: the two coefficients are computed from the same label pairs. Because they define expected agreement differently, they will come clearly apart on the balanced subsample below.

A kappa of 0.27 is, on the conventional (and arbitrary) scales, “fair”. The useful reading is a comparison rather than a decomposition: 93.42 points of observed agreement against 90.99 that two annotators with these marginals would reach while labelling independently, leaving 2.43 points above that baseline. Resist turning that into “90.99 points were chance”: \(p_e\) is the agreement expected under an independence null, not a measurable portion of what actually happened. A low kappa on a corpus this skewed is consistent with careless labelling and with careful labelling alike; the coefficient does not tell you which one you have.

That correction is doing all the work, so it is worth seeing exactly what it computes rather than taking the number on trust.

\(p_e\) is the share of agreement expected when the observed class rates of the two label sequences are kept and the decisions are assumed to be independent of each other. The calculation rests on how often each sequence uses the negative and the positive label.

Annotator A calls 632/10000 = 6.32% of items negative. Annotator B calls 308/10000 = 3.08% negative. So:

\[ \begin{aligned} p_e &= P(\text{both say negative}) + P(\text{both say positive}) \\ &= (0.0632)(0.0308) + (0.9368)(0.9692) \\ &= 0.00195 + 0.90795 \\ &= 0.90989 \end{aligned} \]

Almost all of \(p_e\) — 0.908 of 0.910 — comes from the second term. Do the arithmetic with the real marginals rather than a round number, because the round number misleads: A calls 93.68% positive and B 96.92%, and two labellers with those rates, choosing independently, land on the same answer \(0.9368 \times 0.9692 + 0.0632 \times 0.0308 = 0.910\) of the time whatever the texts say. (Two labellers at 90% would reach \(0.9^2 + 0.1^2 = 0.82\), not 0.90 — the expected agreement is not the marginal.) Kappa’s denominator, \(1 - p_e = 0.09\), is the room left for genuine agreement, and the numerator, 0.0243, is how much of that room the annotators actually filled.

This is why kappa comes out low on skewed tasks and why reporting raw agreement alone on a skewed task is misleading. It is also why kappa is volatile there: in this table \(1 - p_e\) is about 0.09, so if the marginals are assumed fixed, a small change in the observed agreement is multiplied by about eleven on its way into kappa. When a cell really changes, the marginals and \(p_e\) move as well. Report raw agreement, class-specific agreement, kappa and the confusion table together, and let the reader see the skew.

Cohen’s kappa is defined for exactly two annotators. When you have three or more — the normal case once a crowd platform is involved — use Fleiss’ kappa (Fleiss 1971), which is the generalisation of Scott’s pi rather than of Cohen’s: it computes \(p_e\) from a single marginal pooled across all annotators, not from each annotator’s own rates, and it does not require the same annotators to have labelled every item (it does assume every item received the same number of judgements). Krippendorff’s alpha (Krippendorff 2018) takes the idea further, writing agreement as \(\alpha = 1 - D_o/D_e\) — observed disagreement over expected disagreement — with a distance function you choose, so one coefficient covers nominal, ordinal and interval labels and tolerates missing judgements. The interpretation, and the skew problem, are unchanged for all three.

That difference between “each annotator’s own rates” and “one pooled rate” is not bookkeeping; it is the second of the two known situations that affect how kappa is read, and this corpus illustrates it. See below.

The same reference and lexicon rule, a different sample

To see the effect of the class ratio, take all 632 reviews A calls negative and a random 632 of the reviews A calls positive. This 1,264-row artificially balanced subsample does not estimate agreement on the full test distribution; it shows how the same A and B behave on differently selected texts.

### the identical annotators, on a class-balanced subsample

                    full test set (n=10,000)   balanced (n=1,264)
raw agreement                       93.42%              60.28%
p_e                                 90.99%              50.00%
Cohen's kappa                        0.2698              0.2057
Krippendorff's alpha                 0.2655              0.0722

On this subsample raw agreement falls from 93.42% to 60.28%. The decision rules of A and B are the same; the texts evaluated and the class mix are different. The result shows that raw agreement is not a fixed property of the annotators but a measure that changes with the sample.

The two chance-corrected coefficients also part company on the right-hand column, and that gap is the second half of the story. On the full set they agree (0.2698 against 0.2655); on the balanced subsample kappa is nearly three times alpha (0.2057 against 0.0722), from the same data. The mechanism is the one named in the \(p_e\) callout above: Cohen’s kappa computes \(p_e\) from each annotator’s own marginals, while alpha and Scott’s pi use a single pooled marginal. On the balanced subsample A is 50% negative by construction and B is roughly 12% — a large divergence in how often the two of them reach for the minority label. Divergent marginals lower Cohen’s chance baseline (here to 0.50, against 0.57 for the pooled marginal), so the same observed agreement counts as further above chance; alpha’s pooled baseline is not lowered by the divergence, so it does not give that credit.

This table illustrates two known situations that affect how kappa is read (Feinstein and Cicchetti 1990; Cicchetti and Feinstein 1990). On the full set a high raw agreement comes with a low kappa (the prevalence effect); on the balanced subsample the different marginals of A and B widen the gap between kappa and alpha (the bias effect). These are not malfunctions of the coefficient; they are examples of the coefficients using different null models and being sensitive to the composition of the sample. Report kappa and alpha together with the marginals and the sampling design.

One number in that table is exact rather than measured, and the reason is worth a line. On the balanced subsample \(p_e = 0.5000\) exactly, because when one annotator is a perfect 50/50 split the expected agreement is \(0.5b + 0.5(1-b) = 0.5\) whatever the other annotator’s rate \(b\) is. The coefficient’s denominator, \(1 - p_e = 0.5\), stops depending on B entirely.

Choosing the coefficient after looking at the table can shift the interpretation. The five summaries of the same 2×2 table below answer different questions; so settle which coefficient you will use, and why, before you look at the results:

### the identical 10,000-row table, five coefficients
raw agreement  p_o                    0.9342
Gwet's AC1                            0.9277
PABAK  (2*p_o - 1)                    0.8684
Cohen's kappa                         0.2698
Krippendorff's alpha                  0.2655

Not one of these is wrong. They differ because they make different assumptions about what “chance” means when one class dominates: kappa and alpha assume annotators guess at their observed rates and charge you for the skew; PABAK and AC1 assume they do not, and refuse to (Gwet 2008). Which is right depends on which null you think is the fair comparison for your task, not on diagnosing from the table whether anybody guessed. The coefficients differ in what they assume chance agreement would look like; none of them can tell you what the annotators were doing. The five summaries of this table range from 0.27 to 0.93. A good report gives the confusion table, class-specific agreement and a coefficient justified in advance, together.

There is a second lesson hiding in B’s scores. Annotator B achieves 93.42% accuracy against gold. Chapter 4’s majority-class baseline — label everything positive, read nothing — achieves 93.68%. B’s hand-built lexicon, 73 phrases of careful Turkish, performs worse than a constant. On macro-F1 the picture is different (63.27 for B against 48.37 for the constant), which is the whole reason Chapter 4 refused to report accuracy alone.

There is a third number worth reporting. For one class, specific agreement is twice the number of examples on which both chose that class, divided by the total number of times A and B chose it: \(2 n_{kk} / (n_{k\cdot} + n_{\cdot k})\). In this binary table the calculation is numerically identical to B’s class F1 against A: 96.55% on positive and 30.00% on negative. The equality comes from the definition of the metric; it does not imply that B is a human or that A is free of error. Specific agreement does what no single chance-corrected number can: it shows where the divergence is. These two label sequences overlap almost completely on praise and very little on complaint.

What agreement is worth at a given prevalence

The sliders change raw agreement, A’s minority rate and the ratio of B’s rate to A’s hypothetically. The same 93.4% agreement is worth kappa ≈ 0.27 at the rates of this table; it would be worth less at more skewed rates and more at balanced ones. Not every setting of the three sliders describes a 2×2 table that can exist; the curve is drawn only over the range where the three choices form a possible table, which is why it covers a short arc rather than the whole axis. The hollow circle marks the measured point of this chapter’s table; the other points of the curve are not new experimental results.

The verbal labels under the number — slight, fair, moderate, substantial, almost perfect — are Landis and Koch (1977)’s benchmark scale, and they are worth naming precisely because everyone uses them and almost nobody says where they come from. The authors proposed the bands as “clearly arbitrary” divisions for discussion, not as measured thresholds. Report the number and the interval; treat the adjective as a label for a conversation, never as a passing grade.

Adjudication

adjudication — an additional review that examines why judgements disagree and records either the decision or the remaining uncertainty. Some disagreements improve the guideline; others show a lasting difference of interpretation. ↩︎

Agreement measurement is diagnostic; adjudication is the step that follows it. A loop you can follow:

  1. Two annotators label the same 200 items independently of each other. Not a sample of each — the same items. To be able to see minority cases you can stratify the pilot toward the minority class: at this corpus’s 6.3% minority share a plain random 200 contains about thirteen minority items, and most of kappa’s numerator comes from those cells. The kappa of such a pilot belongs to the class ratio you chose. If you want to report agreement on the natural distribution, use a separate, representative sample, or state the sampling design explicitly.

  2. Compute the confusion table and an uncertainty interval for kappa together. The interval is not optional at this size. A kappa’s standard error depends on the marginals as well as the sample size, so instead of a ready-made formula use a bootstrap that, as below, resamples the two judgements of the same item together. Resampling this chapter’s reference-versus-lexicon table gives a standard error of about 0.02 at 10,000 items and about 0.14 for a plain random 200, which puts the 95% interval around a point estimate of 0.27 at roughly \([-0.02, 0.54]\). On a stratified pilot of 100 gold-negative plus 100 gold-positive items the standard error drops to about 0.04, and the point estimate moves to around 0.21 (kappa depends on the sample’s class mix). These numbers belong to the reference-versus-lexicon table; in your own pilot with two humans, compute the interval separately. Fix the target coefficient and the composition of the sample before you look at the results, and end the pilot loop (steps 3–5) when the lower bound of the interval clears the target; an interval that contains the target is not evidence in either direction. Nor does clearing a threshold show by itself that the schema measures the target correctly.

    from sklearn.metrics import cohen_kappa_score
rng = np.random.default_rng(42)
idx = (rng.integers(0, len(a), len(a)) for _ in range(2000))
boot = [cohen_kappa_score(a[i], b[i]) for i in idx]
lo, hi = np.percentile(boot, [2.5, 97.5])

  3. Read every disagreement together. Not to decide who was right, but to find out why they differed. Disagreements often cluster into a few recurring situations.

  4. Add each situation to the guideline as a decided case with examples on both sides. This is the deliverable.

  5. Annotate a fresh 200 independently and re-measure. If agreement did not rise, examine the disagreements again: the cause may be a gap in the guideline, the guideline being applied differently, a change in the sample, or a limit of the schema.

Do not treat step 3 as a vote. Resolving disagreements one by one improves the pilot data. Adding the recurring kinds of disagreement to the guideline can also pay off in the next round of annotation; measure its effect on new, independent items.

What to do with three annotators instead of two

Sending every item to a third person is expensive at scale. So most projects combine a majority decision, a reliability model and expert adjudication of selected disagreements. The diagram at the top of this chapter also named collapse — the record of disagreement not being kept when the judgements are merged into one label — as a problem of its own, without saying what to do about it. With a crowd you have several judgements per item; how you reduce them to one label is a choice, and keeping the raw judgements lets you change that choice later.

Majority vote is a simple starting method. If the annotators’ error rates differ markedly, or their errors depend on one another, weighting the votes equally can lose information. Latent-variable aggregation (Dawid and Skene 1979) estimates each annotator’s error pattern (confusion matrix) and the hidden class with a joint probability model, by EM (expectation–maximisation: guess the true labels, re-estimate each annotator’s error rates from the guesses, re-guess, and repeat until nothing changes); reference labels are not required. The result depends on the annotators’ judgements overlapping enough and on the model’s assumptions. Snorkel-style weak supervision also models the reliability of its sources; there the dependence between the labelling functions and the model family have to be specified separately. A third option is to keep the vote distribution without reducing it to one label at all, and to train against a soft target: an item two of three annotators called positive gets a positive vote share of 2/3 as its target instead of a hard 1. That number is not, by itself, a calibrated estimate of the probability that the item really is positive; whether the disagreement has to do with the task has to be read separately. In this chapter half of the 40 test reviews the model disputed most strongly were classified as texts that fit a single binary label poorly; on such items a 2–1 split can carry information that a hard label deletes (Uma et al. 2021; Plank 2022).

5.6 Auditing labels at scale

Reading all 40,000 rows by hand is expensive for most projects. What you can do is set up a procedure that chooses which rows to read first.

Here we use out-of-fold prediction to produce a review priority. It is a simple threshold rule inspired by confident learning (Northcutt, Jiang, et al. 2021), which uses the model’s own confidence to find candidate label errors; it is not the method itself. The full method does not use a single global threshold: it thresholds each class at the average probability the model assigns to that class over the examples labelled with it, and builds a count matrix (the confident joint) from those thresholds. The single 0.90 threshold below is a shortcut, and per-class thresholds exist to correct exactly this when classes are so imbalanced. Fit on four folds, predict the fifth, repeat. Every training row now has a prediction from a model that never saw that row’s label. Rows where the model gives a high score to the other class are not automatic verdicts of error; they are candidates to be read by hand.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Xtr here comes from a vectoriser fitted on all 40,000 rows, before the folds.
# For a fully out-of-fold prediction, fit the vectoriser inside each training fold too.
oof = np.zeros(len(y))
for train_idx, val_idx in skf.split(Xtr, y):
    oof[val_idx] = clf().fit(Xtr[train_idx], y[train_idx]).predict_proba(Xtr[val_idx])[:, 1]

# the score the model gives to the opposite class; not the probability that the label is wrong
wrong_conf = np.where(y == 1, 1 - oof, oof)

On the 40,000 training rows:

Rows whose opposite-class score exceeds the threshold, to be read by hand
threshold rows flagged share gold positive gold negative
> 0.50 3,026 7.57% 2,232 794
> 0.90 92 0.23% 49 43
> 0.95 34 0.09% 18 16
> 0.99 2 0.005% 2 0

We measured what the comment in the code implies. With the vectoriser also fitted on the training part of each fold only, the number of rows above the 0.90 threshold rises from 92 to 123, and at 0.95 from 34 to 47; 89 of the first 92 candidates are in the new list too. The numbers below, and the repair experiment, belong to the first list of 92. In your own audit, fit the vectoriser inside the fold; a Pipeline does that by itself.

Three things are worth noticing.

The flags are class-balanced, the corpus is not. At the 0.90 threshold, 43 of 92 flags are on gold-negative rows — 47% — in a corpus that is 6.3% negative. Per row, 1.70% of negative labels are flagged against 0.13% of positive ones — the flag rate of a negative label is thirteen times higher. That is where to start reading. But it is a difference in flag rates, not error rates: the model may simply be less certain, or worse calibrated, on the minority class.

The high threshold flags very few rows. 92 of the 40,000 training rows exceed the 0.90 threshold. The random 100 test rows showed zero outright errors; if the prior-dependent scenario of about 1% from there were right you would expect around 400 errors, but that value is not a measured error rate and cannot be used here as a count of real errors. The rule also leaves out the errors the model cannot detect: it is blind to errors on texts the model itself finds hard. Rows that are not flagged cannot be declared clean.

A flag is a request to read, not a verdict. The 92 candidates above are in the training set. In a separate audit I read, as a single person, the 40 reviews of the test set on which the model disputed the stored label with the highest score (in the test set a wrong label costs you a wrong leaderboard rather than a slightly lower-scoring model). The two selections have different sampling frames; the proportions of one cannot be carried over to the other. I classified each review by hand into one of three categories: error (the gold label clearly contradicts the text), schema (a single binary label cannot represent the text), or model (the label is defensible and the classifier is wrong).

### 40 highest-confidence gold/model disagreements, test set, read by hand

error   (gold label is wrong)                        11    27.5%
schema  (binary label cannot represent the text)     20    50.0%
model   (label defensible, classifier wrong)          9    22.5%
                                                     --
data-side problems                                   31 / 40

schema breakdown:  mixed 12 · truncated 4 · lukewarm 3 · no verdict 1

Of the 40 selected candidates, 31 show a problem on the data side according to the single reader: 20 were classified as schema and 11 as a clear label contradiction. These proportions describe only the test examples the model selected with its highest scores; they are not an estimate of a rate for the corpus or for the 92 training candidates. With \(n = 40\) the 95% Wilson interval (a confidence interval for a proportion that stays sensible at small \(n\)) on the schema share runs from about 35% to 65%, and on the error share from 16% to 43%, and the two intervals overlap; but these intervals describe counting uncertainty only and do not correct for purposive selection. The reading was done by a single unblinded reader (me, who had written the hypothesis). The same forty items should be read by a second, blinded reader and the agreement reported; that step tests the subjective classification, and does not by itself solve the problem of generalising. I did not do that, and you should before you quote the number.

Four of the forty show what the categories mean:

  • Row 6874 (gold = negative, model p(positive) = 0.984). Clear praise, no visible reservation; it plainly contradicts the negative label. At which stage the error arose is not known.
  • Row 8356 (gold = positive, p = 0.059). “yastıklar ince yeterli değil” — “the pillows are thin, not enough.” Labelled positive; another clear contradiction.
  • Row 8006 (gold = positive, p = 0.034). “hiç bir sebeb göstermeden iade kurallarına uyarak beğenmedim dedim, iade ettim.ücret iadem anında yapıldı.kargo ücreti de dahi.” — “without giving any reason, following the return rules, I said I did not like it and returned it. my refund was made at once. the shipping fee too.” The review praises the returns process; it gives no positive verdict on the product, and it reports “I did not like it” as the stated reason for the return. On this example the largest negative contributions to the model’s decision score come from the features iade (“return”, −1.78), beğenmedim (“I did not like it”, −1.30) and iade ettim (“I returned it”, −0.65); about half of the negative contributions come from features containing iade. Under a “was this a good experience” target the positive label is defensible, which is why we counted this row as a model error. Under the product-targeted guideline v1, the returns process is not the product (case 1): if “I did not like it” counts as a product verdict the label would be 0, and if it does not the row is SKIP (case 4). The category depends on the definition of the target, which is one more reason to read the counts as a direction rather than a rate.
  • Row 2544 (gold = positive). “paketleme başarısız. ampüller fena değil” — “packaging failed. the bulbs are not bad.” Two verdicts of opposite polarity about two different aspects. Which label fits cannot be decided before the target and the rule for mixed reviews are set.

Five of the nine selected model errors involve negation: fena değil (“not bad”), özelliklerinin altında bir ürün değil (“not a product below its specifications”), and one triple negation. A model that uses word unigrams and bigrams can learn frequent patterns such as fena değil as a separate feature; but it does not structurally represent the scope of a long or nested negation. That is what happened in this model: on row 2544 the contribution of the bigram fena değil to the decision score is only −0.06, while the unigrams değil (−1.77) and fena (−0.55) dominate. These examples show the limit of this feature setup, not that every n-gram model will get every negation wrong. Models that represent word order and scope are the subject of later chapters (Chapter 7).

What cleaning actually bought

Given 92 suspect rows, the obvious next move is to remove them or flip them and retrain. We compared both conditions on the same test split used for reporting throughout the chapter. This is an exploratory comparison; it should not be read as a final result on a never-seen test set:

### repairs, judged on the test split used throughout the chapter

                          acc    macro-F1   F1(neg)
baseline (no repair)     92.57      76.21     56.47
drop 92  (p > 0.90)      92.37      75.72     55.61
flip 92  (p > 0.90)      92.47      75.94     55.99
drop 34  (p > 0.95)      92.39      75.83     55.83
flip 34  (p > 0.95)      92.52      76.03     56.15
drop  2  (p > 0.99)      92.54      76.15     56.37
flip  2  (p > 0.99)      92.52      76.11     56.31

None of these six repair conditions beat the baseline on the test split used. The largest difference is about half a macro-F1 point (dropping the 92 rows: −0.49). These are the differences observed for the fixed sets of rows that were tried; they are not a confidence interval for the benefit of the method, nor an upper bound that rules out a large gain. Nor does the seed deviation of other experiments stand in for the uncertainty of these deterministic repairs. The right comparison is to apply the same operation to the same number of random rows, and we ran that too: 92 random rows with the same class mix as the 92 flags, 20 separate draws.

### control: 92 random rows with the class mix of the flags (49 pos / 43 neg), 20 draws
                            macro-F1     sd      min      max
drop 92 random rows  (mean)    76.15   0.13    75.91    76.45
flip 92 random rows  (mean)    75.87   0.36    75.01    76.47
drop the 92 flagged rows       75.72
flip the 92 flagged rows       75.94

Dropping the 92 flagged rows (75.72) is below even the lowest of the 20 draws of dropping 92 random rows (75.91): on this test split, dropping the flagged rows loses more than dropping random rows. Flipping the labels of the flagged rows (75.94) cannot be distinguished from the mean of flipping 92 random labels (75.87). So the rows selected by the model’s disagreement do behave differently from random rows, but not in the expected direction of raising the score.

This result shows that producing a review list and raising the score by dropping or flipping the rows on that list wholesale are separate goals. No gain was observed here on the second goal. Two notes:

  1. 92 rows out of 40,000 is 0.23% of the training data. Editing that few rows is not expected to move a model trained on 40,000 examples by much: in the control, dropping 92 random rows moves the score by 0.06 points on average. That the flagged rows move it more (0.49 points) shows that rows near the decision boundary can matter more than their count suggests.
  2. The flagged rows may not all be of one kind. Of the 40 selected disputes in the test set, 20 were schema problems according to the single reader; for the 92 training candidates that proportion is unknown, and those rows were not read separately. Flipping (which keeps the row) edged out dropping (which removes it) at the 92- and 34-row thresholds, and the order reversed at 2 rows; such small differences are not evidence about why keeping the rows would be better. Read the rows before either operation.

The concrete return of this audit is not the retrained model; it is that schema problems became visible in the 40 selected disputes. That gives a reason to re-examine the guideline; it does not show that half of the confusion in the whole corpus comes from the schema. The automated tool finds the candidates; it is the human reading them who revisits the definition of the target. In separate work, Northcutt, Athalye, et al. (2021) found label errors in the test sets of ten of the most-used ML benchmarks and showed that model rankings can change when the test labels are corrected: scored against corrected labels, on ImageNet a ResNet-18 overtakes a ResNet-50 once the originally mislabelled examples make up just 6% more of the test set. No such change of ranking was measured in this chapter.

5.7 What label noise costs, and in which direction

Suppose your annotation process has an error rate. How much does it cost? The effect of a label error depends jointly on its direction, on the sizes of the classes and on how many rows change. In the table below the same percentage does not mean the same number of wrong labels in the two directions: the percentages are shares of the source class.

I corrupted the training labels at controlled rates under three regimes, retrained from scratch, and evaluated against the original test labels (no artificial noise added). Three seeds each; standard deviation never exceeded 0.50 macro-F1.

### macro-F1, clean test set, model retrained at each rate
### rates are the share of the SOURCE class flipped; "symmetric" flips that share of EACH class

flip rate      0%     1%     2%     5%    10%    20%    30%    40%
symmetric   76.21  74.18  72.43  68.58  63.34  55.43  49.86  43.71
neg -> pos  76.21  76.14  76.29  76.27  76.42  76.85  77.12  76.47
pos -> neg  76.21  74.17  72.87  68.63  63.63  56.69  52.01  47.99

The middle row barely moves: with this model and this class ratio, the macro-F1 of the condition in which 40% of the negative labels are flipped to positive stays close to the baseline; at 30% it is 0.91 above it. We looked at where the rise comes from: in the 30% condition the model calls fewer reviews negative (about 845 instead of 1,075); precision on the negative class rises from 44.8% to about 50%, and recall falls from 76.3% to 67%. So as negative training labels get scarcer the decision boundary shifts, and on this test split macro-F1 rises a little. That is not a recommendation to corrupt labels. Mind what the percentages mean, too: 40% of the negative class is about 1,000 rows, 40% of the positive class about 15,000. The bottom row loses 28 points over the same range.

The same error rate, opposite consequences
80 67 54 41 macro-F1 0% 1% 2% 5% 10% 20% 30% 40% share of the source class flipped · ordinal scale, spacing is not proportional 76.21 clean neg → pos 76.47 pos → neg 47.99 symmetric 43.71 why the flat line is flat flipping 40% of 2,529 negatives shrinks the minority class but leaves it pure; flipping 10% of 37,471 positives leaves the negative class 59.7% mislabelled.

Three noise regimes on the same 40,000 training rows, evaluated on the same test split; each condition repeated with three seeds. The percentages are shares of the source class, and the two one-directional curves flip different numbers of rows. The 93.7/6.3 class imbalance is an important factor in this asymmetry, not its only explanation.

class imbalance — one label is far more common than the other. Here 93.7% of reviews are positive, so a model that answers “positive” every time looks accurate while finding nothing. ↩︎

First the counts on this dataset. The same source-class percentage changes very different numbers of labels when the classes are imbalanced; the effect on the score has to be measured separately on other tasks.

neg → pos at 10%: 253 of 2,529 negatives are relabelled positive. The positive class grows to 37,724, of which 253 are wrong — 0.67% contamination. The negative class shrinks to 2,276 but every remaining member is correct. A smaller, pure minority class is still learnable.

pos → neg at 10%: 3,747 of 37,471 positives are relabelled negative. The negative class is now 2,529 correct plus 3,747 wrong — 59.7% mislabelled. The model is fitting a class whose majority membership is noise. The true negatives still form a coherent cluster, so the model still learns something (63.63, against 48.37 for the majority baseline), but the signal is diluted in the noise, at a cost of 12.58 points.

The same asymmetry appears at rates far too small to notice in a spreadsheet. At 1% pos → neg, only 375 rows flip — one percent of the positive class, which is 0.94% of all training rows, a rate most teams would call excellent. But those 375 rows are 12.9% of everything now labelled negative (375 of 2,904, the 2,529 real negatives plus the 375 arrivals), and they cost 2.04 macro-F1 points — more than the entire gap between bag of words and tf-idf unigrams in Chapter 4, which was 1.56.

Because the percentages mean different numbers of rows in the two directions, we also ran the comparison with the same number of labels:

### the same NUMBER of flipped training labels in each direction (3 seeds, mean macro-F1)
    250 labels  neg->pos 76.35 (sd 0.17)   pos->neg 74.98 (sd 0.25)   | share of source class: neg 9.9%  pos 0.67%
    500 labels  neg->pos 76.80 (sd 0.20)   pos->neg 73.64 (sd 0.20)   | share of source class: neg 19.8%  pos 1.33%
   1000 labels  neg->pos 76.95 (sd 0.68)   pos->neg 71.11 (sd 0.27)   | share of source class: neg 39.5%  pos 2.67%
   2000 labels  neg->pos 68.60 (sd 1.02)   pos->neg 67.91 (sd 0.15)   | share of source class: neg 79.1%  pos 5.34%

With the same number of labels flipped, the direction still matters: flipping 1,000 positive labels to negative loses 5.1 points, while flipping 1,000 negative labels to positive does not lower the score. But the asymmetry has a limit: with 79% of the negative class (2,000 labels) flipped, that direction loses 7.6 points too.

Two practical suggestions follow from this experiment, each to be tested separately on your own data:

  • Measure your annotation error rate per class, and say which class the errors come from. An overall 1% is not informative by itself: errors that turn minority rows into majority ones barely registered here, while errors that turn majority rows into minority ones cost 12.58 points at the same 10% class-conditional rate. Name the source class and the target class every time, and give the number of rows along with the percentage.
  • Do not close uncertainty with a guess. In this experiment the examples wrongly moved from the majority to the minority were the expensive ones; but it does not follow that an annotator should systematically avoid the minority decision — that raises the number of missed minority cases. Send an uncertain item to SKIP and to adjudication; that is what the guideline earlier in this chapter is for. Measure the cost of errors, the class ratio and the target metric separately on your own task: where a missed minority case is the expensive error — screening, safety, fraud — the calculation comes out differently.

This experiment shows that rates conditional on the source class can mislead when the classes are imbalanced. Which direction does more harm in spam, fraud, medical screening or content moderation does not follow from this table: a comparison at a fixed number of errors, the class weights, the metric and the operational cost all have to be reassessed. Theory for learning under class-conditional noise is surveyed in Frénay and Verleysen (2014).

In this experiment the label corruptions were chosen independently of the text: at a fixed rate, without looking at the text. Real errors can depend on properties of the text — they can concentrate on hard items, for example — and the effect of the same nominal rate then differs (Wei et al. 2022). The hard reviews the model flagged in the previous section do not show the distribution of verified label errors; they only suggest that such concentration is possible. Frénay and Verleysen (2014) separates noise independent of everything, noise depending only on the class, and noise depending on the instance. This experiment’s symmetric regime is an example of the first and its two one-directional regimes of the second; the cost of the third was not tested here. So do not read the curves above as a bound.

5.8 Your test labels change what the measurement means

The previous experiments changed the training labels artificially and measured the result against the original test labels. Now the reverse: the predictions of one fixed model, trained once on the original data, are scored against versions of the test labels that have been flipped in a controlled way. The model never changes; what changes is the labels used for scoring. A “clean” or “original” test label here means the version with no artificial noise added; it does not mean a label that has been independently verified as true.

Note the change of denominator, because the word “noise” is doing different work in the two experiments. In the previous section a “1% flip rate” meant one percent of the source class. Here it means one percent of all test rows, flipped in whichever direction the row happened to be — at this corpus’s balance, about 94 positives and 6 negatives per hundred flips. Uniform flipping is a controlled stress test, not a claim about how test labels actually go wrong — the margin note on instance-dependent noise at the end of the previous section applies here too, and real test errors concentrate on the hard rows.

### one fixed model, corrupted test labels, mean of 5 seeds

test noise      acc   macro-F1   F1(neg)   sd(macro-F1)
### noise here = share of ALL test rows, flipped uniformly at random
    0%        92.57      76.21     56.47          0.00
    1%        91.73      74.68     53.90          0.23
    2%        90.85      73.13     51.31          0.12
    5%        88.31      69.41     45.36          0.17
   10%        84.19      64.79     38.66          0.24
   20%        75.48      57.00     28.82          0.36

Line these up against chapter 4’s leaderboard (Chapter 4), which was computed on the original labels:

Scores against different label versions of the same test split; the rows cannot be ranked on one common scale
system macro-F1
tf-idf 1–2 gram, original test labels 76.21
tf-idf 1–2 gram, 1% test-label noise 74.68
bag of words, original test labels 73.51
tf-idf 1–2 gram, 2% test-label noise 73.13
tf-idf unigram, original test labels 71.95
tf-idf char_wb 3–5, original test labels 70.39

Two percent artificial noise in the test labels takes the same model’s macro-F1 from 76.21 down to 73.13. That number is below the 73.51 bag-of-words scored on the original labels. The model did not get worse: it is byte-for-byte the same model, making byte-for-byte the same predictions. Because the two models are not compared on a common version of the labels, do not conclude that the ranking has reversed: 73.13 is one model measured against noisy labels, 73.51 the other model measured against the original ones.

Would the ranking survive on a common noisy version of the labels? In expectation, for independent symmetric flips at rate \(\eta < 0.5\), it does: a classifier’s noisy accuracy is \(\eta + (1 - 2\eta)\,a_{\text{clean}}\), which is increasing in \(a_{\text{clean}}\), so the order is preserved on average. On one realised test set it need not be. Take 100 items where A is wrong only on item 3 and B only on items 1 and 2 — on the original labels A scores 99 and B 98. Flip the labels of items 1 and 2 and score both against those labels: A drops to 97 and B rises to 100. We measured it for this chapter’s pair: scoring bigram TF-IDF and bag-of-words 200 times against the same noisy test labels, the 2.69-point gap between them shrinks to 2.50 on average at 2% noise and to 1.36 at 20%, and the order reverses in none of the 200 draws. That result holds for uniform, independent flips; not for real errors concentrated on hard rows.

What the table shows is that scores reported against different versions of the labels cannot be compared directly: the difference between a 73.13 measured on a test set with 2% label noise and someone else’s 73.51 measured on the original labels is neither a ranking of models nor evidence that they are statistically indistinguishable. When comparing across studies, report the evaluation set, the definition of the labels and the uncertainty calculation together; “which test set, with which labels” belongs in every comparison.

These comparisons have three consequences for evaluation design:

The test labels determine how the score is read. In a hypothetical setup where each true label is flipped independently with probability 2%, the expected accuracy of a model that is perfect against the truth is 98% against the noisy labels. That is not a hard upper bound for a single run, nor a ceiling on macro-F1: a model that has learned to reproduce the errors your annotators made can report a higher score. Once your model’s error rate approaches your labels’, a high score does not by itself show closeness to the real target, and further improvements can become invisible.

Small differences on a noisy benchmark are not results until you have read the rows the two models disagree on. Get the reasoning right. The 1.53 points above are a systematic shift of one model’s score from the original labels to a 1%-noisy version of them (seed standard deviation only 0.23). Compare two models on the same test set and that shift applies to both: under independent symmetric noise the expected accuracy gap shrinks only by the factor \((1-2\eta)\), i.e. to 0.98 of itself at 1% noise; for macro-F1 the relation is not linear, but the shift still moves the level, not the gap. What makes a small gap fragile is how many label errors sit on the rows where the two models disagree — and real label errors can cluster on the hard rows where models disagree. On this chapter’s test set, changing the label of one row moves one model’s macro-F1 by 0.02–0.045 points. The effect on the gap between two models depends on which rows they disagree on and on their predictions; there is no fixed threshold. We measured the worst case: to erase chapter 4’s 1.2-point gap (tuned TF-IDF 74.71, raw counts 73.51), changing the 18 most damaging labels among the 231 rows the two models disagree on is enough; for the 2.69-point gap between bigram TF-IDF and bag-of-words the number is 41. Those are extreme cases in which the errors fall in the worst places; with randomly scattered errors the gap survives, as shown above. Read the disagreement rows, blind if you can, fix their label errors, and test the gap on a common version of the labels with a paired bootstrap (McNemar for accuracy). Recht et al. (2019) and Beyer et al. (2020) document what happens when this is taken seriously on a famous benchmark. The 1.53-point shift itself is chiefly a warning against comparing numbers measured against different versions of the labels.

Set aside a separate quality budget for the test labels. Independently examining the rows on which models disagree, and the rows of the minority class, is especially worthwhile. Plan quality and sample size for training, validation and the final test from the costs of your own task; a single 10,000/40,000 allocation is not a general rule.

5.9 Leakage: the errors that make you look good

Everything so far has been about labels being wrong. This section is about an evaluation risk separate from wrong labels: if information that should not be seen during training, or the test examples themselves, reach the model, the reported score may not represent the intended conditions of use.

label leakage — target information the model will not have at the time of use getting into the input, directly or indirectly. The same text sitting in both training and test is contamination between the splits, which falls under the wider term data leakage. ↩︎

Label noise and leakage (label leakage) can both stop the reported score from representing performance on the target. Noise does not always lower the score, and leakage does not raise it under every condition; in both cases the direction has to be measured with a controlled comparison.

The raw corpus has 235,165 rows. Exact de-duplication on the text removes 1,657 extra rows from them (0.70%). That is not the number of duplicate groups: 1,055 distinct texts occur more than once (2,712 rows in all), and 77 of those groups carry both labels. Chapter 4 did this de-duplication before splitting (Chapter 4). Suppose it had not:

### split-before-dedup, the ordinary careless mistake

rows removed by exact de-duplication           1,657   (0.70%)
test rows also present in train                   45   (0.45% of test)
near-duplicates after normalisation               34   (0.34% of test)

same model, same run:
  accuracy on the 45 leaked rows                93.33
  accuracy on the 9,955 clean rows              91.90
  macro-F1 on the leaked rows                   76.76
  macro-F1 on the clean rows                    74.08

In this run the 45 test rows that are also in training have a macro-F1 of 76.76, and the other 9,955 rows 74.08. The two subsets differ in their texts and in their class mix — 42 of the 45 rows are positive, and their median length is 2 tokens (19 for the rest) — so the 2.68-point gap cannot be read as the causal gain of memorisation; and at n = 45 macro-F1 is volatile. In 5 of the 45 rows the copy in training carries the other label. Nor can the 74.08 here be compared directly with the 76.21 baseline used elsewhere in the chapter: this run deliberately did not de-duplicate before splitting, so it is a different split. The presence of contamination is shown by matching rows; its effect on the score, by a shared evaluation design. A 0.45% contamination is far too little to move the headline number.

To see how the effect changes with the dose, copy a controlled fraction of the test rows into the training data; the vectoriser is refitted on that enlarged training set as well. This is an artificial contamination: it need not produce the same distribution, or the same effect on the score, as natural splitting mistakes.

What leakage is worth, at controlled doses
test rows copied into train accuracy macro-F1 F1 (negative)
0% 92.57 76.21 56.47
1% 92.57 76.27 56.60
2% 92.62 76.41 56.86
5% 92.65 76.53 57.08
10% 92.82 77.14 58.20
25% 93.33 78.59 60.82
50% 94.00 80.89 65.07
100% 95.62 85.89 74.17

In this controlled copying experiment the 10% condition takes macro-F1 from 76.21 to 77.14, and the 100% condition to 85.89 (9.68 points — more than twice the 4.3 points that moving from unigram to bigram TF-IDF bought in chapter 4). The effect is specific to this model, this dataset and this copying setup; it is not a fixed gain per 10%, nor an estimate of the effect of a natural leak. Which rows are copied moves the result too: re-running the 10% dose with three different draws, macro-F1 ranged from 76.81 to 77.13.

Controlled copying can produce an increase in score that looks plausible: the 10% dose gives a difference of the size you would expect from a genuine modelling improvement, and it cannot be told apart by looking at the score alone. So verify an unexpectedly good result with checks on the split, on duplicates and on preprocessing.

Now the near-duplicates. On the de-duplicated split of chapter 4, normalising away case, punctuation and whitespace finds 34 test rows whose normalised form also appears in training. Their macro-F1 is 67.62 against 76.23 on the rest. On such a small, selected subset the reason for the gap cannot be determined without examining the texts one by one; macro-F1 on thirty-four rows is close to meaningless. In our recount (32 rows, depending on the details of the normalisation) 31 of these rows are positive and their median length is 3 tokens (“güzel ürün”, “tavsiye ederim”, “teşekkürler hb”); so the F1 of the negative class is decided by a single row. Duplicates have a side benefit as well: every duplicate group in which the same text carries different labels is a candidate for review. The possible reasons are a clear error, a different target, missing context or schema ambiguity; the count does not by itself give a label-noise rate (exercise 9).

The leaks worth checking for

Check these six risks separately, against your own data and your own setup of use:

  1. Exact duplicates across splits. Before splitting, group by the text; if the same text carries different labels, examine that conflict first (77 groups in this corpus). Then de-duplicate by a rule you have documented (df.drop_duplicates(subset="text") keeps the first row of each text), or make sure each group stays in a single split.
  2. Near-duplicates. Normalise case, punctuation and whitespace; then compare. For longer texts, compare the sets of overlapping word n-grams (“shingles”) with Jaccard similarity — shared shingles divided by all shingles — or with MinHash, a fast approximation of it that scales to millions of documents.
  3. Group leakage. Multiple reviews by the same user, multiple sentences from the same document, multiple images of the same patient. If your target use requires it, split on the group, not the row; because the rows genuinely differ, this leak is easy to miss. This corpus has no reviewer identifiers (see the datasheet below), so user-level groups cannot be verified directly; report that limitation.
  4. Temporal leakage. If your model will run on future data, your test set must be from the future. A random split lets the model see next month while predicting last month.
  5. Preprocessing leakage. Fit the vectoriser, the scaler and the feature selector on the relevant training part only; on the test set, only transform (chapter 4’s pipeline had vec.fit_transform(train) and then vec.transform(test)). In cross-validation apply the same rule inside every fold: in this chapter’s out-of-fold audit the vectoriser had been fitted before the folds, and fitting it inside the fold raised the candidate list from 92 to 123 (Section 5.6).
  6. Benchmark contamination. Public test texts may have entered the pretraining data of the pretrained model you are fine-tuning. Kapoor and Narayanan (2023) catalogue kinds of leakage across 17 scientific fields; Magar and Schwartz (2022) measure, with controlled pretraining, the effect of contamination on scores; Oren et al. (2024) propose a statistical test of whether a specific dataset is in a black-box model’s pretraining data. Report the assumptions and error probabilities of such a test; do not read its result as a definitive provenance record for individual texts. In LLM evaluation this becomes the central problem (Chapter 22).

5.10 Where to spend the next thousand labels

On a fixed annotation budget you choose between the number of examples and the cost of producing and checking each label: careful labels cost more, so you get fewer. This section examines the effect of that choice on the score by varying the number of labels and the level of artificial noise for the same model.

Before the grid, one paragraph on what “cost more” means, because there is a person on the other end of it. A cost example: at a labelling speed of 200 short reviews an hour, 40,000 labels take about 200 person-hours. The real speed depends on the guideline, the length of the texts, breaks and adjudication; measure it in a small pilot before planning. Three points follow, and they are engineering matters and not only ethical ones. Pay and time targets set the working conditions and can affect an annotator’s attention. This chapter did not measure the effect of piece-rate versus hourly pay on quality; plan fair pay, rest, a clear guideline, a reasonable number of control items (questions with known answers, attention checks, monitoring of agreement between annotators) and a route for appeal and adjudication together. Task design is a working condition: a SKIP option and decided examples can keep an annotator from being forced into a binary decision on an ambiguous case; measure the effect on speed, fatigue and agreement in the pilot. And on some tasks — moderation, safety, abuse detection — exposure is a real cost, and a project that budgets annotator hours without budgeting rotation and support is under-costed (Gray and Suri 2019; Sambasivan et al. 2021). Record in your datasheet who labelled, how they were paid and at what rate, stating the unknowns explicitly as well; Gebru et al. (2021) asks for this.

The grid below trains the same model at seven budget sizes and six noise levels, three seeds each, and evaluates all 42 cells against the same original test labels. In each cell the training rows are drawn at random, the vectoriser is fitted on those rows only, and the labels of each class are flipped at the same rate. (In chapter 4’s learning curve the vectoriser had been fitted on all 40,000 reviews, so the two curves cannot be compared directly — there bigram TF-IDF averaged 51.7 at 1,000 labels.)

### macro-F1 by budget and label noise (clean test set, 3 seeds)

                 0%      5%     10%     20%     30%     40%
    500       59.44   58.56   55.57   51.18   47.40   42.12
  1,000       63.71   60.91   57.77   52.32   46.99   42.78
  2,500       68.76   64.17   60.03   53.36   48.23   42.93
  5,000       71.69   66.00   60.21   53.17   47.86   42.42
 10,000       73.72   66.47   61.81   54.46   48.53   42.88
 20,000       74.89   66.99   62.04   55.21   49.09   43.81
 40,000       76.21   68.18   63.20   55.74   49.34   44.16

The grid does not separate the direction of the noise: it is a separate experiment for noise that corrupts each class at the same rate. The last row (40,000) is close to the symmetric row of the earlier table (68.18 against 68.58 at 5%, 63.20 against 63.34 at 10%), but it comes from separate runs. Re-running the same cells with ten seeds each gave a standard deviation of about half a point (68.38 ± 0.48 at 5%; 62.66 ± 0.46 at 10%), so these differences are at the level of seed-to-seed variation. If you know the direction of your noise, use the earlier table.

Read across a row and you learn what noise costs. Read down a column and you learn what volume buys. Read diagonally and you get the finding:

  • In this grid 1,000 clean labels scored 63.71 macro-F1 and 40,000 labels at 10% noise 63.20. The observed difference is 0.51 points and rests on three seeds. We re-ran the two cells with ten seeds each: 1,000 clean labels scored 62.24 (sd 2.24; 59.29–65.44), 40,000 labels at 10% noise 62.66 (sd 0.46). The small clean set is ahead in 37 of the hundred seed pairs. So this pair is a tie, and the score of the small set depends heavily on which 1,000 rows were drawn.
  • 500 clean labels (59.44) are ahead of 40,000 labels at 20% noise (55.74). In the ten-seed re-run the difference is 3.9 points, in the same direction in all hundred seed pairs.
  • What volume buys depends on the label noise. Going from 500 to 40,000 labels brings 16.77 macro-F1 points at 0% noise and 2.04 at 40%. The grid shows the balance of quality and quantity; assessing the return per unit of money takes the costs of annotation and quality control, which have to be known separately.
Your annotation budget: number of labels and noise

On each curve, look at how the gain changes as the number of labels grows; the high-noise curves flatten very early. The dashed circle, when it appears, marks a cell that reaches at least this score with fewer labels; monetary cost was not computed.

Not all labels are worth the same

The grid assumed labels are drawn at random. They need not be. Uncertainty sampling, one of the common methods of active learning (Settles 2009), fits a model on a small seed set and chooses the next labels from the examples that model is least decided about.

# schematic: X_seed, y_seed, X_pool and pool have to be prepared beforehand
m0 = clf().fit(X_seed, y_seed)                       # 200 random labels
unc = np.abs(m0.predict_proba(X_pool)[:, 1] - 0.5)   # distance from 0.5
picks = pool[np.argsort(unc)[:800]]                  # the 800 examples scored closest to 0.5

This fragment is not a runnable experiment by itself; it summarises the workflow. The lowest unc selects the examples whose model score is closest to 0.5; the calibration of the scores was not checked here.

With a budget of exactly 1,000 labels, mean of three seeds:

### 1,000 labels, spent two ways

                              negatives found    acc   macro-F1   F1(neg)
1,000 drawn at random                      62  93.50      65.49     34.40
200 random + 800 uncertain                248  93.59      68.80     40.99

The 1,000 examples chosen by uncertainty contain 248 negative labels, the randomly chosen ones 62 — four times as many. That may have improved coverage of the minority class; but the selection also changes how hard the texts are and how close they lie to the decision boundary. How much of the 6.59-point difference in negative-class F1 comes from the number of negative examples cannot be said without a separate comparison.

In these two rows, with the same budget of 1,000 labels, uncertainty sampling scored 68.80 macro-F1 and random selection 65.49: an observed difference of 3.31 points. Because paired differences per seed are not reported, do not generalise it as a definite effect size. Re-running the experiment with a protocol we set up ourselves (the seed model on the vectoriser of the 40,000 texts, the final model on a vectoriser fitted on the 1,000 selected rows), the difference came out at 5.3 and 5.2 points in two groups of three seeds (63.13 → 68.46 and 61.96 → 67.19); the number of negatives found averaged between 188 and 290 depending on the group. The direction is the same; the size is sensitive to the protocol and to the seed.

Putting it next to the budget grid — 1,000 uncertainty-sampled (68.80) matches 2,500 random (68.76) — is the more quotable claim and the weaker one: the two tables do not share a baseline (random 1,000 is 63.71 in the grid and 65.49 here). That 1.78-point gap is inside the seed-to-seed deviation of the 1,000-label cell (2.24 points over ten seeds); so no separate explanation is needed for two three-seed means differing, but do not treat the 2.5× “discount” as a measurement either.

Keep three limits in view. A training set chosen by uncertainty does not represent the natural test distribution; build the test set with independent, appropriate sampling, and never select it this way. How small a seed set is enough depends on the task and the class ratio; 20 and 200 seed labels were not compared here. And the selected set can be tied to the model that selected it: a set chosen as uncertain for a linear TF-IDF classifier is not necessarily a good set for the transformer you fine-tune later (Section 12.3). Such a set has been measured to underperform a random sample of the same size once the architecture changes (Lowell et al. 2019); test the transfer to another architecture against random sampling with an equal budget.

5.11 Labels you do not write by hand

Hand annotation is not the only source of labels, and on a 235,000-row corpus it takes a great deal of labour; choose its scope from your budget and your goal. The three methods below can reduce the need for human labels, but they do not by themselves take the place of quality control. Each comes with its own kind of problem.

Weak supervision. You write partial, noisy labelling functions and combine them. The lexicon rule in this chapter is one such source. Frameworks like Snorkel (Ratner et al. 2017) fit a generative model over many functions; under the chosen model assumptions it estimates the accuracy of the sources and the dependencies between them from the functions’ agreements and conflicts, and it needs no reference labels. The thing to watch is dependence: five variations on the same keyword list should not count as five independent pieces of evidence, and an aggregation model that assumes independence trusts that “consensus” too much. Test the dependence and the coverage of the functions on a small sample with verified labels.

Distant supervision. Use an existing signal as a proxy label: star ratings for sentiment, a knowledge base for relation extraction, user clicks for relevance. Cheap and abundant. The risk is that the proxy is not the target — a 3-star review is not “neutral”, it is whatever that platform’s users mean by three stars, and the mapping shifts across platforms and across years.

LLM labelling. You prompt a large model to label your data; Chapter 17 covers the mechanics. On some well-specified tasks, labelling by prompt has matched or beaten non-expert crowdworkers (Gilardi et al. 2023); on subjective or culturally loaded tasks the margin shrinks and sometimes reverses (Ziems et al. 2024). In the cost comparison, count the labour of checking and correcting as well as the model calls. Three things to hold on to. First, LLM labels inherit the model’s systematic biases; those do not disappear by collecting more rows, and their size and direction have to be measured on your task. Second, LLM labels may not be independent of a model you then evaluate: if the same model family labelled your test set, your evaluation partly measures agreement with that family, and models have been shown to prefer their own output (Panickssery et al. 2024). Third, you still need a human-labelled test set to know whether any of it worked: treat LLM labels as an unvalidated instrument and report accuracy and disagreement on a held-out sample that humans have examined independently (Pangakis et al. 2023). One more caution for Turkish: the measurements above were made on English data; on user text with misspellings and missing Turkish characters the result may differ (Chapter 16), so make this measurement on your own Turkish data.

Generating training text with an LLM — Self-Instruct (Wang et al. 2023) and its descendants — is a common option. Its benefit depends on selection, filtering and comparison with real data as well as on the generation method.

Three kinds of problem are worth knowing by name:

Lack of diversity. You can ask for 10,000 negative reviews and get variations on a small number of templates. Examine examples by hand; measures such as the ratio of distinct n-grams and embedding-space coverage (with the tools of chapter 3, Chapter 3) are not sufficient by themselves, but they make repetition visible. This is a separate problem from the recursive model collapse below.

Distributional mismatch. Generated text is fluent, well-punctuated, and correctly spelled. Real Turkish product reviews are none of those things; Chapter 16 showed how much of the signal lives in exactly the messiness a generator smooths away. A model trained on clean synthetic text and deployed on real text is being asked to generalise across a shift you introduced yourself.

Recursive contamination. Training on the output of models trained on model output degrades over generations — Shumailov et al. (2024) calls it model collapse. The tails of the distribution disappear first, which means the rare cases go first, which means exactly the cases you needed the model for. The condition matters, though, and it is easy to drop: that result replaces the real data at each generation. When real data is accumulated alongside the synthetic instead of swapped out — which is what actually happens in practice — the test error stays bounded and the collapse largely disappears (Gerstgrasser et al. 2024). The cautious working rule that follows is narrower than “synthetic data is dangerous”: watch the risk of new synthetic generations taking the place of the real data. Using them together with real data, and keeping a fixed, independent evaluation set outside the loop, makes that risk easier to measure (Shumailov et al. 2024; Gerstgrasser et al. 2024). Which ratio and which filtering help is something to try on your own task.

For the test set: the main test set should rest on real texts selected independently from the distribution of the target use, and on labels that have been reliably examined. Synthetic examples can be used in additional stress tests; they do not replace the main estimate of performance.

5.12 Write it down: datasheets and model cards

The audits in this chapter produced observations that need documenting: 20.03% of the test reviews contain a contrastive connective; 20 of the 40 test reviews the model disputed with its highest scores are schema problems according to a single reader; in training, the flag rate of negative-labelled rows is thirteen times that of positive ones. Unless you write these observations down, together with how they were sampled and what their limits are, they disappear.

Gebru et al. (2021) proposed datasheets for datasets: a standard document covering motivation, composition, collection, preprocessing, uses, distribution, and maintenance. Mitchell et al. (2019) proposed model cards, the same idea for trained models, with performance disaggregated across the groups that matter. Both are conventions, not standards, and both are frequently treated as compliance paperwork.

Do not leave the datasheet as a form that merely gets filled in. Tie the provenance of the data, the conditions of use and the known limits to a stage that the decision-makers review before release. Sambasivan et al. (2021) found, across 53 practitioner interviews, that data problems were pervasive, expensive, and consistently under-valued relative to modelling — “everyone wants to do the model work, not the data work”. Actually using the document can reduce that invisibility.

Here is what an honest extract for this chapter’s corpus looks like:

### datasheet extract — Turkish product reviews

COMPOSITION
  235,165 reviews; 233,508 after exact de-duplication (1,657 extra rows,
  0.70%). 93.7% positive. 77 texts carry both labels. Product category,
  date, and reviewer are not available.

LICENCE AND PROVENANCE
  The licence file of the original GitHub repository and the licensing
  section of the card say CC BY-SA 4.0; the licence field at the top of
  the Hugging Face page says "unknown" (checked on 10 October 2026).
  The site and the date of collection are not documented.

LABELS
  Provenance unknown; how the labels were derived is not documented
  (possibly from star ratings). No guideline document exists. Whether
  inter-annotator agreement was measured is not reported.

KNOWN LIMITATIONS
  - 20.03% of the test reviews contain one of the connectives ama/fakat/
    ancak/lakin; that alone does not show mixed sentiment or a problem
    with the binary schema. Model macro-F1 on this subset is 72.11,
    against 77.72 on the rest.
  - The 40 test reviews the model disputed with its highest scores were
    examined by one reader: 11 clear label contradictions, 20 schema
    problems. These proportions cannot be generalised to the corpus.
  - In 100 randomly selected test rows the same reader found 0 clear
    contradictions and 7 undecidable rows. The "rule of three" upper
    bound of about 3% for zero observations holds under independent
    sampling and the same definition of an error.
  - Selection bias: reviewers are self-selected. Non-reviewers absent.
    (Not measured.)

RECOMMENDED USE
  Within the terms of the licence, teaching Turkish text classification
  and comparing methods.
NOT RECOMMENDED / NEEDS FURTHER VALIDATION
  Deployments where errors on the negative class (missed or false
  complaints) are costly are not recommended without independent,
  task-specific validation. The highest-scoring model here reaches a
  negative-class F1 of about 56; for an operational decision the
  threshold and the cost of errors have to be assessed separately.

Note the shape of it. This draft makes the measured numbers and the undocumented fields visible in one place, and the “not recommended” section is specific enough to stop someone. A good datasheet also records, explicitly, the limitations it cannot put a number on.

5.13 Where the effort actually goes

That is a long list of things that can go wrong. To plan a week of work, use the activities below as an example order. This chapter did not measure their durations or their returns; they are not written into the table, and you have to measure them in your own pilot.

An example workflow for data-quality activities
activity duration the information it can give when to do it
read 100 random rows by hand measure in a pilot candidates for clear contradictions and ambiguity; a wide uncertainty interval for the rate when the data arrives, before modelling
write a guideline with examples measure in a pilot how clear the decision rules are before bulk annotation
measure two humans’ agreement on a shared sample measure in a pilot a disagreement table and a kappa that depends on the sampling in the guideline pilot
out-of-fold flagging + hand reading measure in a pilot candidate rows to examine once training data exists
leakage audit measure in a pilot risks in the split and the preprocessing before reporting a result

Strengths — reading random rows, writing a guideline and running an agreement pilot can bring out, early, problems that would be invisible later; a leakage audit reduces the risk of publishing a wrong number. Measure their costs and returns on your own task.

Weaknesses — out-of-fold flagging produces a list of candidates and feels like automation. In this chapter, dropping or flipping the list wholesale without examining it did not raise the score; dropping the flagged rows lost more than dropping random rows. Put the labour of review and adjudication into the plan.

5.14 The thing this chapter is really about

Chapter 4 compared representations (Chapter 4); this chapter asked what that comparison is measured against. With that chapter’s highest-scoring model held fixed and 2% of the test labels flipped artificially, macro-F1 fell from 76.21 to 73.13. That number is below the 73.51 another model scored on the original labels; but the two numbers belong to different versions of the labels. So this experiment does not show a ranking reversing on a shared test set — when we scored the two models against the same noisy labels, the order changed in none of 200 draws. What it shows is the risk of comparing across studies without reporting which version of the labels was used.

The result means that the quality of the evaluation labels has to be reported along with the model’s score. The concrete version, in five lines:

  • Read your data. A hundred rows, by hand, before modelling. Most of the observations in this chapter came from reading rows.
  • Audit the schema too. Of the 40 model-selected test disagreements examined by a single reader, 20 were counted as schema problems and 11 as clear label contradictions. That is a reason to review the guideline; it is not an estimate of rates in the corpus.
  • Report the source class together with the number of rows affected. In this experiment, flipping 40% of the negatives to positive kept macro-F1 close to the baseline; flipping 1% of the positives to negative lowered it by 2.04 points. The two percentages correspond to different numbers of rows; the direction also mattered when the same number of labels was flipped, but the result does not carry over directly to other tasks.
  • Test quality and volume together on your own budget. In this grid 500 clean labels came out ahead of 40,000 labels with 20% artificial noise, while 1,000 clean labels and 40,000 labels at 10% noise could not be told apart. The choice also takes the real cost of labels, the target distribution and the quality of an independent test set.
  • Audit for leakage before you report a result, especially when the number is unexpectedly good.

These constraints do not relax as the models get larger. Chapter 6 and Chapter 7 introduce systems with orders of magnitude more capacity, and every one of them is trained against labels somebody manufactured and evaluated against a test set somebody could have contaminated. The models change; the problem of the quality of the evaluation labels does not — it only gets harder to see, which is why Chapter 22 and Chapter 35 have to return to it.

5.15 Intuition checks

Unanswerable without the marginals of the two sources. The 93.42% raw agreement and the kappa of about 0.27 in this chapter were computed between the reference labels and the lexicon rule; they are not a human–human agreement. The agreement expected under independence, \(p_e\), was 90.99%. On the balanced subsample raw agreement fell to 60.28%; the texts sampled there were different too. Raw agreement is largely a property of the sample: report kappa, the marginals and the confusion table together.

In this experiment, flipping 10% of the negatives to positive took macro-F1 from 76.21 to 76.42, and flipping 10% of the positives to negative took it to 63.63. The first difference is +0.21 — not “zero”, but inside the seed deviation; the second is −12.58. Note what “10%” means in each case: the class-conditional rate is the same, the number of rows flipped is not — about 253 against 3,747, fifteen times as many. With the same number of labels flipped the direction still matters (76.95 against 71.11 at 1,000 labels), but part of the gulf comes from the number of rows. On your own task, measure direction, quantity and class composition separately.

Not on the strength of 1.2 macro-F1 points alone. On this corpus 1% artificial test-label noise shifted one model’s score by 1.53 points; that shift in level does not by itself erase the gap between two models on a common version of the labels (with randomly scattered noise the order of bigram TF-IDF and bag-of-words never changed). How sensitive the gap is to label errors depends on the rows where the two models disagree: chapter 4’s 1.2-point gap is erased by 18 label changes placed in the worst possible way. Read the rows where A and B disagree, blind if you can, fix their label errors, and test the gap with a paired bootstrap.

No; read them first. In the condition that dropped these 92 rows, macro-F1 fell from 76.21 to 75.72 — below even the lowest of 20 draws of dropping 92 random rows (75.91); flipping the labels did not beat the baseline either. In a separate audit of the test set, half of the 40 selected candidates were schema problems according to a single reader; the distribution of the 92 training candidates is unknown. Use the flags as an order of review, not as an automatic decision to delete.

In this grid 5,000 clean labels scored 71.69; the 40,000 scored 76.21 at 0% noise, 68.18 at 5% and 63.20 at 10%. Linear interpolation between the two measured cells puts the break-even rate at about 2.8%; that is a value read off a line, not a measured threshold. A real decision also takes the cost of the labels, the mechanism and direction of the noise, the uncertainty, and the quality of the test set. Whichever you choose, set aside a separate quality budget for the test labels.

5.16 Interview questions

The questions below test whether you can read this chapter’s numbers within their scope and set apart the claims that have not been measured yet. Some of the answers were observed in this chapter’s experiments; others need verification on other data.

When you answer these, give the direction and which split rather than a single rate: from which class to which the noise goes, how many rows it affects, and whether the errors are in training or in test.

  1. Two sources of labels agree on 93% of items. Is the dataset good? You cannot tell from that number alone. In this chapter 93.42% raw agreement gave a Cohen’s kappa of 0.27, because at a 93.7/6.3 class split the agreement \(p_e\) expected under independence is already 90.99%. The same two sources agreed on only 60.28% of a balanced subsample. State the condition when you quote the numbers: the second “annotator” here is a lexicon rule, not a human, so this is a demonstration of the arithmetic, not a study of human agreement. Ask for the confusion table and the marginals of both sources and compute kappa yourself; report raw agreement, kappa, class-specific agreement and the marginals together.

  2. What is the effect of flipping 10% of the source class? Ask first from which class to which, and how many rows in all. Here negative to positive was about 253 rows and +0.21 macro-F1; positive to negative was about 3,747 rows and −12.58 points. The same class-conditional rate, fifteen times as many rows. The gap arises from the direction, the number of rows changed, the model and the metric together: with the same number of labels flipped the direction still mattered (76.95 against 71.11 at 1,000 labels). The condition that flipped 40% of the negatives stayed close to the baseline; that is a result of this experiment only.

  3. Does the annotation budget go to training or to test? Give each a share according to the costs of the task, and audit the label quality of the final test set independently. Under the assumption that each label is flipped independently with probability 2%, the expected accuracy of a model that is perfect against the truth is 98%; that is not a hard ceiling on the score. In this chapter 2% artificial test noise took the same model’s macro-F1 from 76.21 down to 73.13 — below the 73.51 another model scored on the original labels. State the condition as well: that is a comparison across two versions of the labels; when the two models were scored against the same noisy labels, the order did not change.

  4. A new model beats the production one by 1.2 macro-F1 points. Ship it? Not on a single number. On one version of the labels, examine the rows on which the two models disagree, independently and blind if you can, and compute a paired uncertainty interval (a bootstrap for macro-F1, McNemar for accuracy). That 1% artificial noise shifted one model’s score by 1.53 points does not mean the gap between models shifts by that much on a common test. The question is not “is the gap big” but “does the gap rest on label errors in the rows where the two models disagree”: in this chapter a 1.2-point gap was erased by 18 label changes placed in the worst possible way, while randomly scattered noise never changed the order of a 2.69-point gap.

  5. How do you find the candidates for review among 40,000 rows? With out-of-fold prediction: in each fold fit every transformation (the vectoriser included) on the training part only, and use the score the model gives to the opposite class as an order of review. This is a single-threshold rule inspired by confident learning (Northcutt, Jiang, et al. 2021); the full method uses per-class thresholds. Then say what the measurement taught: the 0.90 threshold flagged 92 candidates in training (123 with the vectoriser fitted inside the fold), and 92 is not a count of real errors; the flag rate of a negative-labelled row is thirteen times higher, which is not a difference in error rates; dropping the flagged rows lowered the score more than dropping random rows did (75.72 against a mean of 76.15). In a separate audit of the test set, 20 of the 40 selected candidates were schema problems according to a single reader; do not carry the proportions of one set over to the other. A flag is a reading list, not a decision.

  6. How do you check for leakage? Audit exact and near duplicates, the split by group and by time, the preprocessing inside every fold, and possible benchmark contamination (Kapoor and Narayanan 2023) separately. In this corpus, de-duplicating on the text removed 1,657 extra rows (1,055 duplicate groups; 77 of them carry both labels). In the controlled experiment that copied test rows, the 10% condition brought +0.93 macro-F1; that is of a size you could not tell from a genuine improvement, but it is not the return of every 10% leak. When you find a leak, measure performance again on a split that represents the target use.

  7. The human annotators disagree. What do you do? First read the shared examples and separate the reasons for disagreement: a gap in the guideline, the guideline being applied differently, ambiguity in the text itself, or a limit of the target definition. This chapter did not measure the disagreement of two humans; on the 40 test reviews the model disputed most strongly, a single reader recorded 20 schema problems — mixed reviews, truncated texts, reviews about delivery instead of the product. Adding a class such as mixed or not applicable can be an option; but it requires a new target definition, a new guideline, a new class distribution and a re-evaluation. When agreement is low the question is whether the annotators are inconsistent or the label set is under-specified; only reading can separate the two.

5.17 Where to go deeper

Beyond the papers
resource what it is
Data-Centric AI (MIT) a course that treats data quality and model development together; which intervention helps more depends on the task
cleanlab the method of Northcutt, Jiang, et al. (2021) as a library; it ranks candidate label issues from out-of-fold probabilities. Check the API and input requirements of the version you install, and read the flags by hand
doccano an annotation tool you can host yourself; plan the installation and administration time for your own organisation
argilla annotation with disagreement kept rather than averaged away, which is Aroyo and Welty (2015)’s argument made operational
Eugene Yan’s writing practical posts on labelling, guidelines and the economics of annotation
  • Gebru et al. (2021) and Mitchell et al. (2019) — the two documentation conventions everything else references. Short, readable, and worth copying the section headings from directly.
  • Northcutt, Jiang, et al. (2021) for the theory behind out-of-fold flagging, and Northcutt, Athalye, et al. (2021) for what it found in ten standard benchmarks — including how little label noise it takes to reorder models scored against corrected test labels.
  • Artstein and Poesio (2008) — a comprehensive survey of agreement measures in NLP: their assumptions, the units they measure, and which coefficient suits which task.
  • Aroyo and Welty (2015) and Plank (2022) — the argument that annotator disagreement is signal about the task, not noise to be averaged away. This reframes most of what this chapter treats as a problem.
  • Sambasivan et al. (2021) — 53 interviews on why data work is under-valued in practice. Read it before your first industrial project rather than after.
  • Kapoor and Narayanan (2023) for a taxonomy of leakage across 17 scientific fields, and Bender and Friedman (2018) on data statements as a documentation practice with a specific bias focus.
  • Klie et al. (2023) surveys automatic annotation-error detection methods, if you want more than the single out-of-fold technique used here.
  • Pustejovsky and Stubbs (2012) is the book-length treatment of running an annotation project — schema design, guideline iteration, adjudication. If this chapter is the argument, that book is the manual.
  • Marcus et al. (1993) is worth reading as a historical document: the Penn Treebank guidelines are the reason a generation of NLP agreed on what a parse tree looks like, and they show what “decided cases” look like when taken seriously over years.
  • Bowman and Dahl (2021) on why benchmark quality, not model capacity, is increasingly the binding constraint — the natural sequel to what this chapter says about the quality of test labels.

5.18 Lab: run a small annotation project end to end

Work in pairs. The output of the lab is a revised annotation guideline, agreement measures for two rounds, and a short analysis of the disagreements.

  1. Prepare two balanced rounds of 100. From the Turkish review corpus, with a fixed seed, pick 50 gold-negative and 50 gold-positive items for each round, and shuffle the order within a round with the same seed. The reason is given under Adjudication: a plain random 200 would contain about thirteen negatives. Shuffling a balanced 200 and cutting it in half is not enough, because the halves need not come out 50/50 and the two rounds’ kappas would then differ by composition as well. Both partners get the identical items and do not see the gold labels. This purposive balance does not represent the natural class ratio of the corpus; the kappa you get belongs to the ratio you chose.
  2. Write guideline v0 together — the class definitions only, no decided cases. Ten minutes, no more.
  3. Annotate the 100 items of the first round independently. Allow SKIP. Record how long it took.
  4. Report raw agreement, the confusion table, and Cohen’s kappa with a bootstrap interval. Say how you counted the SKIP decisions: as a category of their own, or left out. Do not compare your number to this chapter’s 0.2698 — that one is gold-versus-a-lexicon, not human-versus-human, and they are different quantities. Use kappas from published studies only as context, and only where the task, the class ratio and the coding scheme resemble yours. A low value does not by itself show that the guideline is flawed.
  5. Read every disagreement together. Group the recurring situations as far as the data supports; record the one-off cases and the ones you cannot resolve as well.
  6. Write guideline v1, adding each recurring situation as a decided case with a worked example on both sides. Follow the format in this chapter.
  7. Annotate the 100 items of the second round independently under v1 and recompute the agreement. Report the change, but do not credit it to the guideline alone: the texts and your own practice changed along with the guideline. Separating the guideline’s contribution takes a comparative design, for example a second pair labelling the same items under v0.
  8. Compare to the provided gold labels. File every item where the two of you agreed but gold disagreed under one of four headings: clear contradiction, different target (the gold label answers another question), schema limit (a binary label cannot express the text), and unresolved ambiguity. Two people agreeing does not by itself make the gold label wrong; do not force a single “right” label without a third independent judgement or an adjudication step. Clear contradiction and schema limit are the same two categories used in Section 5.6.

Deliverable: guideline v1, the two confusion tables, both kappa values with their intervals, the list of disagreement groups, and a one-paragraph statement of what your schema cannot represent.

5.19 Exercises

  1. Pick a public dataset you might use and answer the six questions in this chapter: who produced the text, who produced the labels and how, what question they were answering, whether the licence and terms of use cover your use, what legal basis for processing personal data is documented, and what is missing. Record what you cannot answer as “undocumented”. Do not treat public availability as permission or as consent.
  2. Two annotators label 100 items and agree on 90. First both use the positive label 50 times, then both use it 95 times. For each case build the 2×2 table (with these marginals and 90 agreements only one table is possible; show why), write down each annotator’s marginals and \(p_e\), and compute Cohen’s kappa. Then hold raw agreement at 90% and change the marginals, and show how kappa moves. Why can kappa not change when both annotators’ marginals and the raw agreement are held fixed together? Finally, explain the difference to someone who believes 90% agreement is 90% agreement.
  3. Reproduce the random-100 audit on a dataset of your own. Classify each row as error, undecidable, or ok, using criteria you write down before you start reading. Report the counts. If the number of outright errors is zero, give the rule of three (\(3/100\)) as an approximate 95% upper bound, under the assumption of independent sampling. If it is not zero, the rule of three does not apply; compute a binomial confidence interval (Wilson, for example).
  4. Reproduce the noise-direction experiment on a balanced dataset. Does the asymmetry between the two directions survive? Explain the result from the class counts, as this chapter did.
  5. Build a second annotator for a task of your own — a lexicon, a regex, or an LLM prompt. Compute kappa against the gold labels, then compute it again on a class-balanced subsample. Report both, and say which one you would put in a paper.
  6. Produce out-of-fold probabilities for your own training set with the preprocessing fitted inside each fold, and read the twenty highest-scoring candidates without seeing the stored label. Classify each as clear contradiction, schema, or model error. Then compare three conditions on an independent test set: dropping the candidates, flipping their labels, and repairing only the ones you verified by hand. Finally, run cleanlab’s find_label_issues, following the input requirements of the version you installed; it uses class-specific thresholds and is not the same method as a fixed 0.90 threshold. Report the overlap of the two lists and the hand-verified yield separately: overlap is not a measure of correctness.
  7. Take a benchmark result you believe — yours or a published one — and audit it for all six kinds of leakage listed in Section 5.9. Report which checks you could actually perform and which you could not.
  8. From the budget grid, take the score of 2,500 clean labels and the scores of 40,000 labels at each noise level. Say between which two measured levels the break-even rate lies. Then, assuming a linear change between them, compute an approximate rate and mark it as an interpolation. Finally, write down what this table does not contain for deciding how to split a budget between training and test annotation: it has neither a cost per label nor an estimate of the error rate in the test labels.
  9. In the raw corpus, find which duplicate groups the 1,657 extra rows removed by de-duplicating on the text come from, and reproduce the chapter’s counts: 1,055 groups, 77 of which carry both labels. Read examples from those 77 groups and separate clear error, missing context (the same sentence may have been written about another product) and schema ambiguity. Do the same on the normalised near-duplicates (is “güzel ürün”, “nice product”, labelled both positive and negative?). Do not equate the conflict rate of these selected groups with the overall error estimate of the random-100 audit; say in one sentence why the two cannot be equated.