12  Text Classification

Chapter status: drafted. Every number on this page was measured on Chapter 4’s exact split — the same 40,000 training and 10,000 test reviews, recovered rather than re-drawn, for reasons Section 12.1 explains. The LLM comparisons in Section 12.8 and Section 12.9 use a stratified 2,000-document subsample of that same test slice, with every competing model re-scored on it; Section 12.8 says why, and what it costs in resolution. Data: fthbrmnby/turkish_product_reviews on Hugging Face, 235,165 rows as downloaded and 233,508 after exact-text deduplication. Software: scikit-learn 1.9.0, transformers 5.15.0, PyTorch 2.13.0, imbalanced-learn 0.14.2, scipy 1.18.0 on Python 3.14.5; encoder training on an Apple M-series GPU; local LLMs through ollama 0.24.0.

Learning objectives. After this chapter you can:

• build the strong sparse baseline before reaching for anything larger, and say what it costs at every label budget rather than only at the largest one;
• fine-tune an encoder and name the budget at which it starts to be worth the bill;
• tell the difference between a metric that moves when your model improves and one that does not;
• treat class imbalance as a decision-threshold problem rather than a data-surgery problem, and defend that with a measurement;
• test whether the number your model calls a probability actually is one, and repair it when it is not;
• turn a cost matrix into a threshold, and say how much that saves over guessing;
• decide between a fine-tuned small model and a prompted LLM on measured accuracy, latency and cost;
• run error analysis as a repeatable process instead of anecdote.

12.1 The one number to beat

Chapter 4 ended with a leaderboard. Seven representations, 40,000 Turkish product reviews, one held-out set of 10,000, and a winner: tf-idf over word unigrams and bigrams, fed to a class-weighted logistic regression, at 76.21 macro-F1. That the reference point is itself class-weighted matters, and Section 12.5 will charge it for that. This chapter’s job is to find out what it takes to beat that, and what each increment costs.

macro-F1 — compute the F1 score separately for each class, then average without weighting. A rare class counts as much as a common one, which is exactly why it falls apart on this corpus when accuracy does not. ↩︎

That framing only works if the test set is literally the same one. A model scored on a different 10,000 reviews is not competing with 76.21; it is competing with a number nobody measured. So the first thing this chapter did was rebuild Chapter 4’s split and check it.

That turned out to be harder than expected, and the difficulty is the first lesson.

Recovering a split from its fingerprint

Chapter 4 recorded its subsample as “50,000 rows, stratified, seed 42”. Chapter 11 recorded its subsample the same way. Both are true. They are also different samples, because “stratified with seed 42” does not name a procedure — it names a family of procedures that disagree.

The first attempt to rebuild the split drew the subsample group by group, the way Chapter 11 does, and produced this:

                             dims      acc   macroF1   negF1
target (chapter 4)        101,659    92.57     76.21   56.47
groupby.sample            100,350    92.21     74.65   53.55

Every headline statistic is close and every one is wrong. A reader would not notice; 74.65 and 76.21 both round to “about 75”. But the vectoriser dimension is not a statistic, it is a fingerprint: 101,659 distinct surviving n-grams is a fact about which exact documents were in the training half. Two different 50,000-row samples will not agree on it by accident.

So we tried four constructions against that fingerprint:

                             dims      acc   macroF1   negF1
target                    101,659    92.57     76.21   56.47
groupby.sample            100,350    92.21     74.65   53.55
train_test_split 50k      101,659    92.57     76.21   56.47   ← exact
train_test_split + shuffle 102,221   92.26     75.20   54.63
plain sample 50k          101,539    91.68     74.40   53.36

One matched on all four numbers at once. Chapter 4 drew its 50,000 with a single stratified train_test_split, and every experiment below runs on that recovered split.

Note the near-miss. “plain sample 50k” lands within 120 dimensions of the target and within one accuracy point (91.68 against 92.57) — close enough that a careless check would accept it. Be careful about why the match convinces, though: these are not four independent witnesses. dims is a deterministic function of which documents landed in the training half, and the three scores are three readings of one fitted model on one test set, so they move together. The argument is not four agreements but one very sharp agreement: 101,659 is a six-digit count two different 50,000-row samples have no reason to share, and the near-miss row shows the coarser statistics would not have separated them. Nor did we prove uniqueness — we enumerated four plausible constructions and one matched exactly. If you ever have to recover somebody’s experimental setup, pick the quantity that is most sensitive to the thing you are trying to pin down, not the quantity you happen to have written down.

The by-product was a correction to a different chapter. Intersecting the two subsamples showed that Chapter 11 and Chapter 4 share 23,819 of their 50,000 documents — 47.6%. Both are stratified, both use seed 42, both contain exactly 46,839 positive and 3,161 negative reviews, and they are less than half the same corpus. Chapter 11’s margin note now says so. No validator would have caught that; only intersecting the two sets and counting would.

And the forensics were avoidable, which is the lesson the story does not state. What Chapter 4 recorded was prose — “stratified, seed 42”. What it should have recorded is the call itself and a fingerprint of the result:

train_df, test_df = train_test_split(
    df, train_size=50_000, stratify=df["label"], random_state=42)
print(hashlib.sha256(
    ",".join(map(str, sorted(train_df.index))).encode()).hexdigest()[:12])

Chapter 4 now prints both. Three chapters depend on that one call, and one of them drew a different sample without anybody noticing. Print the call, print the hash, and the fingerprint stops being a detective story.

The split, and the slice this chapter adds

Chapter 4 used two slices, train and test. This chapter needs three, and the extra one is the whole reason its numbers can be trusted: somewhere to tune thresholds that is neither what the model learned from nor what it is finally scored on.

rows as downloaded      235,165
after de-duplication    233,508   (1,657 exact duplicates)
stratified subsample     50,000   46,839 positive   3,161 negative

  train                  40,000   37,471 positive   2,529 negative
    fit                  32,000   29,977 positive   2,023 negative
    cal                   8,000    7,494 positive     506 negative
  test                   10,000    9,368 positive     632 negative

Chapter 4 used train and test. This chapter splits train once more, into a 32,000-document fit slice and an 8,000-document cal slice, because three of the things it measures — a decision threshold, a calibration map, a cost-optimal operating point — are fitted quantities that must not be fitted on test.

decision threshold — the cut that turns a score into a label. Almost every classifier ships with 0.5 by default, but 0.5 is a convention, not a measurement: it is a free parameter you are supposed to choose, on held-out data, from what the mistakes cost you. ↩︎

Every threshold in this chapter is chosen on cal and reported on test. One consequence to carry into every cost comparison in this chapter: those 8,000 cal documents are labelled, so they are part of the label budget. A method that needs a calibration slice costs more labels than one that does not, and a comparison that prices only the training set is understating it. Where a number was obtained by tuning on test, it is labelled an oracle and used only as an upper bound to measure regret against.

And every threshold is stated on the complaint side: flag the review when \(P(\text{complaint})\) exceeds \(t\). A low \(t\) therefore means catch more complaints. Elsewhere you will meet the other convention — the same operating point written as 0.834 on the satisfied side — so when you compare notes with somebody, establish which side you are both talking about before comparing numbers. One convention, stated once, is worth more than any amount of care afterwards. That distinction does most of the work in Section 12.7.

Which slice is allowed to decide what

The middle tier is what Chapter 4 did not need. A plain setup fits its objects — the weights and the vectorizer vocabulary — during training. This chapter adds three more, and all three are fitted after training, on data the classifier never saw. Without the cal slice every threshold and every calibration curve in this chapter would be reporting on data it was tuned on.

8,000 documents is not an arbitrary size. Section 12.6 fits isotonic regression on this slice, and Niculescu-Mizil and Caruana report that isotonic regression needs roughly a thousand calibration points before it beats a parametric sigmoid, below which it overfits (Niculescu-Mizil and Caruana 2005). At 8,000 — with 506 of the minority class — we are comfortably in the regime where their result says isotonic is safe. Our own measurement agrees, which is the only reason we say so.

The baseline reproduces exactly:

tf-idf 1-2 gram + LogisticRegression(class_weight="balanced"),
101,659 features, 31.7 non-zeros per row

accuracy      92.57
macro-F1      76.21
complaint P   44.84    R 76.27    F1 56.47

                 predicted
              complaint  satisfied
 complaint         482        150
 satisfied         593      8,775

Where this chapter ends up

Before the work, the destination. Every row below is measured later in the chapter, and several of them came back the opposite way round from what the literature would lead you to expect. Some of its terms (calibration error, AUROC, a paired bootstrap) are defined only in the sections that measure them, so read the table now for its shape and come back to it at the end.

What this chapter measured

what we asked	what came back
Does fine-tuning beat 40,000 labels’ worth of tf-idf?	Yes — with 2,000 labels
Is the sparse baseline therefore obsolete?	No. Below ~1,000 labels it wins, and it trains in 1.2 seconds
Does resampling fix class imbalance?	No. Moving one threshold matched the best of them, for free
Does class weighting cost anything?	Yes: it multiplied calibration error by 3.6×
Is the model’s “probability” a probability?	Not until you fix it — and the fix is two parameters
Does the textbook cost-optimal threshold work?	Only on calibrated scores. On raw scores it overspends by up to 56%
Does accuracy tell you which threshold to pick?	No. It moved 1.3 points while recall moved 48 points
Is a 7.35-point model upgrade a pure win?	No. It fixed 332 test documents and broke 149
Is the encoder uniformly better than tf-idf?	No — it is better on long documents, 1.88× against 1.18×
Are the mistakes that remain the model’s fault?	Often not. The most confident ones are label errors
Does a prompted LLM ever beat a fine-tuned encoder?	Yes — at 1,000 labels and below. At 2,000 they tie to 0.08 points
Does writing the prompt in English help?	Only if the model is weak: +16.90 points on 3B, +0.54 on 9B
Does a confidence stated in words mean anything?	On average yes, +0.58. As a detector, AUROC 0.667 against 0.843
Is the tie at 2,000 labels a real choice?	Not once you price it. The LLM costs 142× more per document

What we will look at. First the number every result in the chapter is measured against, which on this corpus is embarrassingly high before any model exists. Then the two ways of building a classifier — a sparse baseline that is harder to beat than it looks, and a fine-tuned encoder — and the metrics that tell them apart, since accuracy will not. Then the part most treatments skip: whether the number a model hands you is a probability, and how to turn it into a decision using what an error actually costs. Finally the comparison the chapter exists for, LLM against small model, and the two axes that settle it.

12.2 Baselines that are hard to beat

Before adding parameters, find out what the cheap thing does at your label budget rather than at the budget somebody else had. Chapter 4 reported one point on a curve. Here is the curve, same vectoriser, same classifier, same test set, training on stratified subsets of train:

labels   macro-F1   complaint-F1   features   fit time
   200      55.46          14.63        720       0.1s
   500      60.82          25.32      1,762       0.1s
 1,000      61.60          27.03      3,458       0.1s
 2,000      67.42          38.77      6,586       0.2s
 5,000      72.77          49.18     15,828       0.3s
10,000      73.44          50.87     29,149       0.4s
20,000      75.00          54.03     54,822       0.6s
40,000      76.21          56.47    101,659       1.2s

Three things in that table are worth more than the headline.

The curve is still climbing at 40,000. Going from 20,000 to 40,000 labels bought 1.21 macro-F1 points. Doubling again would presumably buy something similar. Nothing here has saturated, so “we used all our data” is a statement about the budget, not about the method.

Almost all of the gain is in the minority class. Macro-F1 rises 20.75 points across the whole sweep; complaint-F1 rises 41.84. The majority class was essentially solved at 200 labels and stayed solved. When a class is 93.7% of your data, the average metric is mostly reporting on the other 6.3% — which is exactly why Chapter 4 used macro-F1 and not accuracy, and why Section 12.4 returns to the point with more force.

The whole sweep costs under four seconds. Every fit in that column takes about a second or less on a laptop CPU, three seconds in total. Whatever else is true about the sparse baseline, you can afford to run it, and you should run it before you believe any number from anything more expensive.

The features column is a reminder of what tf-idf actually is. At 200 labels there are 720 surviving n-grams, because min_df=2 discards everything seen once. The model is not “small” by choice; it is small because the vocabulary it is allowed to see is small. That is also why it degrades so gracefully — there is nothing to overfit.

12.3 Fine-tuning encoder models

The competitor is BERTurk, the Turkish BERT released as dbmdz/bert-base-turkish-cased (Schweter 2020).

BERTurk is a 2020 checkpoint. We use it because it is the strongest widely-used Turkish-specific encoder — not because it is the frontier. The 2024–2026 encoder revival rebuilt the BERT-base recipe at a similar size (~150M parameters for ModernBERT-base) with rotary embeddings, unpadding and an 8,192-token context: ModernBERT (Warner et al. 2025) itself is trained on English and code, but the multilingual families that followed it (mmBERT, for example) cover Turkish and train markedly faster. Either of this chapter’s cost headlines would move on one of them. We did not re-run the sweep, which is the honest boundary of the 970× figure.

BERTurk was pretrained on roughly 35 GB and 4.4 billion tokens of Turkish — filtered OSCAR, a Wikipedia dump, OPUS corpora, and a corpus contributed by Kemal Oflazer — with a 32,000-token vocabulary, on a TPU v3-8.

The model card says 2M training steps; the repository’s cheat-sheet passes --num_train_steps=3000000. We could not resolve which is right, so the chapter does not depend on it. The model card also prints the token count as “44,04,976,662”, a digit-grouping typo for 4,404,976,662 — a small reminder that model cards are prose, not data.

Loading the checkpoint and counting gives 110,618,882 parameters: 24,972,288 in the embedding table, 85,054,464 in the twelve encoder layers, 590,592 in the pooler, and the remaining 1,538 in the two-class classification head (\(768 \times 2\) weights plus two biases) that replaces the pretraining head we throw away. Those four add up exactly. That embedding number is worth pausing on — 22.6% of the model is a lookup table, and it exists because the vocabulary is 32,000 tokens wide.

The sequence-length decision, made with a measurement

Attention is quadratic in sequence length (Chapter 7), so max_length is the single biggest lever on training cost. The usual advice is 512, which is the model’s limit. Here is what the corpus actually needs, in BERTurk subwords:

median    29 subwords
p95       86
p99      146
max      466

truncated at max_length=64    10.97% of documents
truncated at max_length=128    1.54%
truncated at max_length=192    0.36%
truncated at max_length=256    0.11%
truncated at max_length=512    0.00%

Those counts are over the 40,000-document training slice and include the [CLS] and [SEP] tokens, because that is what the GPU actually pads and truncates. The test slice behaves the same way — median 29, p95 87 — so the decision does not depend on which slice you measure.

We use 128. It truncates 1.54% of reviews and costs a sixteenth of the attention compute of 512. Picking 512 “to be safe” would have quadrupled the sequence length to protect fewer than two documents in a hundred, and those documents are long precisely because they ramble. The five-minute measurement that produced this table is the highest-return five minutes in the chapter.

The learning curve

Same budgets, same subsets, same test set. Class-weighted cross-entropy, AdamW at 2e-5, batch 32, max_length 128, linear warmup over the first 10% of steps. Budgets of 2,000 and below get 20 epochs, larger budgets get 3 — otherwise the small runs would see so few gradient updates that we would be measuring the budget schedule rather than the budget.

labels   macro-F1   complaint-F1   complaint-P   complaint-R   steps
   200      54.31          12.83         20.49          9.33     140
   500      56.46          16.37         40.12         10.29     320
 1,000      71.57          45.95         63.76         35.92     640
 2,000      79.61          61.63         66.48         57.44   1,260
 5,000      81.27          65.12         60.22         70.89     471
10,000      82.71          67.50         71.30         64.08     939
20,000      83.71          69.41         72.26         66.77   1,875
40,000      85.02          71.92         72.32         71.52   3,750

Two caveats before you read the numbers, and the first is about what is not here. Every point on this curve is a single training run at a single seed. At the small budgets that matters most: with a randomly initialised classification head and 500 examples, macro-F1 varies by seed on a scale of points, so the 15.1-point jump between 500 and 1,000 labels is a jump we have measured once. Three seeds at 500, 1,000 and 2,000 would put an error bar on the crossover and it is the measurement this section most needs. Note also what the bootstrap intervals later in this chapter measure. A bootstrap interval redraws the test documents with replacement many times, recomputes the metric on each redraw, and reports the middle 95% of the values. It resamples the test set, not the optimiser, so it describes evaluation noise and says nothing about training noise.

The second caveat is one the steps column makes visible. The epoch schedule changes at 2,000, so the 2,000-label model gets 1,260 updates and the 5,000-label model only 471 — nearly seven times more gradient updates per example. That is the most likely reason complaint precision falls from 66.48 to 60.22 across that one step, and why the 5,000-label model scores below the 2,000-label one on the shared subsample in Section 12.8 (81.21 against 82.00). A learning curve that dips as the budget grows is not normal, and a schedule change is the first thing to suspect. Both of this chapter’s headline crossovers sit at 2,000, the largest budget on the 20-epoch schedule. We did not re-run 5,000 at 20 epochs to separate the two, so read every crossover below as at this schedule.

Three readings, and the third is the one that changes how you plan a project.

The encoder loses at the bottom of the curve. At 200 labels BERTurk scores 54.31 against tf-idf’s 55.46; at 500 it scores 56.46 against 60.82. A 110-million-parameter model pretrained on 4.4 billion tokens of Turkish is worse than a bag of words, on Turkish, with 500 examples. Nothing is wrong: the classification head is randomly initialised and 500 examples containing 32 complaints is not enough signal to move a network that size into a useful region without overfitting first.

The crossover is between 500 and 1,000 labels, and it is abrupt. Across that one doubling BERTurk gains 15.1 macro-F1 points and tf-idf gains 0.8. Complaint recall goes from 10.29% to 35.92%. Somewhere in there the model stops predicting the majority class with occasional accidents and starts having a representation of what a complaint is.

BERTurk at 2,000 labels beats tf-idf at 40,000. 79.61 against 76.21 — twenty times fewer labels for a better model. If labelling is your bottleneck, and it usually is, that ratio is the number to take to a planning meeting. At equal budget the gap is 8.81 macro-F1 points and 15.45 points of complaint-F1, which is the number a person actually experiences.

bertRows = [
  {n: 200,   bert: 54.31, bert_negf1: 12.83, tfidf: 55.46, tfidf_negf1: 14.63, steps: 140},
  {n: 500,   bert: 56.46, bert_negf1: 16.37, tfidf: 60.82, tfidf_negf1: 25.32, steps: 320},
  {n: 1000,  bert: 71.57, bert_negf1: 45.95, tfidf: 61.60, tfidf_negf1: 27.03, steps: 640},
  {n: 2000,  bert: 79.61, bert_negf1: 61.63, tfidf: 67.42, tfidf_negf1: 38.77, steps: 1260},
  {n: 5000,  bert: 81.27, bert_negf1: 65.12, tfidf: 72.77, tfidf_negf1: 49.18, steps: 471},
  {n: 10000, bert: 82.71, bert_negf1: 67.50, tfidf: 73.44, tfidf_negf1: 50.87, steps: 939},
  {n: 20000, bert: 83.71, bert_negf1: 69.41, tfidf: 75.00, tfidf_negf1: 54.03, steps: 1875},
  {n: 40000, bert: 85.02, bert_negf1: 71.92, tfidf: 76.21, tfidf_negf1: 56.47, steps: 3750}
]

viewof curveMetric = Inputs.radio(
  new Map([
    ["Macro-F1 — the average over both classes", "macro"],
    ["Complaint F1 — the minority class on its own", "negf1"]
  ]),
  { value: "macro", label: "" }
)

curvePlot = {
  const suffix = curveMetric === "macro" ? "" : "_negf1";
  const long = bertRows.flatMap(d => [
    {n: d.n, model: "tf-idf + LR", y: curveMetric === "macro" ? d.tfidf : d.tfidf_negf1},
    {n: d.n, model: "BERTurk fine-tuned", y: curveMetric === "macro" ? d.bert : d.bert_negf1}
  ]);
  return Plot.plot({
    width: 700,
    height: 360,
    marginLeft: 60,
    marginBottom: 48,
    x: { label: "labelled training documents", type: "log",
         ticks: [200, 500, 1000, 2000, 5000, 10000, 20000, 40000],
         tickFormat: d => d.toLocaleString(), grid: true },
    y: { label: curveMetric === "macro" ? "macro-F1 on the 10,000-document test set"
                                        : "complaint F1 on the 10,000-document test set",
         domain: [0, 90], grid: true },
    color: { legend: true, domain: ["tf-idf + LR", "BERTurk fine-tuned"],
             range: ["#1F4E79", "#B3453E"] },
    marks: [
      curveMetric === "macro"
        ? Plot.ruleY([48.37], { stroke: "currentColor", strokeDasharray: "4 3" })
        : null,
      curveMetric === "macro"
        ? Plot.text([{n: 300, y: 48.37}], { x: "n", y: "y", text: ["always “satisfied”"],
            dy: -9, fill: "currentColor", textAnchor: "start" })
        : null,
      Plot.ruleX([700], { stroke: "#2E7D6F", strokeDasharray: "3 3", strokeOpacity: 0.6 }),
      Plot.line(long, { x: "n", y: "y", stroke: "model", strokeWidth: 2.5 }),
      Plot.dot(long, { x: "n", y: "y", fill: "model", r: 4,
        title: d => `${d.model}\n${d.n.toLocaleString()} labels\n${d.y.toFixed(2)}` })
    ].filter(d => d)
  });
}

The teal rule marks the crossover region. Switch to complaint F1 and notice that the two curves cross in the same place but the gap afterwards is roughly twice as wide: at 40,000 labels macro-F1 differs by 8.81 points and complaint-F1 by 15.45. Averaging over a 93.7%-majority class hides most of what the encoder bought you.

What the sweep cost, and which of those numbers we trust

Training cost is measured in optimiser steps, which is reproducible, and in seconds, which is not — and this run is a good illustration of why the distinction matters.

labels   steps   wall-clock   s/step
 1,000     640       525.5s    0.821   ← first GPU run after a cold start
 2,000   1,260       387.0s    0.307
 5,000     471       147.5s    0.313
10,000     939       289.5s    0.308
20,000   1,875       581.5s    0.310
40,000   3,750     6,303.1s    1.681   ← machine slowed mid-run

Four consecutive runs agree on 0.307 to 0.313 seconds per step at batch 32 and max_length 128. That is a real measurement: four independent runs spanning a tenfold range of dataset size (2,000 to 20,000 labels), agreeing within 2%. The other two are not. The 1,000-label run paid a one-off Metal kernel-compilation cost, and the 40,000-label run took 386 seconds for its first epoch, 386 for its second, and 5,531 for its third — a 14× slowdown with no change in the work being done.

So the honest statement is: at 0.31 s/step the whole eight-point sweep is 9,395 steps, about 49 minutes of GPU time, and the 40,000-label model on its own is about 19 minutes. The 6,303 seconds we actually observed is a fact about the laptop that evening, not about the model, and quoting it as a training cost would mislead every reader who tried to reproduce it.

Inference is much better behaved: 499, 434, 462, 472, 466, 475 and 510 documents per second across seven runs, on the same 10,000 documents each time. Call it 470 documents per second, single process, batched, on a laptop GPU. Only the 200-label run disagrees, at 17.5 — that was the first inference pass of the whole session, and it was paying for kernel compilation on behalf of everything after it. When one measurement in a series is 27× off, suspect your harness before your model.

Compare against the sparse baseline, whose whole eight-point sweep fits in three seconds of laptop CPU. At the top budget that is 3,750 steps × 0.31 s — 1,162 seconds of corrected GPU time, not the 6,303 the contaminated run recorded — against 1.2 seconds for the vectoriser. The encoder is roughly 970× more expensive to train for 8.81 macro-F1 points, and serves about 470 documents per second against tens of thousands. Whether that is a good trade is not a machine-learning question; Section 12.9 is where we try to answer it with money.

12.4 Metrics beyond accuracy

Every number so far has been macro-F1, on the assertion — inherited from Chapter 4 — that accuracy is useless on this corpus. It is time to show that rather than assert it.

Take one model, the uncorrected tf-idf baseline, and move nothing except the threshold at which it declares a review a complaint. “Uncorrected” means no class weights, and this copy is fitted on the 32,000-document fit slice so that cal stays clean — so it is not the class-weighted 76.21 model above. Section 12.5 uses the same fitted model. The table below is read on test, and it is illustration rather than selection: it exists to show what the metrics do across the range, not to pick an operating point. The operating point actually used in Section 12.5 was chosen on cal and lands at 0.166 — between the 0.15 and 0.20 grid points, close to the best row here (0.15). Its regret is 77.66 against the oracle’s 78.00 on test, 0.34 macro-F1 points: roughly what doing it properly costs.

   t      acc   bal-acc    prec     rec   compF1   macroF1     MCC
0.05    68.98     81.53   16.46   95.89    28.09     54.16   0.318
0.10    89.21     85.76   34.91   81.80    48.94     71.45   0.490
0.15    94.23     81.06   53.53   65.98    59.11     78.00   0.564
0.20    95.00     73.28   63.75   48.42    55.04     76.19   0.530
0.25    95.16     68.57   72.16   38.13    49.90     73.68   0.503
0.30    95.00     64.57   77.05   29.75    42.92     70.15   0.460
0.40    94.59     58.90   83.21   18.04    29.65     63.42   0.372
0.50    94.28     55.41   88.46   10.92    19.44     58.24   0.299
0.60    94.07     53.31   93.33    6.65    12.41     54.67   0.240
0.70    93.94     52.13   96.43    4.27     8.18     52.52   0.196
0.80    93.77     50.71  100.00    1.42     2.81     49.79   0.116
0.95    93.70     50.16  100.00    0.32     0.63     48.69   0.054

Read the accuracy column top to bottom over the range \(t \ge 0.2\). It moves from 95.00 to 93.70 — 1.3 points, and never spans more than 1.5. Over the same range, recall on the class somebody has to act on falls from 48.42% to 0.32%, and complaint-F1 falls from 55.04 to 0.63. A metric that changes by one point while the system goes from catching half the complaints to catching none of them is not measuring the system.

Worse than useless, it is actively misleading. The highest accuracy in the table is 95.16 at \(t = 0.25\). The best model in the table is at \(t = 0.15\) — an oracle, by the rule in Section 12.1, since we are reading the maximum off the test set — where macro-F1 is 78.00 and accuracy is 0.93 points lower. Tuning on accuracy picks the wrong operating point, and it does so confidently.

The widget below plots the whole table. Accuracy is drawn in grey behind every selection, so whatever you switch to, you are looking at it against the metric that does not move.

threshRows = [
  {t: 0.05, acc: 68.98, bal: 81.53, prec: 16.46, rec: 95.89, cf1: 28.09, mf1: 54.16, mcc: 0.318, tp: 606, fp: 3076, fn: 26},
  {t: 0.10, acc: 89.21, bal: 85.76, prec: 34.91, rec: 81.80, cf1: 48.94, mf1: 71.45, mcc: 0.490, tp: 517, fp: 964, fn: 115},
  {t: 0.15, acc: 94.23, bal: 81.06, prec: 53.53, rec: 65.98, cf1: 59.11, mf1: 78.00, mcc: 0.564, tp: 417, fp: 362, fn: 215},
  {t: 0.20, acc: 95.00, bal: 73.28, prec: 63.75, rec: 48.42, cf1: 55.04, mf1: 76.19, mcc: 0.530, tp: 306, fp: 174, fn: 326},
  {t: 0.25, acc: 95.16, bal: 68.57, prec: 72.16, rec: 38.13, cf1: 49.90, mf1: 73.68, mcc: 0.503, tp: 241, fp: 93, fn: 391},
  {t: 0.30, acc: 95.00, bal: 64.57, prec: 77.05, rec: 29.75, cf1: 42.92, mf1: 70.15, mcc: 0.460, tp: 188, fp: 56, fn: 444},
  {t: 0.35, acc: 94.79, bal: 61.36, prec: 80.66, rec: 23.10, cf1: 35.92, mf1: 66.60, mcc: 0.415, tp: 146, fp: 35, fn: 486},
  {t: 0.40, acc: 94.59, bal: 58.90, prec: 83.21, rec: 18.04, cf1: 29.65, mf1: 63.42, mcc: 0.372, tp: 114, fp: 23, fn: 518},
  {t: 0.45, acc: 94.36, bal: 56.71, prec: 82.69, rec: 13.61, cf1: 23.37, mf1: 60.22, mcc: 0.322, tp: 86, fp: 18, fn: 546},
  {t: 0.50, acc: 94.28, bal: 55.41, prec: 88.46, rec: 10.92, cf1: 19.44, mf1: 58.24, mcc: 0.299, tp: 69, fp: 9, fn: 563},
  {t: 0.55, acc: 94.18, bal: 54.32, prec: 91.67, rec: 8.70, cf1: 15.90, mf1: 56.44, mcc: 0.273, tp: 55, fp: 5, fn: 577},
  {t: 0.60, acc: 94.07, bal: 53.31, prec: 93.33, rec: 6.65, cf1: 12.41, mf1: 54.67, mcc: 0.240, tp: 42, fp: 3, fn: 590},
  {t: 0.65, acc: 93.99, bal: 52.53, prec: 96.97, rec: 5.06, cf1: 9.62, mf1: 53.26, mcc: 0.214, tp: 32, fp: 1, fn: 600},
  {t: 0.70, acc: 93.94, bal: 52.13, prec: 96.43, rec: 4.27, cf1: 8.18, mf1: 52.52, mcc: 0.196, tp: 27, fp: 1, fn: 605},
  {t: 0.75, acc: 93.85, bal: 51.42, prec: 94.74, rec: 2.85, cf1: 5.53, mf1: 51.18, mcc: 0.159, tp: 18, fp: 1, fn: 614},
  {t: 0.80, acc: 93.77, bal: 50.71, prec: 100.00, rec: 1.42, cf1: 2.81, mf1: 49.79, mcc: 0.116, tp: 9, fp: 0, fn: 623},
  {t: 0.85, acc: 93.77, bal: 50.71, prec: 100.00, rec: 1.42, cf1: 2.81, mf1: 49.79, mcc: 0.116, tp: 9, fp: 0, fn: 623},
  {t: 0.90, acc: 93.75, bal: 50.55, prec: 100.00, rec: 1.11, cf1: 2.19, mf1: 49.48, mcc: 0.102, tp: 7, fp: 0, fn: 625},
  {t: 0.95, acc: 93.70, bal: 50.16, prec: 100.00, rec: 0.32, cf1: 0.63, mf1: 48.69, mcc: 0.054, tp: 2, fp: 0, fn: 630}
]

viewof threshMetric = Inputs.radio(
  new Map([
    ["Recall on complaints — of the 632, how many did we find?", "rec"],
    ["Precision on complaints — of what we flagged, how much was right?", "prec"],
    ["Complaint F1 — the harmonic mean of those two", "cf1"],
    ["Macro-F1 — the metric this chapter optimises", "mf1"],
    ["Balanced accuracy — accuracy with the classes weighted equally", "bal"],
    ["Matthews correlation × 100 — the chance-corrected summary", "mcc100"]
  ]),
  { value: "rec", label: "" }
)

threshLabels = ({
  rec: "complaint recall (%)", prec: "complaint precision (%)",
  cf1: "complaint F1", mf1: "macro-F1",
  bal: "balanced accuracy (%)", mcc100: "MCC × 100"
})

threshPlot = {
  const pick = d => threshMetric === "mcc100" ? d.mcc * 100 : d[threshMetric];
  return Plot.plot({
    width: 700,
    height: 340,
    marginLeft: 56,
    marginBottom: 46,
    x: { label: "decision threshold t — flag the review when the score exceeds this", grid: true },
    y: { label: threshLabels[threshMetric], domain: [0, 100], grid: true },
    marks: [
      Plot.line(threshRows, { x: "t", y: "acc", stroke: "currentColor", strokeWidth: 2, strokeDasharray: "5 3" }),
      Plot.dot(threshRows, { x: "t", y: "acc", fill: "currentColor", r: 2.5,
        title: d => `t = ${d.t}\naccuracy: ${d.acc}%` }),
      Plot.text([threshRows[13]], { x: "t", y: "acc", text: ["accuracy"],
        dy: -12, fill: "currentColor", fontWeight: 600 }),
      threshMetric === "mf1"
        ? Plot.ruleY([48.37], { stroke: "#B3453E", strokeDasharray: "3 3", strokeOpacity: 0.5 })
        : null,
      Plot.line(threshRows, { x: "t", y: pick, stroke: "#1F4E79", strokeWidth: 2.5 }),
      Plot.dot(threshRows, { x: "t", y: pick, fill: "#1F4E79", r: 3.5,
        title: d => `t = ${d.t}\n${threshLabels[threshMetric]}: ${pick(d).toFixed(2)}\n` +
                    `${d.tp} caught, ${d.fp} false alarms, ${d.fn} missed` })
    ].filter(d => d)
  });
}

The faint red rule at 48.37, drawn only when macro-F1 is selected, is the macro-F1 of a model that always answers “satisfied”. Switch the radio to macro-F1 and watch the blue line dive towards it as the threshold rises: past \(t = 0.8\) the model has effectively stopped being a classifier, and accuracy is still 93.77.

The reason is arithmetic, not philosophy:

call everything satisfied      acc 93.68   macroF1 48.37   bal-acc 50.00   MCC  0.000
guess at the base rate         acc 87.98   macroF1 49.76   bal-acc 49.76   MCC -0.005
call everything a complaint    acc  6.32   macroF1  5.94   bal-acc 50.00   MCC  0.000
coin flip                      acc 50.21   macroF1 38.57   bal-acc 51.44   MCC  0.014

A model that has never seen a feature scores 93.68% accuracy — the base rate. Every accuracy figure in this chapter has to be read against that floor, and most of them are within two points of it. Matthews correlation (MCC), by contrast, gives every no-skill row in that table zero or within 0.02 of it — exactly 0.000 for the two constant predictors — which is what you want from a summary statistic. The chapter leans on it more than once, so here is what it is: the Pearson correlation between the predicted and the true label, treated as two 0/1 variables,

\[ \mathrm{MCC} = \frac{TP \cdot TN - FP \cdot FN} {\sqrt{(TP{+}FP)(TP{+}FN)(TN{+}FP)(TN{+}FN)}} \]

running from \(-1\) to \(+1\). Be precise about the zero, because it is the whole reason to reach for this metric. For a constant predictor the denominator is zero — a model that always answers positive has \(TN = FN = 0\) — so MCC is \(0/0\), undefined, and scikit-learn returns 0 by convention. For a predictor that guesses at random, independently of the input, the expected MCC is 0 and the observed value fluctuates around it with the sample size. Either way the answer is “about nothing”, which is precisely what accuracy fails to say here.

One metric is missing from that table on purpose, and it is the one you will meet first. classification_report prints weighted-F1 by default: the per-class F1 scores averaged by support. On this corpus the support is 93.68% one class, so the always-satisfied model’s weighted-F1 is \(0.9368 \times 96.74 = 90.62\) — higher than any macro-F1 anywhere in this chapter, for a model with no features. Weighted-F1 is accuracy wearing an F1 costume. Micro-F1 — pool the \(TP\), \(FP\) and \(FN\) counts of all classes, then compute one F1 — is worse still: when every document gets exactly one label it equals accuracy, 93.68 here. If you report one number on imbalanced data, report macro-F1 or MCC; if you report weighted-F1, print the majority-class rate next to it so the reader can do the subtraction.

base rate — how much of the corpus belongs to a class. It doubles as the accuracy of a model that always answers with the majority, which makes it the first number to compute and the last one anybody reports. ↩︎

ROC and precision-recall are not interchangeable

Both curves are computed from exactly the same confusion matrices. They disagree because they divide by different things: the false-positive rate divides by the 9,368 satisfied reviews, while precision divides by however many documents you flagged (Davis and Goadrich 2006).

confusion matrix — the four counts a binary classifier produces: true positives, false positives, false negatives, true negatives. Every metric in this chapter is some arithmetic on those four numbers, which is why two metrics can disagree without either being wrong. ↩︎

model                          AP      AUC   AP/base   bal-acc     MCC
tf-idf, uncorrected         60.30    93.24     9.54x     55.41   0.299
tf-idf, class-weighted      60.81    93.44     9.62x     84.34   0.545
tf-idf, SMOTE               58.05    92.81     9.19x     82.03   0.514
BERTurk n=200               12.78    70.12     2.02x     53.45   0.100
BERTurk n=500               26.98    83.00     4.27x     54.62   0.178
BERTurk n=1,000             45.72    82.76     7.23x     67.27   0.454
BERTurk n=2,000             62.34    92.51     9.86x     77.74   0.594
BERTurk n=5,000             66.37    93.98    10.50x     83.86   0.628

AUC, the area under the ROC curve, has a plain reading worth keeping: it is the probability that a randomly chosen complaint gets a higher score than a randomly chosen satisfied review, with ties counted as half. It looks only at the order of the scores, never at a threshold.

An AUC of 93.24 sounds like a solved problem. The average precision for the same model and the same scores is 60.30. Both are correct. The AUC is high because the model is very good at the easy majority of the ranking, and the false positive rate has 9,368 documents in its denominator to hide mistakes in.

average precision — the area under the precision-recall curve, written AP. Unlike AUC it moves with the class balance, so an AP of 60.30 means nothing until you know that guessing at random would score 6.32. ↩︎

The clearest evidence that these are different measurements is BERTurk between 500 and 1,000 labels: AUC goes down (83.00 → 82.76) while average precision goes up by 18.7 points (26.98 → 45.72) and macro-F1 goes up by 15.1. If you had been watching AUC you would have concluded the extra 500 labels made the model worse.

The widget below draws exactly those two disagreeing views. Switch between ROC space and PR space with the same models selected and watch the ordering of the curves change.

curveRows = {
  // [model, curve space, x, y] — thinned from the full curves for the page
  const raw = [
    ["tfidf","roc",0.000,0.000], ["tfidf","roc",0.001,0.122], ["tfidf","roc",0.003,0.185], ["tfidf","roc",0.005,0.263],
    ["tfidf","roc",0.006,0.320], ["tfidf","roc",0.009,0.359], ["tfidf","roc",0.011,0.391], ["tfidf","roc",0.014,0.415],
    ["tfidf","roc",0.016,0.457], ["tfidf","roc",0.019,0.495], ["tfidf","roc",0.023,0.524], ["tfidf","roc",0.026,0.552],
    ["tfidf","roc",0.029,0.587], ["tfidf","roc",0.033,0.619], ["tfidf","roc",0.037,0.649], ["tfidf","roc",0.042,0.679],
    ["tfidf","roc",0.047,0.699], ["tfidf","roc",0.051,0.725], ["tfidf","roc",0.061,0.745], ["tfidf","roc",0.072,0.766],
    ["tfidf","roc",0.081,0.786], ["tfidf","roc",0.096,0.810], ["tfidf","roc",0.118,0.832], ["tfidf","roc",0.141,0.850],
    ["tfidf","roc",0.164,0.872], ["tfidf","roc",0.197,0.891], ["tfidf","roc",0.220,0.908], ["tfidf","roc",0.275,0.929],
    ["tfidf","roc",0.306,0.946], ["tfidf","roc",0.387,0.970], ["tfidf","roc",0.561,0.987], ["tfidf","roc",1.000,1.000],
    ["tfidf","pr",1.000,0.063], ["tfidf","pr",1.000,0.065], ["tfidf","pr",1.000,0.067], ["tfidf","pr",0.997,0.070],
    ["tfidf","pr",0.995,0.072], ["tfidf","pr",0.995,0.075], ["tfidf","pr",0.994,0.078], ["tfidf","pr",0.992,0.081],
    ["tfidf","pr",0.992,0.084], ["tfidf","pr",0.991,0.088], ["tfidf","pr",0.991,0.092], ["tfidf","pr",0.991,0.097],
    ["tfidf","pr",0.991,0.102], ["tfidf","pr",0.986,0.107], ["tfidf","pr",0.984,0.112], ["tfidf","pr",0.981,0.120],
    ["tfidf","pr",0.978,0.127], ["tfidf","pr",0.975,0.136], ["tfidf","pr",0.968,0.145], ["tfidf","pr",0.964,0.155],
    ["tfidf","pr",0.954,0.169], ["tfidf","pr",0.940,0.182], ["tfidf","pr",0.922,0.200], ["tfidf","pr",0.903,0.218],
    ["tfidf","pr",0.883,0.246], ["tfidf","pr",0.861,0.277], ["tfidf","pr",0.835,0.317], ["tfidf","pr",0.797,0.386],
    ["tfidf","pr",0.744,0.468], ["tfidf","pr",0.590,0.571], ["tfidf","pr",0.394,0.707], ["tfidf","pr",0.000,1.000],
    ["b500","roc",0.000,0.000], ["b500","roc",0.012,0.122], ["b500","roc",0.026,0.233], ["b500","roc",0.050,0.342],
    ["b500","roc",0.072,0.427], ["b500","roc",0.100,0.513], ["b500","roc",0.129,0.570], ["b500","roc",0.154,0.623],
    ["b500","roc",0.189,0.682], ["b500","roc",0.210,0.720], ["b500","roc",0.244,0.747], ["b500","roc",0.268,0.769],
    ["b500","roc",0.295,0.804], ["b500","roc",0.319,0.826], ["b500","roc",0.340,0.847], ["b500","roc",0.368,0.854],
    ["b500","roc",0.391,0.869], ["b500","roc",0.420,0.883], ["b500","roc",0.445,0.899], ["b500","roc",0.470,0.910],
    ["b500","roc",0.501,0.921], ["b500","roc",0.530,0.932], ["b500","roc",0.565,0.943], ["b500","roc",0.602,0.956],
    ["b500","roc",0.648,0.965], ["b500","roc",0.693,0.972], ["b500","roc",0.746,0.978], ["b500","roc",0.812,0.986],
    ["b500","roc",0.865,0.994], ["b500","roc",0.929,0.997], ["b500","roc",0.964,0.997], ["b500","roc",1.000,1.000],
    ["b500","pr",1.000,0.063], ["b500","pr",0.997,0.065], ["b500","pr",0.995,0.069], ["b500","pr",0.986,0.077],
    ["b500","pr",0.975,0.084], ["b500","pr",0.960,0.093], ["b500","pr",0.945,0.101], ["b500","pr",0.929,0.107],
    ["b500","pr",0.911,0.114], ["b500","pr",0.899,0.121], ["b500","pr",0.877,0.128], ["b500","pr",0.861,0.133],
    ["b500","pr",0.850,0.142], ["b500","pr",0.828,0.148], ["b500","pr",0.807,0.155], ["b500","pr",0.774,0.161],
    ["b500","pr",0.753,0.169], ["b500","pr",0.731,0.179], ["b500","pr",0.717,0.189], ["b500","pr",0.687,0.196],
    ["b500","pr",0.653,0.206], ["b500","pr",0.622,0.215], ["b500","pr",0.581,0.228], ["b500","pr",0.533,0.235],
    ["b500","pr",0.506,0.260], ["b500","pr",0.464,0.276], ["b500","pr",0.413,0.292], ["b500","pr",0.351,0.316],
    ["b500","pr",0.280,0.328], ["b500","pr",0.214,0.385], ["b500","pr",0.125,0.418], ["b500","pr",0.000,1.000],
    ["b1000","roc",0.000,0.000], ["b1000","roc",0.005,0.174], ["b1000","roc",0.010,0.299], ["b1000","roc",0.021,0.427],
    ["b1000","roc",0.033,0.519], ["b1000","roc",0.058,0.597], ["b1000","roc",0.083,0.639], ["b1000","roc",0.110,0.669],
    ["b1000","roc",0.138,0.682], ["b1000","roc",0.157,0.691], ["b1000","roc",0.180,0.699], ["b1000","roc",0.198,0.714],
    ["b1000","roc",0.223,0.722], ["b1000","roc",0.245,0.728], ["b1000","roc",0.267,0.737], ["b1000","roc",0.294,0.745],
    ["b1000","roc",0.317,0.747], ["b1000","roc",0.341,0.752], ["b1000","roc",0.362,0.755], ["b1000","roc",0.382,0.763],
    ["b1000","roc",0.403,0.774], ["b1000","roc",0.422,0.785], ["b1000","roc",0.448,0.815], ["b1000","roc",0.467,0.837],
    ["b1000","roc",0.487,0.858], ["b1000","roc",0.505,0.878], ["b1000","roc",0.529,0.889], ["b1000","roc",0.569,0.903],
    ["b1000","roc",0.627,0.921], ["b1000","roc",0.732,0.960], ["b1000","roc",0.892,0.986], ["b1000","roc",1.000,1.000],
    ["b1000","pr",1.000,0.063], ["b1000","pr",0.976,0.073], ["b1000","pr",0.927,0.090], ["b1000","pr",0.891,0.100],
    ["b1000","pr",0.872,0.105], ["b1000","pr",0.848,0.108], ["b1000","pr",0.824,0.109], ["b1000","pr",0.802,0.110],
    ["b1000","pr",0.783,0.112], ["b1000","pr",0.771,0.115], ["b1000","pr",0.761,0.120], ["b1000","pr",0.753,0.126],
    ["b1000","pr",0.748,0.136], ["b1000","pr",0.744,0.147], ["b1000","pr",0.736,0.158], ["b1000","pr",0.725,0.174],
    ["b1000","pr",0.720,0.189], ["b1000","pr",0.699,0.205], ["b1000","pr",0.695,0.224], ["b1000","pr",0.687,0.242],
    ["b1000","pr",0.677,0.268], ["b1000","pr",0.665,0.293], ["b1000","pr",0.647,0.324], ["b1000","pr",0.631,0.354],
    ["b1000","pr",0.606,0.395], ["b1000","pr",0.585,0.442], ["b1000","pr",0.547,0.489], ["b1000","pr",0.479,0.543],
    ["b1000","pr",0.408,0.600], ["b1000","pr",0.288,0.650], ["b1000","pr",0.174,0.728], ["b1000","pr",0.000,1.000],
    ["b5000","roc",0.000,0.000], ["b5000","roc",0.004,0.274], ["b5000","roc",0.007,0.384], ["b5000","roc",0.012,0.495],
    ["b5000","roc",0.017,0.581], ["b5000","roc",0.026,0.660], ["b5000","roc",0.034,0.718], ["b5000","roc",0.056,0.766],
    ["b5000","roc",0.076,0.820], ["b5000","roc",0.122,0.866], ["b5000","roc",0.192,0.910], ["b5000","roc",0.253,0.937],
    ["b5000","roc",0.331,0.956], ["b5000","roc",0.388,0.965], ["b5000","roc",0.431,0.970], ["b5000","roc",0.478,0.972],
    ["b5000","roc",0.518,0.973], ["b5000","roc",0.551,0.978], ["b5000","roc",0.584,0.981], ["b5000","roc",0.617,0.983],
    ["b5000","roc",0.646,0.984], ["b5000","roc",0.679,0.984], ["b5000","roc",0.718,0.986], ["b5000","roc",0.743,0.987],
    ["b5000","roc",0.773,0.991], ["b5000","roc",0.802,0.995], ["b5000","roc",0.827,0.995], ["b5000","roc",0.851,0.997],
    ["b5000","roc",0.881,0.998], ["b5000","roc",0.916,1.000], ["b5000","roc",0.942,1.000], ["b5000","roc",1.000,1.000],
    ["b5000","pr",1.000,0.063], ["b5000","pr",1.000,0.065], ["b5000","pr",1.000,0.067], ["b5000","pr",0.998,0.070],
    ["b5000","pr",0.997,0.072], ["b5000","pr",0.995,0.076], ["b5000","pr",0.992,0.078], ["b5000","pr",0.991,0.082],
    ["b5000","pr",0.986,0.085], ["b5000","pr",0.984,0.089], ["b5000","pr",0.984,0.094], ["b5000","pr",0.981,0.098],
    ["b5000","pr",0.981,0.104], ["b5000","pr",0.976,0.110], ["b5000","pr",0.972,0.115], ["b5000","pr",0.972,0.124],
    ["b5000","pr",0.970,0.132], ["b5000","pr",0.965,0.142], ["b5000","pr",0.960,0.151], ["b5000","pr",0.956,0.163],
    ["b5000","pr",0.951,0.178], ["b5000","pr",0.940,0.193], ["b5000","pr",0.927,0.213], ["b5000","pr",0.915,0.235],
    ["b5000","pr",0.900,0.268], ["b5000","pr",0.880,0.302], ["b5000","pr",0.853,0.346], ["b5000","pr",0.818,0.421],
    ["b5000","pr",0.758,0.507], ["b5000","pr",0.644,0.663], ["b5000","pr",0.413,0.789], ["b5000","pr",0.000,1.000]
  ];
  return raw.map(([model, space, x, y]) => ({model, space, x, y}));
}

viewof rocSpace = Inputs.radio(
  new Map([
    ["ROC space — recall against false-positive rate", "roc"],
    ["Precision-recall space — precision against recall", "pr"]
  ]),
  { value: "roc", label: "" }
)

rocNames = ({
  tfidf: "tf-idf + LR, uncorrected, 32,000 labels", b500: "BERTurk, 500 labels",
  b1000: "BERTurk, 1,000 labels", b5000: "BERTurk, 5,000 labels"
})

rocPlot = {
  const data = curveRows.filter(d => d.space === rocSpace);
  const isRoc = rocSpace === "roc";
  return Plot.plot({
    width: 640,
    height: 440,
    marginLeft: 60,
    marginBottom: 48,
    x: { label: isRoc ? "false-positive rate — of 9,368 satisfied reviews, the share flagged"
                      : "recall — of 632 complaints, the share caught",
         domain: [0, 1], grid: true },
    y: { label: isRoc ? "true-positive rate — of 632 complaints, the share caught"
                      : "precision — of what was flagged, the share that was right",
         domain: [0, 1], grid: true },
    color: { legend: true, domain: ["tfidf", "b500", "b1000", "b5000"],
             range: ["#1F4E79", "#C08A3E", "#B3453E", "#0E7C7B"],
             tickFormat: k => rocNames[k] },
    marks: [
      isRoc
        ? Plot.line([[0, 0], [1, 1]], { stroke: "currentColor", strokeDasharray: "5 4" })
        : Plot.ruleY([0.0632], { stroke: "currentColor", strokeDasharray: "5 4" }),
      Plot.line(data, { x: "x", y: "y", stroke: "model", strokeWidth: 2 })
    ]
  });
}

The dashed line means something different in each space. In ROC space it is the diagonal a coin flip traces. In PR space it is the horizontal line at 0.0632 — the base rate, which is what precision equals when you flag documents at random. Notice how much of the ROC plot sits far above its baseline and how little of the PR plot does. Same models, same scores.

The AP/base column is the honest way to read average precision: 60.30 against a 6.32% base rate is 9.54× better than random, which is a real achievement that “60” does not convey on its own.

12.5 Class imbalance

The corpus is 93.68% satisfied. The standard menu of responses to that class imbalance is: do nothing, weight the classes, oversample the minority, undersample the majority, or synthesise new minority points with SMOTE (Chawla et al. 2002), which creates each synthetic point on the straight line between a minority example and one of its nearest minority neighbours. There is a sixth option that is usually listed last and treated as a hack, and it wins.

class imbalance — one label dominates the corpus, so accuracy stops carrying information and the loss function stops caring about the class you built the system for. It is a property of the data, not a bug in it. ↩︎

Everything below is fitted on the 32,000-document fit slice, with any threshold chosen on the held-out cal slice, and scored on test.

treatment                     acc   macroF1   compF1   comp-P   comp-R      AP     AUC
no correction, t=0.5        94.28     58.24    19.44     88.5     10.9   60.30   93.24
threshold moved to 0.166    94.64     77.66    58.19     57.4     59.0   60.30   93.24
class_weight=balanced       92.63     76.09    56.21     45.0     74.8   60.81   93.44
random oversampling         93.63     77.63    58.72     49.7     71.7   61.46   93.45
random undersampling        81.08     62.98    37.10     23.5     88.3   52.49   92.15
SMOTE                       92.32     74.72    53.62     43.4     70.3   58.05   92.81

Four treatments change the ranking. One changes the decision.

The axis starts at 45, not 0, because a model with no features scores 48.37 and everything interesting happens above it. The AP column is the control: average precision depends only on the ranking, so the same fitted model read at two thresholds must show the same AP — which rows 1 and 2 do, to four significant figures.

Row two is the whole section. The same model as row one — not retrained, not reweighted, not resampled, the identical fitted coefficients — goes from 58.24 to 77.66 macro-F1 because we flagged a review when \(P(\text{complaint})\) passed 0.166 instead of 0.5.

The proof that nothing about the model changed is in the last two columns. Average precision and AUC are identical to four significant figures in rows one and two, because both are computed from the ranking, and moving a threshold cannot reorder anything. The four retraining treatments — class weights, oversampling, undersampling, SMOTE — are attempts to change a ranking in order to fix a decision. Moving the threshold fixes the decision directly.

One thing the table does not do fairly, and you should hold it against the headline. Only one of the six rows was allowed to choose its threshold; the other five are read at 0.5. That is a tuned arm racing untuned ones. The chapter’s own argument says how it would come out otherwise: average precision depends only on the ranking, and the ranking is oversampling 61.46, class weights 60.81, uncorrected 60.30 — so if every arm were given its own threshold on cal, oversampling would probably end up marginally ahead, not behind. We did not run those three rows. Read this section as a result about price, not about ranking: moving one threshold buys the same operating point the best resampling method buys, at zero training cost — it does not beat it.

Against that, the retraining methods look expensive at the price they charge:

• Class weights reach 76.09, 1.57 points below the free option, and cost a refit.
• Random oversampling reaches 77.63, within three hundredths of a point of moving the threshold, on 59,954 rows instead of 32,000 — nearly double the training time for what looks like a tie. We did not test that gap formally, and a 0.03-point difference is far below what a test set with 632 complaints can resolve, so “indistinguishable” is all the table entitles us to say.
• Random undersampling is a disaster: 62.98. It threw away 27,954 of 32,000 documents to achieve parity, and the majority class it discarded was carrying most of the information about what a non-complaint looks like. Precision collapses to 23.5%.
• SMOTE reaches 74.72, worse than doing nothing but moving the threshold, and worse than class weights.

SMOTE’s poor showing here is not a refutation of the paper, and the chapter should not pretend it is. Chawla et al. evaluated nine low-dimensional dense numeric datasets — Pima, Phoneme, Adult, Satimage and so on — and explicitly listed text as future work, noting it “would have to be applied in conjunction with a feature selection algorithm” (Chawla et al. 2002). Interpolating between two sparse tf-idf rows — 83,281 dimensions here rather than the 101,659 quoted elsewhere, because this section fits on the 32,000-document fit slice rather than the full 40,000 — produces a point that corresponds to no document anyone could write. We measured it because it is what practitioners reach for, not because the authors claimed it would work here.

This is not a quirk of this corpus. Van den Goorbergh and colleagues found the same thing on clinical risk models, and put it more sharply than we would dare: imbalance corrections buy nothing in discrimination and do real damage elsewhere (van den Goorbergh et al. 2022). Section 12.6 measures the damage.

Elkan’s paper — the one everyone cites in the neighbourhood of this topic — argues the same way, and it is worth quoting the recommendation correctly because it is routinely inverted. Elkan’s advice is to learn a classifier from the training set as given, and then compute optimal decisions explicitly (Elkan 2001). His paper is an argument against resampling, not for it.

12.6 Is that number a probability?

Moving the threshold works. But it raises a question the previous section quietly skipped: 0.166 came from a sweep on held-out data. If you wanted to derive a threshold instead — from a cost matrix, say — you would need the model’s output to mean something. So: when the classifier says a review is 0.834 likely to be satisfied, is that a probability?

Everything here is fitted on the 8,000-document cal slice, which no model in this section was trained on, and scored on test. That separation is the whole point — and it is why the BERTurk row below is a model trained on the 32,000- document fit slice rather than the 40,000-document train that produced the learning curve. A calibrator fitted on documents the classifier already fits well reports that the classifier is calibrated when it is not.

Four statistics in the table need a sentence each. All of them compare the predicted probability of satisfied with what actually happened. ECE (expected calibration error) sorts the test documents into fifteen equal-width bins by predicted probability, takes the gap between each bin’s mean prediction and the fraction of that bin that really was satisfied, and averages the gaps weighted by bin size; it is printed in percentage points. MCE (maximum calibration error) is the largest of those gaps, whatever the bin size. The Brier score is the mean squared difference between the predicted probability and the 0/1 outcome, and log loss is the mean of \(-\log\) of the probability the model gave to the label that actually occurred. For all four, lower is better, and the last two need no bins at all. The three repairs are all fitted on cal. Writing \(z\) for the model’s logit and \(\sigma\) for the logistic sigmoid: temperature scaling fits one number \(T\) and outputs \(\sigma(z/T)\); Platt scaling fits a slope and an intercept and outputs \(\sigma(az + b)\); isotonic regression fits any map that never goes down.

model / method                       ECE    MCE    Brier   logloss   mean p
tf-idf, no correction — raw         3.96  39.32   0.0421    0.1514    0.940
  — temperature                     3.20  39.36   0.0426    0.1489    0.952
  — Platt                           0.94  16.58   0.0368    0.1318    0.935
  — isotonic                        0.69  32.96   0.0367    0.1309    0.935
  — isotonic fitted on TRAIN        2.39  24.66   0.0387    0.1487    0.942

tf-idf, class-weighted — raw       14.18  35.93   0.0675    0.2448    0.795
  — temperature                     7.80  43.29   0.0568    0.1974    0.859
  — Platt                           0.93  15.28   0.0364    0.1300    0.935
  — isotonic                        0.88  72.20   0.0364    0.1295    0.935

BERTurk, class-weighted — raw       2.53  49.84   0.0318    0.1442    0.933
  — temperature                     2.58  50.02   0.0303    0.1224    0.913
  — Platt                           1.25  31.65   0.0275    0.1062    0.936
  — isotonic                        0.56  45.84   0.0265    0.0984    0.936

Every ECE in that table carries an estimator floor, and several of the differences are below it. ECE compares an observed frequency against a mean prediction inside each bin, and an observed frequency is a binomial draw: even with perfect calibration the two differ by chance. With our bin occupancies that noise alone produces an expected \(|\text{ECE}|\) of about half a point. So Platt’s 0.94 against isotonic’s 0.69 — a gap of 0.25 — is not a measurement of a difference between the two methods, and neither is BERTurk’s 0.56, which is not distinguishable from perfect. What does survive the floor is the raw-versus- calibrated gap (3.96 → 0.9) and the class-weighting penalty (3.96 → 14.18), both several times the floor. Estimating ECE without bias is its own small literature (Kumar et al. 2019; Roelofs et al. 2022); the practical rule is to distrust any calibration difference smaller than about a point unless you have bootstrapped it.

Those numbers are summaries of a picture, and the picture is more informative than the summary. Each dot below is one of the fifteen bins, plotted at its mean predicted probability against the fraction of that bin that really was satisfied. Perfect calibration is the diagonal. Dot area is the number of documents in the bin, because a bin holding three documents and a bin holding 7,317 should not look equally important.

relRows = {
  // [model, calibration method, mean predicted probability in bin,
  //  observed frequency of the satisfied class, documents in bin]
  const raw = [
    ["plain","raw",0.0447,0.0000,3], ["plain","raw",0.0864,0.0000,5], ["plain","raw",0.1473,0.0000,1],
    ["plain","raw",0.2326,0.0769,13], ["plain","raw",0.2951,0.0000,10], ["plain","raw",0.3722,0.1538,13],
    ["plain","raw",0.4298,0.2222,18], ["plain","raw",0.5023,0.2258,31], ["plain","raw",0.5692,0.2093,43],
    ["plain","raw",0.6352,0.2419,62], ["plain","raw",0.7024,0.4074,108], ["plain","raw",0.7732,0.5318,173],
    ["plain","raw",0.8387,0.6644,444], ["plain","raw",0.9082,0.9312,1759], ["plain","raw",0.9727,0.9923,7317],
    ["plain","temp",0.0443,0.0000,6], ["plain","temp",0.0938,0.0000,3], ["plain","temp",0.1865,0.2000,5],
    ["plain","temp",0.2315,0.0000,12], ["plain","temp",0.2913,0.0000,7], ["plain","temp",0.3667,0.2000,15],
    ["plain","temp",0.4337,0.1667,18], ["plain","temp",0.5056,0.2593,27], ["plain","temp",0.5726,0.2051,39],
    ["plain","temp",0.6367,0.2653,49], ["plain","temp",0.7022,0.3086,81], ["plain","temp",0.7680,0.5276,127],
    ["plain","temp",0.8390,0.5750,320], ["plain","temp",0.9093,0.8784,1135], ["plain","temp",0.9776,0.9882,8156],
    ["plain","platt",0.0269,0.0984,61], ["plain","platt",0.1025,0.2683,41], ["plain","platt",0.1661,0.2105,38],
    ["plain","platt",0.2322,0.2439,41], ["plain","platt",0.3034,0.3023,43], ["plain","platt",0.3688,0.3571,42],
    ["plain","platt",0.4285,0.4600,50], ["plain","platt",0.5047,0.5098,51], ["plain","platt",0.5703,0.5806,62],
    ["plain","platt",0.6338,0.5545,101], ["plain","platt",0.7033,0.5946,111], ["plain","platt",0.7656,0.6789,190],
    ["plain","platt",0.8388,0.8457,337], ["plain","platt",0.9061,0.9261,852], ["plain","platt",0.9866,0.9893,7980],
    ["plain","iso",0.0892,0.0323,31], ["plain","iso",0.1562,0.2329,73], ["plain","iso",0.2561,0.2695,141],
    ["plain","iso",0.3296,0.0000,1], ["plain","iso",0.3415,0.4308,65], ["plain","iso",0.4074,0.4138,29],
    ["plain","iso",0.4906,0.6197,71], ["plain","iso",0.6315,0.5768,241], ["plain","iso",0.7186,0.6600,150],
    ["plain","iso",0.7568,0.7547,53], ["plain","iso",0.8515,0.8487,476], ["plain","iso",0.9112,0.9442,573],
    ["plain","iso",0.9885,0.9888,8096], ["plain","leak",0.0378,0.1818,143], ["plain","leak",0.0817,0.2353,17],
    ["plain","leak",0.1542,0.3386,127], ["plain","leak",0.2234,0.4545,22], ["plain","leak",0.2952,0.4918,61],
    ["plain","leak",0.3815,0.5588,136], ["plain","leak",0.4200,0.6667,9], ["plain","leak",0.4800,0.6111,18],
    ["plain","leak",0.5866,0.5902,61], ["plain","leak",0.6180,0.6575,73], ["plain","leak",0.7546,0.6000,70],
    ["plain","leak",0.8202,0.7264,106], ["plain","leak",0.9000,0.8484,343], ["plain","leak",0.9953,0.9834,8814],
    ["weighted","raw",0.0347,0.1389,72], ["weighted","raw",0.1005,0.2653,98], ["weighted","raw",0.1649,0.3107,103],
    ["weighted","raw",0.2351,0.4211,114], ["weighted","raw",0.2988,0.5641,156], ["weighted","raw",0.3674,0.6514,175],
    ["weighted","raw",0.4362,0.7535,215], ["weighted","raw",0.5028,0.8271,266], ["weighted","raw",0.5693,0.9286,392],
    ["weighted","raw",0.6347,0.9570,512], ["weighted","raw",0.7033,0.9475,648], ["weighted","raw",0.7694,0.9747,868],
    ["weighted","raw",0.8357,0.9903,1243], ["weighted","raw",0.9030,0.9955,1994], ["weighted","raw",0.9684,0.9981,3144],
    ["weighted","temp",0.0216,0.2646,291], ["weighted","temp",0.1013,0.4615,143], ["weighted","temp",0.1657,0.5487,113],
    ["weighted","temp",0.2337,0.6667,108], ["weighted","temp",0.2972,0.6633,98], ["weighted","temp",0.3667,0.7563,119],
    ["weighted","temp",0.4334,0.7983,119], ["weighted","temp",0.5023,0.8080,125], ["weighted","temp",0.5670,0.9018,163],
    ["weighted","temp",0.6340,0.9126,206], ["weighted","temp",0.7018,0.9643,280], ["weighted","temp",0.7704,0.9474,323],
    ["weighted","temp",0.8362,0.9470,453], ["weighted","temp",0.9039,0.9681,877], ["weighted","temp",0.9870,0.9953,6582],
    ["weighted","platt",0.0281,0.0909,55], ["weighted","platt",0.1045,0.2093,43], ["weighted","platt",0.1615,0.3143,35],
    ["weighted","platt",0.2340,0.2500,52], ["weighted","platt",0.2976,0.2564,39], ["weighted","platt",0.3674,0.3721,43],
    ["weighted","platt",0.4388,0.4400,50], ["weighted","platt",0.5015,0.4364,55], ["weighted","platt",0.5678,0.5000,86],
    ["weighted","platt",0.6336,0.5745,94], ["weighted","platt",0.7026,0.6818,132], ["weighted","platt",0.7708,0.7072,181],
    ["weighted","platt",0.8372,0.8199,311], ["weighted","platt",0.9056,0.9413,801], ["weighted","platt",0.9872,0.9889,8023],
    ["weighted","iso",0.0702,0.0345,29], ["weighted","iso",0.1406,0.1951,41], ["weighted","iso",0.2215,0.2850,200],
    ["weighted","iso",0.2780,1.0000,1], ["weighted","iso",0.3349,0.4250,40], ["weighted","iso",0.4622,0.0000,1],
    ["weighted","iso",0.4706,0.4222,45], ["weighted","iso",0.5909,0.5161,93], ["weighted","iso",0.6570,0.5674,141],
    ["weighted","iso",0.6934,0.6837,196], ["weighted","iso",0.7660,0.7853,191], ["weighted","iso",0.8598,0.8161,174],
    ["weighted","iso",0.9043,0.9410,814], ["weighted","iso",0.9893,0.9889,8034],
    ["bert","raw",0.0056,0.2210,525], ["bert","raw",0.1029,0.5500,20], ["bert","raw",0.1690,0.5833,12],
    ["bert","raw",0.2289,0.7273,11], ["bert","raw",0.2826,0.4286,7], ["bert","raw",0.3778,0.8462,13],
    ["bert","raw",0.4267,0.5000,10], ["bert","raw",0.5041,0.8462,13], ["bert","raw",0.5681,0.6429,14],
    ["bert","raw",0.6310,0.5833,12], ["bert","raw",0.7014,0.6842,19], ["bert","raw",0.7704,0.6000,25],
    ["bert","raw",0.8351,0.8077,52], ["bert","raw",0.9056,0.7759,116], ["bert","raw",0.9950,0.9857,9151],
    ["bert","temp",0.0194,0.1917,480], ["bert","temp",0.0944,0.5946,37], ["bert","temp",0.1588,0.3333,18],
    ["bert","temp",0.2310,0.7059,17], ["bert","temp",0.2977,0.6000,15], ["bert","temp",0.3549,0.5556,9],
    ["bert","temp",0.4320,0.6500,20], ["bert","temp",0.5041,0.7895,19], ["bert","temp",0.5676,0.6000,20],
    ["bert","temp",0.6356,0.6957,23], ["bert","temp",0.7035,0.6250,40], ["bert","temp",0.7679,0.8507,67],
    ["bert","temp",0.8399,0.7973,148], ["bert","temp",0.9065,0.9020,490], ["bert","temp",0.9787,0.9915,8597],
    ["bert","platt",0.1149,0.0485,227], ["bert","platt",0.1576,0.2281,171], ["bert","platt",0.2254,0.4524,42],
    ["bert","platt",0.2930,0.5750,40], ["bert","platt",0.3636,0.5263,19], ["bert","platt",0.4335,0.7500,16],
    ["bert","platt",0.4935,0.2667,15], ["bert","platt",0.5724,0.6000,15], ["bert","platt",0.6348,0.6250,16],
    ["bert","platt",0.6918,0.6000,15], ["bert","platt",0.7690,0.7179,39], ["bert","platt",0.8397,0.6471,51],
    ["bert","platt",0.9091,0.7750,200], ["bert","platt",0.9833,0.9860,9134], ["bert","iso",0.0220,0.0427,164],
    ["bert","iso",0.1050,0.0000,2], ["bert","iso",0.1676,0.1479,169], ["bert","iso",0.3018,0.1250,16],
    ["bert","iso",0.3333,0.3966,58], ["bert","iso",0.4762,0.5248,101], ["bert","iso",0.5416,1.0000,1],
    ["bert","iso",0.6419,1.0000,2], ["bert","iso",0.7228,0.6591,220], ["bert","iso",0.7975,0.8108,74],
    ["bert","iso",0.8237,0.7143,42], ["bert","iso",0.9168,0.9276,1105], ["bert","iso",0.9931,0.9937,8046]
  ];
  return raw.map(([model, method, conf, freq, n]) => ({model, method, conf, freq, n}));
}

viewof relModel = Inputs.radio(
  new Map([
    ["tf-idf with no imbalance correction", "plain"],
    ["the same model, class_weight=balanced", "weighted"],
    ["BERTurk, fine-tuned on the fit slice", "bert"]
  ]),
  { value: "plain", label: "model" }
)

viewof relPick = Inputs.checkbox(
  new Map([
    ["raw — what the model says", "raw"],
    ["temperature — one parameter", "temp"],
    ["Platt — slope and intercept", "platt"],
    ["isotonic — free monotone map", "iso"],
    ["isotonic fitted on TRAIN (the leak)", "leak"]
  ]),
  { value: ["raw", "platt"], label: "calibration" }
)

relPlot = {
  // only the uncorrected linear model has the deliberately-leaked calibrator
  const avail = relModel === "plain"
    ? ["raw", "temp", "platt", "iso", "leak"]
    : ["raw", "temp", "platt", "iso"];
  const keep = relPick.filter(m => avail.includes(m));
  const data = relRows.filter(d => d.model === relModel && keep.includes(d.method));
  const names = {raw: "raw", temp: "temperature", platt: "Platt",
                 iso: "isotonic", leak: "isotonic on TRAIN"};
  return Plot.plot({
    width: 640,
    height: 440,
    marginLeft: 60,
    marginBottom: 48,
    x: { label: "mean predicted P(satisfied) in the bin", domain: [0, 1], grid: true },
    y: { label: "observed fraction satisfied in the bin", domain: [0, 1], grid: true },
    r: { range: [2, 14] },
    color: { legend: true, domain: ["raw", "temp", "platt", "iso", "leak"],
             range: ["#B3453E", "#C08A3E", "#1F4E79", "#0E7C7B", "#6B7280"],
             tickFormat: k => names[k] },
    marks: [
      Plot.line([[0, 0], [1, 1]], { stroke: "currentColor", strokeDasharray: "5 4" }),
      Plot.line(data, { x: "conf", y: "freq", stroke: "method", strokeWidth: 1.5, strokeOpacity: 0.55 }),
      Plot.dot(data, { x: "conf", y: "freq", fill: "method", r: "n", fillOpacity: 0.75,
        title: d => `${names[d.method]}\npredicted ${(100*d.conf).toFixed(1)}%\n` +
                    `observed ${(100*d.freq).toFixed(1)}%\n${d.n.toLocaleString()} documents` })
    ]
  });
}

Switch the model to class_weight=balanced and leave only raw ticked. The whole curve sits above the diagonal: every bin contains more satisfied reviews than the model predicted. That is not noise, it is a systematic downward shift of every probability, and it is what the next few paragraphs are about.

The true base rate on test is 0.9368. Five findings, in order of how surprising they are.

Class weighting destroys calibration. ECE goes from 3.96 to 14.18, a 3.6× increase, and the model’s mean predicted probability drops to 0.795 against a true rate of 0.9368. The reweighted model is better by macro-F1 (76.09 vs 58.24) and its probabilities are far worse. If your model’s output is going to be shown to a human, or multiplied by a cost, or thresholded by a rule somebody else writes, class weighting silently breaks all three. This is van den Goorbergh et al.’s result, reproduced on text.

Temperature scaling gets halfway at best, and that is structural rather than bad luck. Temperature has one parameter and it preserves the ranking and the argmax; in a binary problem it can only push probabilities toward 0.5 or toward the extremes, symmetrically about the 0.5 cut. What is wrong with the class-weighted model is a shifted prior — it thinks complaints are commoner than they are — and that needs every logit moved by the same amount, not stretched. Temperature can move the mean only as a side effect of sharpening: here it lifts the mean prediction from 0.795 to 0.859 and ECE falls from 14.18 to 7.80, and then it has no parameter left. Platt scaling has two parameters, a slope and an intercept (Platt 1999), and the intercept is exactly the knob that fixes a prior shift. It takes ECE from 14.18 to 0.93.

Guo et al. popularised temperature scaling and found it to be a strong baseline on image and text classifiers (Guo et al. 2017). That is not in tension with this result: their networks were overconfident but not systematically shifted, which is the failure temperature is designed for. Reaching for temperature scaling here is reaching for a one-parameter fix to a two-parameter problem.

ECE and MCE disagree, and neither is enough on its own. Isotonic regression on the class-weighted model gets the best ECE of its group (0.88) and the worst MCE in the table (72.20). ECE is an \(n\)-weighted average, so a single sparse bin with a large gap barely moves it. MCE ignores bin size, so a single sparse bin decides it — and here that bin holds one document, which the isotonic map put at 0.278 and which happened to be satisfied. The next-worst gap for the same model is also a one-document bin (46.2 points); leaving single-document bins aside, it is 9.0 points, in a bin of 40. So 72.20 describes one review, not the tail: the binomial noise that puts a floor under ECE puts a far higher one under MCE. If your decisions live in the tail — and cost-sensitive decisions usually do — the average is the wrong summary, and the maximum is a noisy one. Look at the reliability diagram with its bin counts, which is what the dot sizes in the widget above are for.

isotonic regression — fits the calibration map as a free staircase that is only required to never go down. It can bend into any monotone shape a sigmoid cannot, which is why it wins with enough held-out data and overfits without it. ↩︎

Fitting the calibrator on training data does not look broken. The row marked “isotonic fitted on TRAIN” is the mistake this section exists to warn about: we fitted the isotonic map on the same 32,000 documents the classifier was trained on. It scores ECE 2.39. That is 1.7 points worse than fitting it properly on held-out data (0.69) — and still better than doing nothing (3.96). The leak does not produce an obviously wrong number. It produces a plausible number that is silently and clearly worse than the one you could have had, which is why it survives code review.

The transformer is better calibrated on average and worse in the tail. BERTurk’s raw ECE is 2.53 against the linear model’s 3.96, its Brier score is 0.0318 against 0.0421, and its mean prediction of 0.933 is within four thousandths of the true base rate. So the folk claim that deep networks are wildly overconfident does not reproduce here in the aggregate. What does reproduce is in the other column: BERTurk’s maximum calibration error is 49.84, the worst raw figure in the table. Somewhere there is a bin where the model says one thing and the data says something 50 points away — and Section 12.10 will show you exactly which documents live there. Calibrating fixes the average (isotonic takes ECE to 0.56 — indistinguishable from perfect at this bin structure, and with the best Brier and log loss in the chapter, which are the statistics that are sharp here). Its MCE stays at 45.84, but after calibration that figure comes from a bin holding a single document, so it says little. The raw model’s tail is the real finding: its three worst bins, holding 11, 13 and 20 documents, are between 44.7 and 49.8 points off.

This BERTurk was fine-tuned on the 32,000-document fit slice, not the 40,000 of train, precisely so that cal stays untouched. That makes it a slightly weaker classifier than the one in Section 12.3 — 84.45 macro-F1 against 85.02, which is what 8,000 fewer labels buys — and the only honest one to calibrate. Reusing the 40,000-label model here would have fitted the calibrator on 8,000 documents it had already been trained on — the exact leak the “isotonic on TRAIN” row measures.

This is Platt’s own warning, and it is usually overstated. He does not say never fit on the training set; he says that for nonlinear SVMs it “sometimes leads to disastrously biased fits”, and that for linear models “it is often possible to simply fit the sigmoid on the training set”. Our model is linear, and indeed the leak here is bad but not disastrous. The remedy he recommends is three-fold cross-validation, with a hold-out of “typically 30%” as the cheaper alternative (Platt 1999).

A note on which ECE

Expected calibration error is not one statistic. Guo et al. bin on the top-label confidence \(\max_k p_k\) and ask how often the argmax is right (Guo et al. 2017). In a binary problem that quantity lives in \([0.5, 1]\), so half the bins are empty by construction, and the number is not comparable to binning on \(P(\text{class } 1)\) directly. Both are reported by our measurement code; the table above uses the binary version, and the top-label version is systematically smaller — 8.98 rather than 14.18 for the class-weighted model.

The choice is not cosmetic, and applying temperature scaling to BERTurk shows why. On the binary statistic it makes things marginally worse, 2.53 → 2.58. On the top-label statistic it looks like a large win, 2.47 → 0.73. Same model, same fitted temperature, same test set, one operation — reported as a 3.4× improvement or a small regression depending purely on which ECE you print. Both numbers are correct, and they are answering different questions: how well the probability of satisfied is calibrated, versus how well the confidence in whichever label won is calibrated. Quoting an ECE without saying which one you computed is wrong.

There is a second degree of freedom, and on this corpus it is the larger one: how many bins, and how they are drawn. Ours are fifteen of equal width, and on a distribution this skewed the top bin holds 73% of the test set while ten bins hold fewer than a hundred documents each — so the statistic is decided by a handful of bins. Equal-mass (adaptive) binning puts the same number of documents in every bin and gives a different number for the same model (Nixon et al. 2019). Both schemes are biased upward by binomial noise, which is where the half-point floor above comes from, and there are estimators designed to reduce it (Kumar et al. 2019; Roelofs et al. 2022). Report the bin count, the binning scheme, and — if the difference you are claiming is under a point — a bootstrap interval.

12.7 Choosing the threshold from a cost matrix

Now the payoff. If the model’s output is a probability, you do not have to sweep for a threshold — you can derive one.

Let \(q = P(\text{complaint} \mid x)\), and suppose a false alarm costs \(C_{FP}\) and a missed complaint costs \(C_{FN}\). Flagging costs \((1-q) \cdot C_{FP}\); letting it pass costs \(q \cdot C_{FN}\). Flagging is worth it when

\[q > \frac{C_{FP}}{C_{FN} + C_{FP}} = \frac{1}{1+r}, \qquad r = \frac{C_{FN}}{C_{FP}}\]

That is Elkan’s threshold (Elkan 2001), and it comes with a condition that gets dropped every time it is quoted: \(q\) has to be a probability. Here is what happens when it is not. Costs are in units of \(C_{FP} = 1\), on the 10,000-document test set; “oracle” is the best possible threshold found by sweeping on test itself, so every row has a regret.

  r    Elkan q*    raw    calibrated   swept    oracle | regret: raw   calib   swept
  1      0.5000    572           501     498       483 |      18.4%    3.7%    3.1%
  2      0.3333    980           797     792       776 |      26.3%    2.7%    2.1%
  5      0.1667   1575          1357    1357      1342 |      17.4%    1.1%    1.1%
 10      0.0909   2218          2173    2167      2052 |       8.1%    5.9%    5.6%
 25      0.0385   4395          3551    3558      3479 |      26.3%    2.1%    2.3%
 50      0.0196   6764          4626    4546      4345 |      55.7%    6.5%    4.6%
100      0.0099   8373          5645    5647      5416 |      54.6%    4.2%    4.3%

Applying the correct formula to uncalibrated scores overspends by 8% to 56%. Applying it to isotonic-calibrated scores from the same model overspends by 1% to 7%. The formula was never the problem.

The mechanism is visible in the flagging counts. At \(r = 100\) the raw model raises 8,073 false alarms against 9,368 satisfied reviews — it flags 86% of the happy customers — because its scores are compressed into a region where a threshold of 0.0099 catches nearly everything. The calibrated model raises 3,345 false alarms and misses 23 complaints instead of 3. Both catch almost every complaint; one drowns the team in false alarms to do it.

costRows = [
  {r: 1,   qstar: 0.5000, raw: 572,  cal: 501,  swept: 498,  oracle: 483,  none: 632,   all: 9368, fpRaw: 9,    fnRaw: 563, fpCal: 140,  fnCal: 361, regRaw: 18.4, regCal: 3.7, regSwept: 3.1},
  {r: 2,   qstar: 0.3333, raw: 980,  cal: 797,  swept: 792,  oracle: 776,  none: 1264,  all: 9368, fpRaw: 38,   fnRaw: 471, fpCal: 279,  fnCal: 259, regRaw: 26.3, regCal: 2.7, regSwept: 2.1},
  {r: 5,   qstar: 0.1667, raw: 1575, cal: 1357, swept: 1357, oracle: 1342, none: 3160,  all: 9368, fpRaw: 270,  fnRaw: 261, fpCal: 492,  fnCal: 173, regRaw: 17.4, regCal: 1.1, regSwept: 1.1},
  {r: 10,  qstar: 0.0909, raw: 2218, cal: 2173, swept: 2167, oracle: 2052, none: 6320,  all: 9368, fpRaw: 1198, fnRaw: 102, fpCal: 1083, fnCal: 109, regRaw: 8.1,  regCal: 5.9, regSwept: 5.6},
  {r: 25,  qstar: 0.0385, raw: 4395, cal: 3551, swept: 3558, oracle: 3479, none: 15800, all: 9368, fpRaw: 4020, fnRaw: 15,  fpCal: 2051, fnCal: 60,  regRaw: 26.3, regCal: 2.1, regSwept: 2.3},
  {r: 50,  qstar: 0.0196, raw: 6764, cal: 4626, swept: 4546, oracle: 4345, none: 31600, all: 9368, fpRaw: 6464, fnRaw: 6,   fpCal: 2776, fnCal: 37,  regRaw: 55.7, regCal: 6.5, regSwept: 4.6},
  {r: 100, qstar: 0.0099, raw: 8373, cal: 5645, swept: 5647, oracle: 5416, none: 63200, all: 9368, fpRaw: 8073, fnRaw: 3,   fpCal: 3345, fnCal: 23,  regRaw: 54.6, regCal: 4.2, regSwept: 4.3}
]

viewof costRatio = Inputs.radio(
  new Map([
    ["r = 1 — a false alarm and a missed complaint cost the same", 1],
    ["r = 2", 2],
    ["r = 5 — a missed complaint costs five false alarms", 5],
    ["r = 10", 10],
    ["r = 25", 25],
    ["r = 50", 50],
    ["r = 100 — a missed complaint costs a hundred false alarms", 100]
  ]),
  { value: 25, label: "cost ratio" }
)

costPlot = {
  const row = costRows.find(d => d.r === costRatio);
  const bars = [
    {policy: "flag nothing", cost: row.none, kind: "trivial"},
    {policy: "flag everything", cost: row.all, kind: "trivial"},
    {policy: "Elkan q* on raw scores", cost: row.raw, kind: "bad"},
    {policy: "Elkan q* on calibrated scores", cost: row.cal, kind: "good"},
    {policy: "threshold swept on cal", cost: row.swept, kind: "good"},
    {policy: "oracle — swept on test itself", cost: row.oracle, kind: "oracle"}
  ];
  return Plot.plot({
    width: 700,
    height: 300,
    marginLeft: 210,
    marginBottom: 50,
    x: { label: `total cost on the 10,000-document test set, in units of one false alarm — q* = ${row.qstar.toFixed(4)}`,
         grid: true },
    y: { label: null, domain: bars.map(d => d.policy) },
    color: { domain: ["trivial", "bad", "good", "oracle"],
             range: ["currentColor", "#B3453E", "#0E7C7B", "#1F4E79"], legend: false },
    marks: [
      Plot.barX(bars, { y: "policy", x: "cost", fill: "kind", rx: 3,
        title: d => `${d.policy}\n${d.cost.toLocaleString()} cost units` }),
      Plot.text(bars, { y: "policy", x: "cost", text: d => d.cost.toLocaleString(),
        dx: 6, textAnchor: "start", fontWeight: 600 }),
      Plot.ruleX([row.oracle], { stroke: "#1F4E79", strokeDasharray: "4 3" })
    ]
  });
}

Step the ratio up from 1 to 100 and watch the red bar. At \(r = 10\) the raw-score policy is respectable, 8% over the best threshold; at every other ratio below 50 it overspends by 17–26%. At \(r = 50\) it costs 56% more than the best threshold available. Uncalibrated scores fail worst exactly where the stakes are highest, because that is where \(q^*\) moves into the part of the score range the model has compressed.

Note what happens at \(r = 1\), where Elkan’s threshold is the familiar 0.5. The raw model flags 78 documents and misses 563 of 632 complaints. This is the same pathology as the 58.24 macro-F1 in Section 12.5, seen from the cost side: at equal costs the textbook threshold is 0.5, and on an uncalibrated imbalanced model 0.5 is nowhere near the right place to stand.

Sweeping directly for the cost-minimising threshold on cal does about as well as calibrating and then applying the formula — regret 1% to 6% against 1% to 7%. That is a genuinely useful practical finding: if all you need is one operating point for one fixed cost ratio, you can skip calibration and sweep. Calibration earns its place when the costs might change, when the number is shown to a person, or when something downstream multiplies it.

For scale, the two trivial policies: flagging nothing costs \(632r\), so 63,200 at \(r=100\); flagging everything costs 9,368 regardless of \(r\). At \(r = 100\) the raw model’s 8,373 is barely better than flagging the entire corpus.

12.8 Zero-shot and few-shot with LLMs

Everything so far assumed you have labels. Most projects start with none. The question this section answers with measurements is what a general-purpose model scores when you simply ask it, and — the part that decides real projects — how that compares to a fine-tuned encoder at each label budget you might plausibly have.

Zero-shot classification is attractive for an obvious reason: the labelling budget is zero and the model ships today. It is also the setting in which every tool this chapter has built stops working, which is the less obvious reason to read the numbers carefully.

One correction to that first sentence, which this section is itself evidence for. The labelling budget for training is zero. The budget for finding out whether it works is not: every number below rests on 2,000 labelled documents, and you will need your own.

zero-shot classification — labelling from an instruction alone, with no examples and no training. The model returns a word, not a score, so there is no threshold to move and no calibration map to fit. Every technique in Section 12.5, Section 12.6 and Section 12.7 needs a number between 0 and 1 that this setting does not give you. ↩︎

Making the comparison honest

A comparison between a prompted model and a fine-tuned one is worthless unless both are scored on the same documents. Prompting 10,000 reviews through a 27B model is between 1.6 and 2.4 hours per condition depending on how busy the laptop is — the lower figure from the median, the higher from the wall clock we actually observed — so the LLM runs use a stratified 2,000-document subsample of the same test slice — 1,874 satisfied and 126 complaints, a 6.30% minority rate against the full slice’s 6.32%.

The rule that makes this legitimate is that everything else gets re-scored on that identical subsample. The tf-idf and BERTurk numbers in this section are not the numbers from earlier sections; they are the same models read on the same 2,000 documents the LLMs saw, in the same order.

re-scored on the 2,000-document subsample     acc   macro-F1  95% CI

always "satisfied"                           93.70    48.37
tf-idf + LR, class-weighted                  93.40    78.20   [74.77, 81.36]
tf-idf + LR, threshold moved to 0.166        95.40    80.23   [76.35, 83.60]
BERTurk fine-tuned,    200 labels            92.25    52.65   [49.77, 55.93]
BERTurk fine-tuned,    500 labels            93.70    59.47   [55.04, 63.97]
BERTurk fine-tuned,  1,000 labels            94.95    72.78   [68.12, 77.02]
BERTurk fine-tuned,  2,000 labels            96.05    82.00   [78.13, 85.42]
BERTurk fine-tuned,  5,000 labels            95.35    81.21   [77.57, 84.51]
BERTurk fine-tuned, 10,000 labels            96.65    84.97   [81.40, 88.07]
BERTurk fine-tuned, 20,000 labels            96.50    84.84   [81.21, 87.94]
BERTurk fine-tuned, 40,000 labels            96.85    86.81   [83.42, 89.60]

These are not the numbers in Section 12.2 and Section 12.3, and they should not be. Those were read on the full 10,000; these are the same fitted models read on 2,000 of them. BERTurk at 500 labels scores 59.47 here and 56.46 there — same model, different slice. Any figure quoted in these two sections carries the subsample with it, so never compare one against a number from earlier in the chapter.

Note that the majority-class baseline is 93.70% accuracy here. Any accuracy figure in this section that starts with a 9 should be read against that number and not against your intuitions.

The intervals are 2,000-resample bootstraps over the 2,000 documents. They are wide — roughly ±3.5 points — because 126 complaints is not many, and the minority class is what macro-F1 is mostly measuring. A subsample buys you affordable LLM runs and pays for it in resolution, and the honest response is to print the interval rather than to pretend the point estimate is the finding.

Four models ran, all 4-bit quantised (Q4_K_M) on one laptop through ollama, which is worth stating plainly: these are not frontier models behind an API. They are what fits on a machine you can carry.

The four local models, all Q4_K_M quantised

model	parameters	on disk	note
llama3.2:3b	3.2B	2.0 GB	the small-model end of the range; no thinking mode, which is why it is the control in the think: false check
qwen3.5:9b	9.7B	6.6 GB	dense, the workhorse — and a reasoning model, which the broken run below is entirely about
gemma3:27b	27.4B	17.4 GB	dense, the largest here
qwen3.6:35b-a3b	36.0B total, ~3B active	23.9 GB	mixture-of-experts (Section 7.7), also a reasoning model

4-bit quantised means most weights are stored in four bits instead of sixteen. Q4_K_M keeps some tensors at higher precision and stores a scale per small block of weights, so the average is closer to five bits: gemma3:27b’s 27.4B weights take 17.4 GB, about a third of the 55 GB they would need at 16 bits. The cost in fidelity is usually small and is never zero. It is why a 27B model fits on a laptop at all. It also means this comparison is not quite fair to the prompted side: BERTurk ran at full precision and these did not. We could not measure the unquantised versions because they do not fit on the machine, so read the crossover as measured for models you can run locally, and expect a full-precision or hosted model to push it to the right.

The run that scored 93.75% by being completely broken

Before any results, the failure that nearly got published, because it is the most useful thing in this section.

The first pass of this matrix scored qwen3.5:9b at 93.75% accuracy and 48.39 macro-F1 over the 400 documents it got through before it was killed. Read those two numbers against the table above: 93.70 and 48.37. The model was predicting “satisfied” on every single document, and the score it earned for that is the majority-class baseline to within a rounding error.

Here is one archived record from that run, verbatim, and all 400 look like it:

{"i": 0, "y": 1, "pred": 1, "conf": null, "ok": false, "raw": "",
 "secs": 2.408, "eval_count": 24, "prompt_eval_count": 99}

           raw : ''    <- what the parser reads. Empty, on all 400.
    eval_count : 24    <- tokens generated: exactly the cap, on all 400.
    parse rate : 0.0%  <- the field nobody was looking at.

qwen3.5 is a reasoning model. ollama returns its chain of thought in a separate thinking field, and the response field stays empty until the thinking finishes. With num_predict capped at 24 tokens the budget was spent entirely inside the reasoning block, response was never written, every answer failed to parse, and every document fell through to the fallback label — which is the majority class, which is why the result looked plausible instead of looking like a crash.

Note what the archived record does not contain: any sign of the thinking field, or the reason generation stopped. The harness was not recording them, which is why the diagnosis needed a fresh query to the model rather than a look at the log. think_chars and done_reason are in the schema now — added because of this, which is the usual way a log field gets born.

An accuracy of 93.75% on this corpus is not a result. It is the shape a broken harness takes when the fallback is the majority class. Six of the ten runs in this matrix would have carried the same defect.

Three changes fixed it, and only the first is about the bug:

1. Send think: false, with a fallback that retries without the field for models that reject it. Parse rate went from 0.0% to 100.0%, and the output on llama3.2:3b — which has no thinking mode — was byte-identical, confirming the flag is a no-op where it should be.
2. Abort a run whose first 40 answers parse below 50%, rather than spending nine hours of GPU time producing a table of majority-class baselines.
3. Smoke-test all six model/condition combinations on 24 documents each before committing the machine to an overnight queue.

The rule those three fixes do not state, and should. All three are specific to this incident: one is a bug fix, two are sentries. The general rule is simpler and it is the one to carry away — an unparseable answer is not a prediction. Never let it fall through to a label: not to the majority class, not to a random class, not to the previous document’s answer. Emit a third outcome, unparseable, and carry it into every metric you report: accuracy over the documents that parsed, and the parse rate, side by side. A model that answers 60% of the time at 95% accuracy is a different product from one that answers every time at 57%. Score both over all documents with non-answers counted as wrong and they look identical, 57% each; give the first a majority-class fallback on this corpus and it scores about 94% and looks like the better model. Only the pair of numbers tells them apart. Note that the abort guard alone would not have saved us: a run parsing at 55% passes it and still quietly imputes the majority class on the other 45%. Log the raw response, the finish reason and the length of the reasoning field on every call, because the field you need for the diagnosis is always the one you did not think to record.

This is the same lesson as the 27× inference outlier in Section 12.3, arriving from the opposite direction. There the number was implausibly bad and the harness was at fault. Here the number was comfortably good and the harness was at fault. The bad number gets investigated; the good one gets published. That asymmetry, not the bug, is the thing to design against — which is what the abort guard does.

Every run reported below parsed at 100.0%.

The results

Four models, three prompt settings, one corpus. The interesting comparison is not which model wins — it is how much of the spread is explained by size, and how much by things that cost nothing.

2,000 documents, temperature 0, seed 42, single process. Every run parsed 100%.
                             acc  macroF1   95% CI       cP     cR    cF1   s/doc  prompt
                                                                                    tokens
llama3.2:3b      zero/tr   66.35    52.32  [50.10,54.68]  15.34  96.03  26.45  0.121   140
llama3.2:3b      zero/en   87.80    69.21  [65.98,72.38]  31.56  80.16  45.29  0.124   123
gemma3:27b       zero/tr   88.65    72.28  [69.18,75.28]  35.01  93.65  50.97  0.592   124
qwen3.5:9b       conf/tr   92.95    78.10  [74.85,81.25]  46.70  84.13  60.06  0.475   188
gemma3:27b        few/tr   93.20    79.45  [76.26,82.51]  47.90  90.48  62.64  0.560   549
qwen3.5:9b       zero/tr   93.70    79.53  [76.18,82.67]  50.00  83.33  62.50  0.365   136
qwen3.5:9b       zero/en   93.65    80.07  [76.88,83.18]  49.78  88.10  63.61  0.357   113
qwen3.5:9b        few/tr   94.35    80.75  [77.38,83.93]  53.37  81.75  64.58  0.290   584
qwen3.6:35b-a3b  zero/tr   94.65    81.56  [78.22,84.83]  55.03  82.54  66.03  0.353   136
qwen3.6:35b-a3b   few/tr   94.95    82.08  [78.62,85.31]  56.98  80.95  66.89  0.302   584

cP / cR / cF1 are precision, recall and F1 on the complaint class.
s/doc is the median; see the crossover section below for why not the mean.

The widget below is the same table, sorted and plotted. Everything it shows is in the block above, so nothing here depends on being able to run it.

Two boundaries worth naming before the findings. Chapter 17 takes prompt design seriously as a subject; this section only asks what prompting scores against a trained alternative. And Chapter 18 treats “getting a parseable answer out of a model” as a problem in its own right — which, as the previous subsection showed, it emphatically is.

llmRows = [
  {tag: "llama3.2:3b zero/tr", model: "llama3.2:3b", cond: "zero", lang: "tr",
   acc: 66.35, mf1: 52.32, lo: 50.10, hi: 54.68, cp: 15.34, cr: 96.03, cf1: 26.45, secs: 0.121, ptok: 140},
  {tag: "llama3.2:3b zero/en", model: "llama3.2:3b", cond: "zero", lang: "en",
   acc: 87.80, mf1: 69.21, lo: 65.98, hi: 72.38, cp: 31.56, cr: 80.16, cf1: 45.29, secs: 0.124, ptok: 123},
  {tag: "gemma3:27b zero/tr", model: "gemma3:27b", cond: "zero", lang: "tr",
   acc: 88.65, mf1: 72.28, lo: 69.18, hi: 75.28, cp: 35.01, cr: 93.65, cf1: 50.97, secs: 0.592, ptok: 124},
  {tag: "qwen3.5:9b conf/tr", model: "qwen3.5:9b", cond: "conf", lang: "tr",
   acc: 92.95, mf1: 78.10, lo: 74.85, hi: 81.25, cp: 46.70, cr: 84.13, cf1: 60.06, secs: 0.475, ptok: 188},
  {tag: "gemma3:27b few/tr", model: "gemma3:27b", cond: "few", lang: "tr",
   acc: 93.20, mf1: 79.45, lo: 76.26, hi: 82.51, cp: 47.90, cr: 90.48, cf1: 62.64, secs: 0.560, ptok: 549},
  {tag: "qwen3.5:9b zero/tr", model: "qwen3.5:9b", cond: "zero", lang: "tr",
   acc: 93.70, mf1: 79.53, lo: 76.18, hi: 82.67, cp: 50.00, cr: 83.33, cf1: 62.50, secs: 0.365, ptok: 136},
  {tag: "qwen3.5:9b zero/en", model: "qwen3.5:9b", cond: "zero", lang: "en",
   acc: 93.65, mf1: 80.07, lo: 76.88, hi: 83.18, cp: 49.78, cr: 88.10, cf1: 63.61, secs: 0.357, ptok: 113},
  {tag: "qwen3.5:9b few/tr", model: "qwen3.5:9b", cond: "few", lang: "tr",
   acc: 94.35, mf1: 80.75, lo: 77.38, hi: 83.93, cp: 53.37, cr: 81.75, cf1: 64.58, secs: 0.290, ptok: 584},
  {tag: "qwen3.6:35b-a3b zero/tr", model: "qwen3.6:35b-a3b", cond: "zero", lang: "tr",
   acc: 94.65, mf1: 81.56, lo: 78.22, hi: 84.83, cp: 55.03, cr: 82.54, cf1: 66.03, secs: 0.353, ptok: 136},
  {tag: "qwen3.6:35b-a3b few/tr", model: "qwen3.6:35b-a3b", cond: "few", lang: "tr",
   acc: 94.95, mf1: 82.08, lo: 78.62, hi: 85.31, cp: 56.98, cr: 80.95, cf1: 66.89, secs: 0.302, ptok: 584}
]

viewof llmMetric = Inputs.radio(
  new Map([
    ["Macro-F1, with its bootstrap interval", "mf1"],
    ["Complaint F1 — the minority class alone", "cf1"],
    ["Complaint precision and recall, separately", "pr"]
  ]),
  { value: "mf1", label: "" }
)

llmPlot = {
  const refs = [
    {label: "BERTurk, 40,000 labels", mf1: 86.81, cf1: 75.29, colour: "#1F4E79"},
    {label: "BERTurk, 2,000 labels",  mf1: 82.00, cf1: 66.09, colour: "#2E7D6F"},
    {label: "tf-idf, threshold moved", mf1: 80.23, cf1: 62.90, colour: "#B3453E"},
    {label: "always “satisfied”",      mf1: 48.37, cf1: 0.00,  colour: "currentColor"}
  ];
  const order = llmRows.map(d => d.tag);
  if (llmMetric === "pr") {
    const long = llmRows.flatMap(d => [
      {tag: d.tag, kind: "precision", v: d.cp},
      {tag: d.tag, kind: "recall", v: d.cr}
    ]);
    return Plot.plot({
      width: 700, height: 400, marginLeft: 175, marginBottom: 46,
      x: {label: "percent, on the complaint class", domain: [0, 100], grid: true},
      y: {label: null, domain: order},
      color: {domain: ["precision", "recall"], range: ["#1F4E79", "#B3453E"], legend: true},
      marks: [
        Plot.ruleX([74.42], {stroke: "#1F4E79", strokeDasharray: "4 3", strokeOpacity: 0.7}),
        Plot.ruleX([76.19], {stroke: "#B3453E", strokeDasharray: "4 3", strokeOpacity: 0.7}),
        Plot.dot(long, {y: "tag", x: "v", fill: "kind", r: 5,
          title: d => `${d.tag}\ncomplaint ${d.kind}: ${d.v.toFixed(2)}%`}),
        Plot.ruleY(llmRows, {y: "tag", x1: "cp", x2: "cr", stroke: "currentColor", strokeWidth: 1})
      ]
    });
  }
  const key = llmMetric;
  return Plot.plot({
    width: 700, height: 400, marginLeft: 175, marginBottom: 46,
    x: {label: key === "mf1" ? "macro-F1 on the 2,000-document subsample"
                             : "complaint F1 on the 2,000-document subsample",
        domain: [0, 100], grid: true},
    y: {label: null, domain: order},
    marks: [
      ...refs.map(r => Plot.ruleX([r[key]], {stroke: r.colour, strokeDasharray: "4 3", strokeOpacity: 0.8})),
      ...refs.map(r => Plot.text([{y: order[0], x: r[key]}], {
        y: "y", x: "x", text: [r.label], dy: -14, dx: 3, rotate: 0,
        fill: r.colour, textAnchor: "start", fontSize: 9})),
      key === "mf1"
        ? Plot.ruleY(llmRows, {y: "tag", x1: "lo", x2: "hi", stroke: "currentColor", strokeWidth: 6, strokeOpacity: 0.35})
        : null,
      Plot.dot(llmRows, {y: "tag", x: key, fill: "#0E7C7B", r: 5,
        title: d => `${d.tag}\nmacro-F1 ${d.mf1.toFixed(2)} [${d.lo.toFixed(2)}, ${d.hi.toFixed(2)}]`
                  + `\ncomplaint P ${d.cp.toFixed(1)} R ${d.cr.toFixed(1)} F1 ${d.cf1.toFixed(2)}`
                  + `\n${d.secs.toFixed(3)}s per document (median)`}),
      Plot.text(llmRows, {y: "tag", x: key, text: d => d[key].toFixed(2),
        dx: 9, textAnchor: "start", fontSize: 10, fontWeight: 600})
    ].filter(d => d)
  });
}

Switch to complaint precision and recall. The grey bar joining the two dots is the gap between them, and every single run leans the same way: recall high, precision low. The two dashed rules are BERTurk at 40,000 labels — 74.42 precision, 76.19 recall — which is the only model on the chart whose two dots sit almost on top of each other.

Three findings, each tested with a paired bootstrap on the identical 2,000 documents rather than by eyeballing whether the intervals above overlap. Overlapping intervals are not a test: two models scored on the same documents share most of their sampling noise, and the difference can be significant while the marginal intervals overlap heavily. Every \(p\) below is from 10,000 paired resamples of the difference.

Writing the prompt in English helps — if the model is weak

                        tr       en      difference        p
llama3.2:3b           52.32    69.21   +16.90 [+14.59, +19.13]   0.0000
qwen3.5:9b            79.53    80.07    +0.54 [ -1.14,  +2.25]   0.5382

The folklore says: write the instruction in English even when the data is Turkish. On the 3B model that is worth +16.90 macro-F1 and is about as unambiguous as a measurement gets. On the 9B model the same change is +0.54 points, \([-1.14, +2.25]\) — which is not a demonstration that the effect is gone. It is a demonstration that if an effect remains, it is smaller than about two and a half macro-F1 points. That is a useful bound and a weaker statement than “noise”, and the distinction is worth keeping: the same design would also have failed to resolve a real two-point gain.

The Turkish-instruction failure on llama3.2:3b is not a subtle degradation. It labels 96.03% of complaints correctly and reaches 15.34% precision doing it — it calls roughly 40% of documents complaints when the true rate is 6.3%. The English instruction does not teach it Turkish; it cuts that over-flagging to about 16%.

This is why “use English prompts” survives as advice. It was true, on the models people first tried it on, and it is still true for small models like the 3B above — but it is repeated long after it stopped buying much on the stronger models people use now. The check costs one extra run. Advice about LLMs has a shelf life measured in model generations, and the only way to know whether a rule still holds is to spend an hour re-measuring it on the model you are actually going to ship.

Eight examples help — if the model is weak

                       zero      few     difference        p
gemma3:27b            72.28    79.45    +7.17 [ +5.35,  +9.04]   0.0000
qwen3.5:9b            79.53    80.75    +1.22 [ -1.15,  +3.60]   0.3222
qwen3.6:35b-a3b       81.56    82.08    +0.51 [ -1.51,  +2.48]   0.6164

Same shape as the language result. In-context learning with 8 labelled examples — four of each class, drawn from fit so that no test document is ever in a prompt — moves gemma3:27b by 7.17 points and the two qwen3 models by amounts their intervals cannot separate from zero. Read those the same way as the language result: qwen3.6:35b-a3b’s \(+0.51\) carries \([-1.51, +2.48]\), so what the run establishes is a ceiling of about two and a half points, not the absence of a gain.

in-context learning — learning a task from examples placed in the prompt. No weight is updated and nothing survives the call: the same eight examples are re-sent, re-read and re-charged for every document you classify. That is the cost structure, and it is the opposite of fine-tuning, which pays once and then serves cheaply. ↩︎

Two practical notes. First, those 8 examples cost 584 prompt tokens against 136, which is a 4.3× larger prompt on every one of your documents forever. Second, the ordering and choice of examples is itself a source of variance large enough to swamp the effect — Zhao et al. showed accuracy moving from near chance to near state-of-the-art on the same task with the same number of examples, just reordered (Zhao et al. 2021). We used one fixed, seeded selection, so the +7.17 on gemma3:27b should be read as one draw from that distribution rather than as the expected value of adding examples.

Every prompted run here was recall-biased, and there is no threshold to fix it with

This is the finding with the most operational consequence, and it is visible in the widget above under complaint precision and recall:

                              complaint P    complaint R
llama3.2:3b zero/tr              15.34          96.03
gemma3:27b zero/tr               35.01          93.65
qwen3.5:9b zero/tr               50.00          83.33
qwen3.6:35b-a3b few/tr           56.98          80.95
BERTurk, 40,000 labels           74.42          76.19

Every prompted run flags far more complaints than it should. The best of them reaches 56.98% precision at 80.95% recall; BERTurk sits at 74.42 and 76.19, nearly balanced. Ask a model “is this a complaint?” and it says yes too often.

The problem is not that the operating point is wrong. It is that there is no operating point to move. Section 12.5 fixed the mirror image of this pathology — a linear model that flagged far too few complaints at 0.5 — by shifting one threshold, and gained 19.42 macro-F1 points for free. That instrument does not exist here. A zero-shot model returns the word ŞİKAYET, not a number, and you cannot threshold a word.

Everything this chapter built downstream of a score goes with it: no ROC curve, no PR curve, no isotonic calibration, no \(q^*\) from a cost ratio, no cost-minimising sweep. You can recover some of it — read the model’s own log-probabilities for the two candidate answer words, ŞİKAYET against MEMNUN, if your serving software exposes them, which turns a word back into a score; or ask for a confidence in words, as the next subsection does. But by default, choosing a prompted model means choosing to give up the threshold.

One boundary on this finding: we did not try to argue the models out of it. A prompt that says “answer ŞİKAYET only if the reviewer states a concrete fault” would plausibly move the operating point, and we did not measure that. What no prompt gives you is a dial — a knob you can turn after deployment, when the cost ratio changes and you need a different point on the same curve.

Is a confidence the model writes down a probability?

The conf condition asked qwen3.5:9b for LABEL|CONFIDENCE, a 0–100 integer. All 2,000 answers parsed. This is verbalised confidence, and it is the closest thing to a score the zero-shot setting offers.

verbalised confidence — a number the model writes out as text, rather than a probability read off its scores. Judge it the way Section 12.6 judges any other confidence: does it match the frequency of being right, and does it separate right from wrong? ↩︎

stated confidence      n      actually right
     50                3          100.0%
     60                1          100.0%
     70                1          100.0%
     75               62           80.6%
     85               25           76.0%
     90              286           83.2%
     92               23           82.6%
     95            1,549           95.4%
    100               50          100.0%

mean stated confidence  93.53      overall accuracy  92.95      gap  +0.58
expected calibration error (over stated values)                 1.80

On the test people actually report — does average stated confidence match average accuracy? — it does remarkably well. The average stated confidence is 93.53% and the model is right 92.95% of the time, a gap of +0.58 points. Binned over the nine stated values its ECE is 1.80, against 8.98 for the class-weighted tf-idf model. Read that comparison loosely: as Section 12.6 insisted, an ECE means nothing without its binning scheme, and this one uses nine bins the model chose rather than fifteen equal-width ones. It is the right order of magnitude, not a like-for-like number. Lin et al. showed that a model can be trained to express calibrated uncertainty in words (Lin et al. 2022), and on this task, out of the box, it is.

Then look at the second column. 1,549 of the 2,000 documents — 77.45% — got the same number, 95. There are 9 distinct values in the entire run. A confidence that is almost always 95 cannot tell you much about which documents to route to a human.

Measured as a detector of its own errors, stated confidence scores an AUROC of 0.667. BERTurk’s own top-label probability, on the same 2,000 documents, scores 0.843 across 1,325 distinct values. Both models are somewhat able to flag their own mistakes; the trained one is better at it and gives you a dial rather than a switch.

Read that gap with one caveat, because the chapter has already told you the reason and not drawn the consequence. AUROC awards half credit on ties, and 77.45% of the LLM’s documents share a single confidence value — so roughly three-fifths of all document pairs are ties for it. Of the pairs AUROC actually compares, one right answer against one wrong answer, 45% are ties; for BERTurk, with 1,325 distinct values, almost none are. Part of 0.667-against-0.843 is a worse signal and part of it is a coarser one, and this comparison does not separate them. The practical conclusion survives either way — nine values give you a switch, 1,325 give you a dial — but do not read the gap as “the LLM’s confidence is 0.18 of AUROC worse”.

The one useful split is real, though: at stated confidence ≥ 95 the model is right 95.56% of the time (n = 1,599), and below 95 it is right 82.54% (n = 401). If your workflow only needs “send a fifth of these to a person”, that is enough to build on — with the caveat the rest of this chapter would insist on. We picked 95 by looking at that table, and that table is the evaluation set. By the rule stated in Section 12.1, 95.56 against 82.54 is therefore an oracle split and an upper bound, not a result. Choose the cut on held-out documents before you route anything on it.

Xiong et al. evaluated black-box confidence elicitation across models and tasks and report AUROCs in the 0.522–0.605 band, together with a systematic tendency to overconfidence they attribute to models imitating how people express certainty (Xiong et al. 2024). Our 0.667 is slightly above their range and our average gap is small — but the 77.5%-at-one-value concentration is precisely the failure mode they describe. A confidence can be right on average and still separate almost nothing.

12.9 Small model vs. LLM: the crossover

Now the decision. Both previous sections produced a number on the same 2,000 documents, so they can finally be compared directly. Two axes decide it, and neither is quality-versus-quality: how many labels do you have, and what does a document cost to classify.

The crossover is at 2,000 labels, and it is a tie rather than a win

So where do the two curves meet? The honest way to answer is not to compare two point estimates, because two numbers that differ can still be the same number wearing noise. Both systems ran on the same documents, so we can pair them and bootstrap the difference directly.

paired bootstrap, same 2,000 documents, 10,000 resamples

BERTurk  1,000 labels  vs  qwen3.6:35b-a3b zero-shot    -8.79 [-13.53,  -4.24]   p = 0.0002
BERTurk  2,000 labels  vs  qwen3.6:35b-a3b zero-shot    +0.43 [ -3.17,  +3.85]   p = 0.8030
BERTurk  2,000 labels  vs  qwen3.6:35b-a3b 8-shot       -0.08 [ -3.78,  +3.61]   p = 0.9728
BERTurk 40,000 labels  vs  qwen3.6:35b-a3b 8-shot       +4.73 [ +1.56,  +7.90]   p = 0.0028

At 2,000 labels the fine-tuned encoder and the best prompted model are statistically indistinguishable at this sample size. That is a weaker claim than a tie, and the interval says how much weaker. With 126 complaints in the subsample the paired standard error is 1.89 macro-F1 points, so this design can only rule out differences larger than about 3.2 points (the 90% interval runs \([-3.18, +3.02]\), which is what an equivalence test would work with) and has 80% power only against differences above 5.3 points. Read the 2,000-label row as we could not tell them apart, not as they are the same: a genuine equivalence claim at, say, ±1 point would need roughly ten times this many documents at this class balance. Note also that our own +4.73 at 40,000 labels sits below that 80%-power threshold — it reached significance, but the design was not built to guarantee it would.

At 1,000 labels the LLM wins by 8.79 points and the result is not close. At 40,000 the encoder wins by 4.73 and that result is not close either.

So the crossover on this task sits at roughly 2,000 labelled documents, and the useful way to hold it is as three regimes rather than a verdict — plus a fourth option that the “prompt or fine-tune” framing hides:

• Under ~1,000 labels. Prompt. The encoder has not yet learned what a complaint is — at 500 labels it scores 59.47 macro-F1 on this subsample against the majority baseline’s 48.37, catching 14.29% of complaints — and we doubt a learning-rate sweep would change that, though we did not run one. A different recipe might: setfit (Section 12.13) is contrastive fine-tuning of a sentence encoder plus a logistic head, it is built for exactly this budget, and it routinely reports usable macro-F1 from 8–64 examples per class (Tunstall et al. 2022). We did not measure it, and running it at 200, 500 and 1,000 labels is the single measurement most likely to change this bullet.
• Around 2,000. Genuinely a coin flip on quality. Decide on cost, latency, operational control, and whether you need a threshold. All four of those point the same way, which is the subject of the next subsection.
• Above 2,000. Fine-tune. We have exactly one more measured point — at 40,000 the encoder wins by 4.73 — so where between 2,000 and 40,000 the gap becomes decisive is interpolation from the encoder’s own learning curve, not a measurement. What is measured is the direction, and that the encoder is the only one of the two that hands you a score.
• The option this framing leaves out. You do not have to choose between paying the prompted model forever and labelling by hand. Spend it once: label your unlabelled pool with the big model, then fine-tune the encoder on those labels — the LLM as annotator, the encoder as student (Hsieh et al. 2023; Gilardi et al. 2023). Our own numbers price it. At 352.6 seconds per thousand documents, labelling all 40,000 is under four hours of laptop GPU, and what it buys is a student that serves roughly 165× cheaper than this zero-shot teacher and hands back the score, the ROC curve and the threshold that Section 12.8 showed prompting takes away. The labels are noisy, so the natural reference point is the teacher’s 81.56 rather than a human annotator’s. A student trained on a teacher’s labels can land below the teacher or, as Hsieh et al. (2023) report, above it; where this one lands is exactly the measurement we did not make, and it is the one we would make first.

This is consistent with the published picture rather than a contradiction of it. Edwards and Camacho-Collados found fine-tuning ahead by 0.119 micro-F1 with full training data across 16 datasets but prompting ahead outright in the one-shot regime (Edwards and Camacho-Collados 2024); Bucher and Martini put the crossover above roughly 200 training observations, though against zero-shot BART-large-MNLI rather than a modern instruction-tuned model (Bucher and Martini 2024). Our crossover is an order of magnitude higher than theirs, which is what you would expect when the prompted model is a 2026 36B mixture-of-experts rather than a 2019 entailment model. The crossover is a property of the pair you are comparing, not a constant — which is the argument for measuring it on your own task rather than citing anyone’s number, including ours.

What a document costs

Quality is where the argument starts. Cost is where it usually ends.

option                              docs/sec    seconds per 1,000 docs

tf-idf + LR, batched                  72,972              0.014
tf-idf + LR, one at a time             6,543              0.153
BERTurk fine-tuned, batched              470              2.128
llama3.2:3b  zero-shot, one at a time      8.3           121.0
qwen3.5:9b  8-shot, one at a time          3.4           289.9
qwen3.6:35b-a3b  8-shot, one at a time     3.3           301.5
qwen3.6:35b-a3b  zero-shot, one at a time  2.8           352.6
qwen3.5:9b  zero-shot, one at a time       2.7           365.0
gemma3:27b  8-shot, one at a time          1.8           559.8
gemma3:27b  zero-shot, one at a time       1.7           592.5

Read the right-hand column as a ratio and the argument stops being close. Classifying a million documents costs 14 seconds with the vectoriser, 35 minutes with the fine-tuned encoder, and three and a half days with the best prompted model. The encoder is 142× cheaper per document than the prompted model it ties with at 2,000 labels; the vectoriser is roughly 22,000× cheaper.

Two things that ratio is and is not. It is a batched encoder against a single-stream local server, which is the comparison most small teams actually face. It is not the most favourable arrangement for the prompted side: the table’s own tf-idf pair shows batching alone is worth 11× (72,972 against 6,543), and a hosted API batching properly would recover a good part of that. What batching cannot recover is the mechanism underneath. The encoder reads a document once and emits a number. The prompted model must write its answer one token at a time, and every token costs a full pass through a network thirty to 250 times larger in active parameters. Batching amortises the overhead; it does not change the arithmetic.

One row of that table should stop you: 8-shot is faster than zero-shot in all three pairs it can be compared in — 301.5 against 352.6 on qwen3.6, 289.9 against 365.0 on qwen3.5, 559.8 against 592.5 on gemma3 — despite a prompt 4.3× longer.

Part of it is straightforward. Latency tracks output tokens, generated one at a time, not prompt tokens, processed in parallel. The conf run generates 7.1 tokens against 4.1 and is duly the slowest qwen3.5 condition.

But that does not close the case. qwen3.5 zero and few both generate 4.1 tokens, so output length cannot explain this pair.

The likely remainder is that the 8-shot prefix is identical across all 2,000 requests and the server caches it. We did not measure that, so we are not claiming it — and on a laptop this noisy, a 20% gap between two runs made at different times is not something to build an argument on.

Every LLM figure is a median, and that choice is not cosmetic. The qwen3.5:9b zero-shot run has a mean of 3.371 seconds per document against a median of 0.365 — a 9× discrepancy produced by 18 documents that stalled, the worst for 980 seconds. Those stalls are a property of a laptop that decided to do something else for a while, not of the model. Quoting the mean would have inflated this model’s cost by an order of magnitude. That would have made the comparison above meaningless, and it would have been unfair to this model, not in its favour.

Note where the mixture-of-experts model lands. qwen3.6:35b-a3b has 36B parameters but activates about 3B per token, and it serves a document in 0.353s — faster than the dense 9.7B model at 0.365s and 1.7× faster than the dense 27.4B at 0.592s, while scoring higher than both. This is Section 7.7’s argument arriving as a serving bill: what you pay for at inference is active parameters, not total ones.

There is a second asymmetry, and it runs the other way. The prompted models pay per document forever, while the encoder pays once: the 2,000-label model that ties with the best prompt is 1,260 optimiser steps, about 6.5 minutes of corrected GPU time, repaid after roughly 1,300 documents.

But 6.5 minutes is the GPU bill, and it is the smaller half. The encoder also needs 2,000 labelled documents; the 8-shot prompt needs eight. This chapter cannot price your labelling — at ten seconds a document that is five and a half hours of somebody’s attention, and at a minute a document it is a working week. The GPU bill is repaid in a morning. The labels are the real cost, and they are the reason Section 12.3 calls labelling the bottleneck. What the labels buy is the right to be in this comparison at all.

The decision, on the two axes that decide it

Enough measurements. What a reader actually needs is not the scoreboard but a way to locate their own situation on it — so put your label count and your document volume in, and read off which system the numbers point to.

crossRows = [
  {name: "tf-idf + LR, threshold moved", mf1: 80.23, s1k: 0.014, kind: "trained", labels: 40000},
  {name: "BERTurk, 1,000 labels",     mf1: 72.78, s1k: 2.128, kind: "trained", labels: 1000},
  {name: "BERTurk, 2,000 labels",     mf1: 82.00, s1k: 2.128, kind: "trained", labels: 2000},
  {name: "BERTurk, 40,000 labels",    mf1: 86.81, s1k: 2.128, kind: "trained", labels: 40000},
  {name: "llama3.2:3b zero/tr",       mf1: 52.32, s1k: 121.0, kind: "llm", labels: 0},
  {name: "llama3.2:3b zero/en",       mf1: 69.21, s1k: 124.0, kind: "llm", labels: 0},
  {name: "qwen3.5:9b zero/tr",        mf1: 79.53, s1k: 365.0, kind: "llm", labels: 0},
  {name: "qwen3.5:9b few/tr",         mf1: 80.75, s1k: 289.9, kind: "llm", labels: 8},
  {name: "gemma3:27b zero/tr",        mf1: 72.28, s1k: 592.5, kind: "llm", labels: 0},
  {name: "gemma3:27b few/tr",         mf1: 79.45, s1k: 559.8, kind: "llm", labels: 8},
  {name: "qwen3.6:35b-a3b zero/tr",   mf1: 81.56, s1k: 352.6, kind: "llm", labels: 0},
  {name: "qwen3.6:35b-a3b few/tr",    mf1: 82.08, s1k: 301.5, kind: "llm", labels: 8}
]

crossPlot = Plot.plot({
  width: 700,
  height: 420,
  marginLeft: 62,
  marginBottom: 52,
  marginRight: 20,
  x: {label: "seconds to classify 1,000 documents  (log scale, lower is better)",
      type: "log", grid: true,
      ticks: [0.01, 0.1, 1, 10, 100, 1000],
      tickFormat: d => d < 1 ? String(d) : d.toLocaleString()},
  y: {label: "macro-F1 on the 2,000-document subsample", domain: [40, 95], grid: true},
  color: {domain: ["trained", "llm"],
          range: ["#1F4E79", "#B3453E"],
          legend: true,
          tickFormat: d => ({trained: "trained on labels",
                             llm: "prompted, no training"})[d]},
  marks: [
    Plot.ruleY([48.37], {stroke: "currentColor", strokeDasharray: "4 3"}),
    Plot.text([{x: 0.02, y: 48.37}], {x: "x", y: "y", text: ["always “satisfied” — 48.37"],
      dy: -8, fill: "currentColor", textAnchor: "start", fontSize: 9}),
    Plot.dot(crossRows, {x: "s1k", y: "mf1", fill: "kind", r: 6,
      title: d => `${d.name}\nmacro-F1 ${d.mf1.toFixed(2)}\n${d.s1k}s per 1,000 documents`
                + `\n${d.labels.toLocaleString()} labels used`}),
    Plot.text(crossRows, {x: "s1k", y: "mf1", text: "name", dy: -12,
      fontSize: 9, textAnchor: "middle"})
  ]
})

Down and to the right is worse on both axes at once. Everything trained sits in the left half of the chart by two to four orders of magnitude; everything prompted sits in the right. The best prompted model (82.08) and BERTurk at 2,000 labels (82.00) are at the same height and 142× apart horizontally — which is the whole argument of this section in one pair of dots. Both coordinates of every point are in the two text tables above, so the chart adds convenience rather than information.

The two models fail on different documents

Section 12.10 will cross-tabulate tf-idf against BERTurk document by document and find they miss different ones; that non-overlap is the condition under which combining two models could help. Here is the same cross-tabulation for qwen3.6:35b-a3b 8-shot against BERTurk at 40,000 labels, on the shared 2,000:

                          BERTurk right    BERTurk wrong
LLM right                     1,871              28
LLM wrong                        66              35

they agree on 95.30% of documents
an oracle picking the better of the two per document: 92.62 macro-F1
BERTurk alone:                                       86.81

The 28 documents in the top-right corner are real: they are cases the fine-tuned encoder gets wrong and the prompted model gets right. An oracle that could always pick the winner would score 92.62 against BERTurk’s 86.81 — 5.81 points on the table.

We are not shipping that, and the reason is the point of this section rather than a footnote. To capture those 28 documents you would have to send documents to the LLM, and the LLM costs 142× more per document. A router that scores everything with BERTurk and forwards even 10% of traffic to the prompted model runs at 32.3 seconds per 1,000 documents against 2.1 — 15× the serving bill to chase 28 documents in 2,000. The 5.81 points are real and the mechanism to collect them is unaffordable, which is the honest shape of most ensemble opportunities: visible in an offline cross-tabulation, deleted by the cost column.

The version that does pay is the one the confidence measurement above points at: use the cheap model’s own score, which has AUROC 0.843 against its own errors, to route the uncertain tail to a person rather than to a bigger model. That is Section 12.7’s machinery, applied to a different decision, and it needs no second model at all.

12.10 Error analysis as a discipline

Every number so far tells you how often a model is wrong. None of them tells you what it is wrong about, and that is the question that decides whether you ship. This section is the habit: after the metrics, read the mistakes.

We compare the two trained models at the operating points already chosen — tf-idf at \(t = 0.166\) on the complaint side, and BERTurk at 40,000 labels read at \(0.5\). On the 10,000-document test set that is 373 complaints caught out of 632 for tf-idf against 452 for BERTurk, at 57.38% and 72.32% precision.

The 2×2 nobody builds, and should

At these two operating points the aggregate delta is 7.35 macro-F1 points — 85.02 for BERTurk against 77.66 for the re-thresholded vectoriser — and that single number hides a trade. The only way to see the trade is to cross-tabulate the two models document by document.

Replacing a model is a trade, not an upgrade

The 7.35-point headline is a net figure. Switching models fixes 332 documents and breaks 149 — a real gain, but 45% of the gross gain is handed back (149/332), leaving a net 183. If those 149 happen to be your highest-value cases, the aggregate number is lying to you.

Three things fall out of that grid that the leaderboard cannot show you.

The gain is smaller than it looks, and it is a trade. 332 documents get fixed; 149 that tf-idf handled break. Nobody who reports a 7.35-point improvement is lying, but nobody who reports it is telling you that they broke 149 documents either. If your complaints route to a human queue and 149 of them were the ones that queue was tuned around, the “upgrade” is a regression for that team.

There is a hard floor, and it is high. 204 documents defeat both models, and 137 of them are complaints — 21.7% of every complaint in the test set is missed by a sparse linear model and a fine-tuned transformer alike. That is not a capacity problem you fix with a bigger encoder. It is either genuinely ambiguous text or a broken label, and the next two subsections argue it is mostly the second.

The two models fail on different things. 685 documents are missed by at least one of the two models, and only 204 of those are missed by both; on the other 481 the models disagree. That non-overlap is exactly the condition under which an ensemble helps — and, more usefully here, it is the condition under which reading the disagreements teaches you something, because each model’s errors have a shape.

The disagreement 2×2 costs one numpy line and takes five seconds. Build it every single time you replace a model in production. The version that matters most is not this one but the one restricted to the class you care about: 122 complaints gained, 43 lost.

Slice before you conclude

The second cheap habit: split the test set on anything you can measure without labelling, and check whether the aggregate holds inside each slice. Document length is free.

length (chars)      n    complaints   tf-idf err   BERTurk err   ratio
<50             1,644         6.14%        5.29%         4.50%    1.18
50–99           2,253         6.39%        5.77%         3.99%    1.44
100–199         3,499         6.09%        4.86%         3.12%    1.56
200–399         2,096         7.16%        5.82%         3.10%    1.88
400+              508         4.72%        5.31%         2.95%    1.80

tf-idf’s error rate is flat. It moves between 4.86% and 5.82% with no trend — a longer review gives a bag-of-words model more words, but it also gives it more chances for a cheerful word to outvote the complaint. BERTurk’s error rate falls almost monotonically, from 4.50% on reviews under 50 characters to 3.10% on reviews of 200–399. The advantage is 1.18× on the shortest documents and 1.88× on the second-longest bucket.

That is the encoder’s actual selling point stated precisely: it is not uniformly better, it is better at long documents, because that is where negation, concession and scope live and where a bag of words has nothing to say about any of them. If your production traffic is mostly one-line reviews, the 1.18× is the number you should be budgeting against, not the 1.88×.

Read the 400+ row with suspicion. It is 508 documents holding roughly 24 complaints, so BERTurk’s 2.95% is about 15 errors. One document moves it by 0.2 points. The trend across the first four buckets is the finding; the last row is consistent with it, not evidence for it.

Sort the errors by confidence, not at random

Reading a random sample of errors mostly teaches you that hard cases are hard. Reading the errors the model was most confident about is where the information is, because a model that is confidently wrong is usually being punished for being right.

Here are BERTurk’s most confident mistakes, glossed into English. The probabilities are the model’s, on the complaint side.

gold        p(complaint)   review (glossed)
complaint         0.0011   "I chose it because the price was right, but I was
                            very pleased once I used it … definitely worth it."
complaint         0.0012   "Thank you so much, prettier than the picture, I
                            loved it, an anniversary gift for my wife :-)"
satisfied         0.9988   "It broke within two months and became unusable
                            (I'm giving five stars so my review shows up at the
                            top, normally it doesn't deserve a star)"
satisfied         0.9986   "The suction was weak and it burned out after 5–6
                            months."
satisfied         0.9986   "Friends, it does nothing for belly fat, that's
                            certain … so if you're buying it for your belly,
                            don't."
satisfied         0.9988   "You'll regret it if you don't buy it."
satisfied         0.9987   "A so-so product."

The third row is the one that needs no adjudication, and it is worth dwelling on. The reviewer states, in the review, that the star rating is strategic rather than evaluative — five stars so the complaint floats to the top of the listing. The dataset card does not document how the labels were derived (Chapter 4), but this row is strong evidence that they come from star ratings. So the label is a faithful record of the stars and a false record of the sentiment, and the reviewer has helpfully documented the exact mechanism by which our ground truth fails.

The two rows below it read the same way to us: a vacuum that burned out, and a review that says the product “does nothing” and ends don’t buy it — both filed as satisfied. The two rows above it go the other direction: two glowing reviews filed as complaints. We are stating that as our reading, not as a measurement: no independent Turkish-speaking annotator has re-adjudicated these, so we will not put a number on how many of the 204 shared failures are label errors. What we will say is that the top of the confidence-sorted error list is dominated by cases where the model looks more right than the label, which is exactly the diagnostic Frénay and Verleysen (2014) describe and exactly what Section 5.6 told you to go looking for.

The last two rows are instructive in a different way. “You’ll regret it if you don’t buy it” is a genuine model error: it contains every lexical marker of a complaint — regret, negation, a warning — arranged to mean the opposite. “A so-so product” is neither a model error nor a clear label error. It is a shrug, and a shrug next to a five-star rating is a labelling convention, not a language problem.

TipThe rule this produces

Sort your errors by model confidence, descending, and read the first fifty by hand. If most of them look like label errors, stop tuning the model — you are now measuring your annotation process, and no architecture change will move the number. Re-label instead, or change the labelling function.

What the encoder buys, in sentences

The 122 complaints BERTurk catches that tf-idf misses have a shape, and it is not vocabulary. Below are examples where tf-idf scored just under its 0.166 threshold (0.13–0.15) and BERTurk scored above 0.9:

p_lin   p_bert   review (glossed)
0.135    0.9987  "After one month's use it snapped in the middle."
0.143    0.9976  "The reception isn't anything to write home about,
                  normal for the price."
0.152    0.9801  "The numbers on the parts are printed wrong; assembling
                  it into a whole is exhausting."
0.143    0.9976  "I put it downstairs in my duplex and it doesn't reach
                  upstairs; upstairs and it doesn't reach downstairs.
                  Very weak … I read the reviews and can't believe it's
                  the same modem."

None of these contain the stock complaint words a bag of words weights heavily. The first carries its entire meaning in one rare verb (koptu, “snapped”). The second is a concessive: the complaint is in the first clause and the second clause partly retracts it, which is a structure tf-idf cannot represent because it has thrown away the order. The fourth is a complaint expressed as a narrative: the one evaluative phrase (“very weak”) comes only after a description of the product failing to do its job twice. Sparse features need the word; the encoder needs the sentence.

And the errors both models share, similarly glossed, fall into four kinds, the largest being the lukewarm family:

"It drops out now and then, but it'll do."          lukewarm
"It'll do, for the money it's fine."                lukewarm
"A so-so product."                                  lukewarm
"You'll regret it if you don't buy it."             negated regret
"The hose looks low quality."                       hedged
"Downside:"                                         empty

The lukewarm family is the interesting one, because it is not really a modelling failure. İdare eder — “it’ll do” — is a shrug that Turkish reviewers routinely pair with four or five stars, so the label says satisfied and the text says neither. No classifier can recover a distinction the labels do not encode. The last row is worse: a review whose entire text is the word downside followed by a colon. It has a gold label. It contains no information. Some fraction of every test set is unlabelable, and it sets a ceiling you cannot see from the metrics.

The four habits

Everything above is four cheap things done in order, and they generalise to any classifier you will ever ship:

1. Cross-tabulate against the model you are replacing. An aggregate improvement is a net figure. Find out what you broke.
2. Slice on something free. Length, source, date, product category, document language. Check that the win survives inside each slice before you claim it.
3. Sort errors by confidence and read the top of the list. Confident mistakes concentrate label errors; random samples concentrate genuine ambiguity.
4. Separate your errors from the dataset’s. Ask of each mistake: could a careful human have got this right from the text alone? If the answer is no, it is not a model problem, and tuning will not fix it.

12.11 Intuition checks

Answer each one before you open it. All six are questions this chapter settled with a measurement rather than an argument, so the answers carry numbers you can check.

Note1. A colleague reports 94.3% accuracy on this corpus. How impressed should you be?

Not at all, and you can say so before seeing anything else. A model that answers “satisfied” to every input scores 93.68% here, because that is the base rate. 94.3% is 0.6 points above a model with no features, no training and no parameters.

The uncorrected baseline in Section 12.4 scores exactly 94.28% at \(t = 0.5\) — and catches 69 of 632 complaints. It is not a slightly-better- than-nothing model; it is a nearly-nothing model wearing a respectable number.

The habit worth forming: before you look at any accuracy figure, compute the majority-class rate and subtract. If the gap is small, ask for a different metric. Matthews correlation is the cheapest honest replacement — it gives the constant predictors exactly 0.000 and the random guessers values within 0.02 of zero.

Note2. Your macro-F1 goes from 58.24 to 77.66 and the only change is one constant. Did you cheat?

No — provided you can show the ranking did not move, and provided the constant was chosen on held-out data.

The proof is in Section 12.5: average precision is 60.30 and AUC is 93.24 both before and after. Those two statistics depend only on the order the model puts documents in. They come out identical to four significant figures because the model is byte-for-byte the same, and a ranking statistic that did not move is the evidence you can show someone else; all that changed is where we cut.

The cheat would be picking 0.166 by sweeping thresholds on the test set. That is why every threshold in this chapter is chosen on the 8,000-document cal slice and only then applied to test. The chapter does report test-swept thresholds — labelled oracle — but only as an upper bound to measure regret against, never as a result.

Note3. You have 800 labels and two days. Fine-tune BERTurk or fit tf-idf?

At 800 labels this is genuinely close, and the honest answer is fit tf-idf first, today, and then decide.

The measured curves bracket you. At 500 labels tf-idf scores 60.82 and BERTurk 56.46 — the sparse model wins. At 1,000 labels tf-idf scores 61.60 and BERTurk 71.57 — the encoder wins by ten points. Your 800 sits inside a crossover that is steep on the encoder side and flat on the sparse side.

What tips it is the cost asymmetry, not the accuracy. The tf-idf fit takes about a tenth of a second. If it lands where you need, you are finished before lunch. If it does not, you have lost nothing and you now have a floor to beat and a labelled set already loaded. Run the cheap thing first not because it usually wins, but because finding out costs nothing.

Now notice the option the question does not offer you. Section 12.9 measured a third answer near this budget, and it needs its own evaluation set to be quoted honestly: on the shared 2,000-document subsample a prompted 36B model scores 81.56 macro-F1 with zero labels, against BERTurk’s 72.78 at 1,000 labels on those same documents — the paired bootstrap puts that gap at 8.79 points in the prompt’s favour. Fit tf-idf anyway; it is a tenth of a second and it is your floor. But “which of these two do I train” stopped being the whole question, and what should settle it is the column the original answer never had: the prompt costs 352 seconds per 1,000 documents against the vectoriser’s 0.014, and it gives you no threshold to move.

Note4. Adding 500 labels made AUC go down. Should you roll back?

Look at what else moved first. Going from 500 to 1,000 labels, BERTurk’s AUC falls 83.00 → 82.76, and over the same step average precision rises 26.98 → 45.72 and macro-F1 rises 56.46 → 71.57.

That is not a contradiction, it is the two metrics measuring different regions. The false-positive rate divides by 9,368 satisfied reviews, so mistakes among the easy majority barely register; precision divides by however many documents you actually flagged, which is a much smaller number (Davis and Goadrich 2006). The extra labels improved the model exactly where the decision happens and slightly disturbed a part of the ranking nobody operates in.

Roll back only if your product lives at low thresholds across the whole ranking. Ours does not — nobody reads all 10,000 reviews — so ship it. The general rule: on an imbalanced problem, quote average precision against the base rate (here 60.30 against 6.32%, a 9.54× lift) and treat AUC as secondary.

Note5. Class weighting raised your F1. What did it cost?

Its calibration, and probably more than you wanted to pay.

class_weight="balanced" takes macro-F1 from 58.24 to 76.09 and takes expected calibration error from 3.96 to 14.18 — a 3.6× increase. The reweighted model’s mean predicted probability is 0.795 against a true rate of 0.9368. It has learned a prior that is not the prior of the world it will run in.

If the only thing you ever do with the output is argmax, this is free. It stops being free the moment the number is displayed to a human (“87% likely a complaint”), multiplied by a cost, fed to a downstream rule, or compared across model versions. And note that you did not have to pay it at all: moving the threshold on the unweighted model reaches 77.66 — better than class weighting — with ECE untouched at 3.96.

Van den Goorbergh and colleagues found the same trade on clinical risk models and argued it is rarely worth making (van den Goorbergh et al. 2022).

Note6. You applied the textbook cost-optimal threshold and the review queue exploded. What went wrong?

The formula is almost certainly fine. The input to it was not a probability.

Elkan’s \(q^* = C_{FP}/(C_{FN}+C_{FP})\) is correct, and it assumes \(q\) is a calibrated probability (Elkan 2001). At a cost ratio of 100 that threshold is 0.0099. Feed it raw logistic-regression scores and the model flags 8,073 of 9,368 satisfied reviews, because its scores are compressed into a band where 0.0099 catches nearly everything. Feed it isotonic-calibrated scores from the same model and it flags 3,345.

Across the seven cost ratios we measured, the formula on raw scores overspends by 8% to 56%; on calibrated scores, by 1% to 7%. The diagnostic is a reliability plot, not a code review — the bug is in the numbers, not the arithmetic.

12.12 Interview questions

On attribution — the same caution as Chapter 2. These are questions that recur across the two credible public sources, both free: Introduction to Machine Learning Interviews by Chip Huyen (Huyen 2021) and AIMLInterviews. Be wary of pages that list “the questions company X asks” without a source; attributing a question to a company takes a source you can check.

1. How do you handle class imbalance? The expected answer lists oversampling, undersampling, SMOTE and class weights. The better answer starts by asking what breaks, because the answer is usually the threshold, not the model. Give the measurement: on a 6.32% minority class, moving one threshold reached 77.66 macro-F1 against 76.09 for class weights, 77.63 for oversampling, 74.72 for SMOTE and 62.98 for undersampling — with no refit and no extra rows. Then give the caveat, because it is the part that shows you understand the design: only the threshold arm was tuned, and average precision says a tuned oversampling arm would probably edge ahead. The finding is that the threshold is free, not that it is best. Then note that SMOTE was evaluated on nine low-dimensional dense numeric datasets and that its authors listed text as future work (Chawla et al. 2002) — that is the sentence that shows you read the paper rather than the blog post about it.

2. Precision, recall, F1, ROC-AUC, PR-AUC — when do you use which? Say what each divides by. FPR divides by the majority count; precision divides by what you flagged. On an imbalanced problem those denominators can differ by up to a couple of orders of magnitude, so the two curves rank models differently (Davis and Goadrich 2006) — ours disagreed in direction between two BERTurk checkpoints. Finish with the practical rule: quote PR-AUC against the base rate so the reader can see the lift, and never quote accuracy without the majority-class rate beside it.

3. What is calibration, and how would you test it? A model is calibrated when among the cases it scores 0.8, about 80% really are positive. Test it with a reliability diagram and an expected calibration error, and then — this is the part most candidates miss — say which ECE. Guo et al.’s version bins the top-label confidence \(\max_k p_k\) and asks how often the argmax is right (Guo et al. 2017); binning on \(P(\text{class }1)\) is a different statistic. On our class-weighted model they read 8.98 and 14.18. Both are correct; a number without its definition is not.

4. Platt scaling or isotonic regression? Isotonic is non-parametric and can fit any monotone distortion; Platt is a two-parameter sigmoid and cannot (Platt 1999; Zadrozny and Elkan 2002). The decision is about how much held-out data you have: Niculescu-Mizil and Caruana found Platt ahead below roughly 200–1,000 calibration points and isotonic level or better above that (Niculescu-Mizil and Caruana 2005). Name the size of your calibration set. Ours is 8,000, and isotonic came out at 0.69 against Platt’s 0.94 — a gap inside our own estimator’s noise, so what we can say is that both are adequate at this calibration-set size, which is what their result predicts. Saying “isotonic won” would be reading a quarter of a point off a statistic whose floor is half a point.

5. How would you pick the decision threshold for a production classifier? Never 0.5 by default. Get a cost ratio from whoever owns the consequences, derive \(q^*\) from it, and — the step that decides whether this works — verify the scores are calibrated before you trust the derivation. If the costs are fixed and you only need one operating point, sweeping the threshold directly on held-out data is a legitimate shortcut; in our measurements its regret stayed within two percentage points of the calibrate-then-derive route. Calibration earns its keep when the costs might change or the number gets shown to a person.

6. When would you not use an LLM for classification? When you have labels, when you have volume, and when you need a number rather than a word. Give the curve, not a verdict: on our task a prompted 36B mixture-of-experts beat a fine-tuned BERTurk by 8.79 macro-F1 at 1,000 labels, tied it to 0.08 points at 2,000 (\(p = 0.97\)), and lost by 4.73 at 40,000 — all paired- bootstrapped on the same 2,000 documents. Then price the tie: the encoder classifies a document 142× cheaper, so a dead heat on quality is not a dead heat. Say which arrangement that number belongs to: a batched encoder against a single-stream 4-bit local server, which is the comparison a small team actually faces. A properly batched hosted API narrows the gap without changing the per-token arithmetic underneath. The part most candidates miss is the third axis. A zero-shot model returns the word ŞİKAYET, so you give up the ROC curve, the calibration map, the cost-derived threshold and the sweep — every instrument in this chapter. Our best prompted run sat at 56.98% precision and 80.95% recall and there was nothing to turn. Close by naming the literature and its limits: Edwards and Camacho-Collados (2024) found fine-tuning ahead by 0.119 micro-F1 with full data across 16 datasets but prompting ahead in the one-shot regime, and Bucher and Martini (2024) put the crossover above roughly 200 observations, measured against zero-shot BART-large-MNLI rather than a modern instruction-tuned model. Ours is an order of magnitude higher than theirs, which is the point — the crossover is a property of the pair you are comparing, not a constant.

7. Your model is 94% accurate offline and users say it misses everything. Debug it. Almost always the threshold, the base rate, or a distribution shift, in that order of likelihood. Check the confusion matrix, not the scalar: ours at 94.28% accuracy was catching 69 of 632 complaints. Then check whether the offline base rate matches production — a model calibrated on 6.32% complaints will systematically under-flag on a stream that runs at 15%. Then check for leakage into the offline number, and specifically whether any threshold or calibrator was fitted on the evaluation set.

What carries over to more than two classes

Everything in this chapter is binary, and that is a choice rather than an oversight: imbalance, thresholds and costs are clearest with one cut. Say which parts move.

Three things carry over unchanged: macro-F1, the majority-class floor, and reading your errors sorted by confidence. Two change shape. Single-label multi-class has no per-class threshold to tune — the classes compete, so under equal error costs the decision is an argmax over calibrated probabilities, and Elkan’s formula becomes an argmin over a full \(K \times K\) cost matrix, \(\hat{y} = \arg\min_i \sum_j p_j C_{ij}\). Per-class thresholds belong to the multi-label case below, where the classes do not compete. One threshold does survive into multi-class, and it is worth building in: a rejection threshold on \(\max_k p_k\) or on the margin between the top two, sending anything below it to a human instead of to a label. Calibration changes most: Platt and isotonic are binary maps, so multi-class needs either one-vs-rest calibration or a matrix- or vector-scaling variant — and this is the setting where the top-label ECE of Guo et al. (2017) finally becomes the natural statistic rather than a simplification. Multi-label is a third case again: \(K\) independent binary problems with \(K\) independent thresholds, and a macro-F1 averaged over labels whose base rates may differ by two orders of magnitude.

12.13 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below. Pinning versions in a book is a hostage to fortune, but not pinning them is worse — the numbers here are only reproducible against a stated stack.

The stack these numbers came from. Python 3.14.5, Apple Silicon

library	version used	what it is for here
scikit-learn	1.9.0	TfidfVectorizer, LogisticRegression, IsotonicRegression, every metric and curve
transformers	5.15.0	loading and fine-tuning BERTurk
torch	2.13.0	the MPS backend the fine-tuning ran on
imbalanced-learn	0.14.2	RandomOverSampler, RandomUnderSampler, SMOTE
scipy	1.18.0	the bounded scalar minimiser that fits the temperature
numpy / pandas	2.5.2 / 3.0.5	array work, Parquet round-trip, the split
ollama	0.24.0	serving the four local LLMs in Section 12.8 — llama3.2:3b, qwen3.5:9b, gemma3:27b, qwen3.6:35b-a3b, all Q4_K_M quantised

Worth knowing about, not used above:

The wider ecosystem, versions as of writing

library	current version	when you would reach for it
setfit	1.1.3	the contrastive few-shot recipe of Tunstall et al. (2022) as one object. The right first try when you have tens of labels per class rather than hundreds
ModernBERT and its multilingual descendants	2025–2026	the encoders to try before BERTurk on a new project (Warner et al. 2025); ModernBERT itself is English and code, so for Turkish use a multilingual descendant such as mmBERT — same fine-tuning recipe, rotary embeddings, an 8,192-token context and markedly faster training
sentence-transformers	5.7.0	frozen embeddings plus a linear head — often within a point or two of full fine-tuning at a fraction of the training cost
peft / accelerate	0.20.0 / 1.14.0	LoRA and distributed training when the encoder no longer fits the way it did here (Chapter 9)
netcal	1.4.0	calibration methods and diagrams beyond what scikit-learn ships, including binning and beta calibration
mapie	1.5.0	conformal prediction — instead of a calibrated probability, a set of labels with a guaranteed coverage rate. The right tool when “I don’t know” must be a valid output
cleanlab	2.9.0	finds label errors in your training data. On a corpus whose labels came from star ratings, this is usually the highest-return hour you can spend
optuna	4.9.0	hyper-parameter search when the sweep stops fitting in a for loop
onnxruntime	1.28.0	exporting a fine-tuned encoder for CPU serving, typically 2–4× faster than PyTorch eager mode at inference
skops	0.14.0	persisting scikit-learn models without pickle, which you should not be deserialising from anywhere you do not control
shap	0.52.0	per-prediction attributions when somebody asks why — see Section 12.10 for the cheaper alternative

Two omissions are deliberate. xgboost (3.4.1) and lightgbm (4.7.0) are excellent on tabular data and rarely competitive on high-dimensional sparse text, where a linear model with an L2 penalty is already close to the right inductive bias. And fasttext (0.9.3) remains a genuinely fast supervised baseline, but it is effectively unmaintained; scikit-learn gets you to the same place on a corpus this size.

12.14 Lab: ship a classifier with a real serving budget

The deliverable is one paragraph of the form “we flag X% of incoming documents, catch Y% of the Z, at a cost of W milliseconds per document — and here is the threshold we chose and why”. Every clause has to be defensible from a measurement you made.

What you need

A corpus of a few thousand documents with a binary label you did not create by hand — star ratings, resolved/unresolved tickets, spam flags, moderation outcomes. It must be imbalanced; if your minority class is over 30%, subsample it until it is not, because the whole discipline of this chapter only matters when the classes are lopsided. scikit-learn alone is enough for steps 1 through 5.

Step 1 — write down the floor before you train anything

Report the majority-class rate, and the accuracy and macro-F1 of three trivial policies: all-majority, all-minority, and a coin flip. This takes four lines and it is the single most useful table in your report, because every later number gets read against it. Ours were 93.68 / 6.32 / 50.21 accuracy, and 48.37 / 5.94 / 38.57 macro-F1.

Step 2 — split three ways, not two

fit, cal, test — 64 / 16 / 20 is a reasonable default and is what this chapter used. Write down, before you continue, which quantities are allowed to touch cal: the threshold, the calibrator, the cost-optimal operating point. Nothing may touch test until step 7.

If you cannot spare a calibration slice, say so explicitly and use cross-validated calibration instead — but do not quietly fit the calibrator on training data. The measured penalty for doing that here was ECE 2.39 against 0.69, and the reason it is dangerous is that 2.39 does not look wrong.

Step 3 — build the sparse baseline and the label-budget curve

tf-idf over 1–2 grams into logistic regression. Then refit at 200, 500, 1,000, 2,000 and all of your labels, and plot macro-F1 against budget. You are looking for two things: whether the curve has flattened (ours had not, even at 40,000), and how much of the gain is in the minority class (ours: 20.75 points of macro-F1 against 41.84 of minority F1).

Step 4 — sweep the threshold on cal and report the table

For \(t\) from 0.05 to 0.95, record accuracy, precision, recall, minority F1, macro-F1 and MCC. Two questions to answer from the table: how much did accuracy move over the range \(t \ge 0.2\), and is the highest-accuracy threshold the same as the highest-macro-F1 one? On this corpus (read on test, for illustration) the answers were 1.3 points and no. (Sweep below 0.2 as well and accuracy collapses to 68.98 — which is why the interesting range is the one where accuracy looks flat.)

Step 5 — check whether your scores are probabilities

Bin the predicted scores, plot observed frequency against mean predicted probability, and compute ECE and MCE. State which ECE you computed. Then fit Platt and isotonic on cal and re-measure. If your ECE is already under about 1 you can stop; if it is over 5, anything downstream that treats the score as a probability is wrong by a margin you can now quantify.

Step 6 — turn a cost into a threshold

Ask someone who owns the consequences for a ratio \(r = C_{FN}/C_{FP}\). Do not accept “they’re both bad”; make them choose a number, even a rough one, because the alternative is that the number gets chosen by whoever wrote 0.5. Compute \(q^* = 1/(1+r)\), apply it to your calibrated scores, and report total cost against the two trivial policies. If your model cannot beat flag everything, that is a publishable finding about your project.

Step 7 — open test once

Run every chosen threshold and calibrator, produce the confusion matrix, and write the paragraph. Then run one more thing: time it. Documents per second, single process, at the batch size you would actually serve. A model you cannot afford to run is not a model, and this is the number people forget to measure until the week they deploy.

What to hand in

The trivial-baseline table, the label-budget curve, the threshold sweep, the reliability plot before and after calibration, the cost table, and the paragraph. Roughly two pages. If any number in the paragraph does not appear in one of the tables, delete the number.

12.15 Exercises

1. Reproduce the floor. On any imbalanced corpus you have, compute the accuracy of a classifier that always predicts the majority class. Then find a published result on that corpus and check how far above the floor it sits. Report both numbers together.

2. Break a threshold. Take a trained binary classifier and sweep its threshold from 0.05 to 0.95. Plot accuracy and minority-class recall on the same axes. Report the threshold that maximises accuracy and the threshold that maximises macro-F1, and explain in one sentence why they differ.

3. Prove the ranking did not move. Repeat the experiment in Section 12.5 on your own data: compare “no correction plus a tuned threshold” against class_weight="balanced". Report macro-F1 and average precision for both. Explain what it would mean if the average precisions were equal, and what it would mean if they were not.

4. Cost the mistakes. Build a cost matrix for a task where a false negative costs 10× a false positive. Derive \(q^*\), apply it to raw and to calibrated scores, and report the total cost of each against the cost of flagging everything. State how much calibration was worth in your units.

5. Break the calibrator on purpose. Fit an isotonic calibrator on your training scores instead of held-out ones, and measure ECE both ways. The point of the exercise is to see how unremarkable the leaked number looks.

6. Find the crossover. Fine-tune an encoder at 200, 500, 1,000 and 2,000 labels and plot it against your sparse baseline on the same axes. Report the budget at which the curves cross, and the wall-clock cost of each point. Compare with our crossover and account for any difference — corpus size, language, document length, or class balance.

7. Read fifty errors. Sample fifty documents your best model got wrong, stratified by confidence, and label the failure mode of each by hand. Report the three most common modes with counts. If the largest category is “the label is wrong”, you have learned something more valuable than any architecture change.

8. Re-measure the folklore. Take one local model and run zero-shot classification on 500 documents twice: once with the instruction in your corpus’s language, once in English. Report the difference with a paired bootstrap over the same documents, not two separate intervals. Then repeat with a model at least 3× larger. State whether the advice “write prompts in English” survives on either.

9. Check the parse rate before you check the score. Instrument a zero-shot run to record, per document, the raw response, the finish reason, and whether your parser succeeded. Now cap the output at 8 tokens and re-run. Report the accuracy and the parse rate, and say what the accuracy would have looked like if you had logged only the accuracy. Use a model that has a reasoning or thinking mode — the failure needs one, which is itself the lesson.

10. Price the tie. Suppose a prompted model and a fine-tuned one score within noise of each other on your task. Measure seconds per 1,000 documents for both, at the batch size you would actually serve. Compute the volume at which the fine-tune’s one-off training cost is repaid, and the monthly bill for each at your real traffic. Then say which you would ship and why — the answer is allowed to be the slower one. State whether your repayment volume counts the cost of producing the labels. If it does not, say so in the same sentence as the number.

12.16 Where to go deeper

One thing first: plot your classifier’s predicted probabilities as a histogram. If they pile up at 0 and 1, that is a reason to check calibration against observed frequencies rather than a verdict on its own — a genuinely easy problem produces confident probabilities that are also correct. Make the reliability diagram, and if the confidence is not earned, Section 12.6 is the section you need rather than a bigger encoder.

Beyond the papers

resource	what it is
scikit-learn calibration guide	reliability diagrams and both calibrators, with runnable examples
Hugging Face LLM course	the fine-tuning chapters, if you want the encoder path spelled out end to end
setfit	few-shot classification without a prompt, and the baseline to beat before you reach for an LLM
Chip Huyen’s blog	the applied-ML framing of when a classifier is the right product decision at all
StatQuest — Josh Starmer	the reference channel for the statistics under this chapter — ROC, precision-recall, and calibration explained slowly and correctly

The primary sources

topic	source
modern neural nets are badly calibrated	Guo et al. (2017)
the two classical fixes	Platt (1999); Zadrozny and Elkan (2002)
and what they cost you	Niculescu-Mizil and Caruana (2005)
thresholds from costs, not from 0.5	Elkan (2001)
why resampling is not free	van den Goorbergh et al. (2022); Chawla et al. (2002)
PR curves when the classes are skewed	Davis and Goadrich (2006)
label noise, which is usually the real ceiling	Frénay and Verleysen (2014)
prompted versus fine-tuned, measured	Bucher and Martini (2024); Edwards and Camacho-Collados (2024)
few-shot without prompting	Tunstall et al. (2022)
what a model’s stated confidence is worth	Xiong et al. (2024); Lin et al. (2022)
prompt-order effects on classification	Zhao et al. (2021)
the Turkish encoder used throughout	Schweter (2020)

12.17 What to remember

• Compute the floor before you compute anything else. This corpus is 93.68% satisfied, so a model with no features, no parameters and no training scores 93.68% accuracy and 48.37 macro-F1. Every accuracy figure in a classification paper should be read as a distance from that number, and on imbalanced data the distance is usually small enough to be embarrassing.
• A threshold is a free 19 points. The identical fitted model — the same coefficients, byte for byte — goes from 58.24 to 77.66 macro-F1 because we flagged at 0.166 instead of 0.5. Average precision stayed at 60.30 and AUC at 93.24 through the change, which is the proof that nothing about the model moved.
• Try the threshold before you touch the data. Across five imbalance treatments, moving the cut reached 77.66 against oversampling’s 77.63, class weighting’s 76.09 and SMOTE’s 74.72 — at zero training cost, and with only the threshold arm tuned, so read it as the same operating point for free rather than as a win. Undersampling actively destroyed the model (62.98, average precision down from 60.30 to 52.49 because 87.4% of the training rows were thrown away). Elkan (2001), who is routinely cited in support of rebalancing, argues the opposite: learn from the data as given, then compute the decision.
• Resampling is not free even when it works. van den Goorbergh et al. (2022) found that imbalance corrections damage calibration while doing little for discrimination, and we reproduced it on text: class weighting took ECE from 3.96 to 14.18 while gaining nothing a threshold could not have given.
• The number your classifier prints is not a probability until you check. Fit the calibrator on data the model has never seen. With 8,000 held-out documents isotonic regression and Platt scaling came out level (ECE 0.69 against 0.94 — a gap inside the noise floor computed above), which is what Niculescu-Mizil and Caruana (2005) predicts above roughly 1,000 calibration points: there, isotonic is safe. Below that, Platt scaling is the safer choice. Fitting isotonic on the training scores instead gave 2.39: worse, but not obviously wrong, which is precisely what makes the mistake dangerous.
• There are two ECEs and they are not comparable. Guo et al. (2017) bin on top-label confidence \(\max_k p_k\), which in a binary problem lives in \([0.5, 1]\); binning on \(P(\text{class }1)\) is a different statistic. Say which one you computed.
• A cost matrix picks the threshold for you. \(q^* = C_{FP} / (C_{FN} + C_{FP})\). Sweeping directly for the cost-minimising cut on held-out data did about as well as calibrating and applying the formula (1–6% regret against 1–7%), so if you need one operating point for one fixed cost ratio you can skip calibration. Calibrate when the costs might change, when a person reads the number, or when something downstream multiplies it.
• The encoder crossover is abrupt and it is early. BERTurk loses to tf-idf at 200 and 500 labels, crosses between 500 and 1,000, and by 2,000 labels scores 79.61 — beating tf-idf trained on the full 40,000 (76.21). Twenty times fewer labels for a better model. At equal budget the gap is 8.81 macro-F1 and 15.45 complaint-F1.
• Price that gap honestly. The encoder costs roughly 970× more to train for those 8.81 points (1,162 GPU-seconds against the vectoriser’s 1.2), and serves about 470 documents per second against the vectoriser’s tens of thousands. Whether that is worth it is a budget question, not a modelling one, and you cannot answer it without both numbers.
• The prompted-versus-trained crossover is a curve, and on this task it sits at ~2,000 labels. At 1,000 labels the prompted model wins by 8.79 macro-F1 points (and by more below that); at 2,000 the two are a dead heat (0.08 points, \(p = 0.97\)); at 40,000 the encoder wins by 4.73. Measure it on your own pair rather than citing anyone’s number, because the crossover is a property of which two models you put on the axis.
• Compare two models with a paired test, not by eyeballing two intervals. Models scored on the same documents share most of their sampling noise, so overlapping marginal intervals prove nothing. Every comparison above is 10,000 paired bootstrap resamples of the difference.
• LLM folklore is true for weak models and small at best for strong ones. Writing the instruction in English was worth +16.90 macro-F1 on a 3B model and +0.54 (95% interval up to +2.25) on a 9B one; eight examples were worth +7.17 on gemma3:27b and +0.51 (up to +2.48) on qwen3.6:35b-a3b. Quote the intervals rather than the p-values: what the strong-model runs establish is a ceiling of two-and-a-half points, not a zero. Both rules were real once. Re-measure them on the model you are shipping.
• Prompting costs you the threshold. Every prompted run here was recall-biased on the minority class — the best reached 56.98% precision at 80.95% recall against BERTurk’s 74.42/76.19 — and none of it is fixable, because a zero-shot model returns a word. No ROC, no isotonic map, no \(q^*\), no sweep. That is a larger loss than the macro-F1 gap.
• A stated confidence can be right on average and still separate nothing. qwen3.5:9b averaged 93.53% stated against 92.95% actual — a +0.58 gap, and an ECE of 1.80 over its nine self-chosen bins — while putting 1,549 of the 2,000 documents on the single value 95. As an error detector it scores AUROC 0.667 against the encoder’s own 0.843, and the ≥95 routing cut that looks useful was read off the evaluation set, so treat it as an oracle until you re-pick it on held-out data.
• The ensemble opportunity is real and usually unaffordable. The best LLM and BERTurk disagree on 94 of 2,000 documents, and an oracle picking the winner each time would gain 5.81 macro-F1 points. Collecting them means paying 142× more per document in the arrangement we measured. Cross-tabulate to find the opportunity; price it before you believe in it.
• Replacing a model is a trade. BERTurk fixes 332 test documents and breaks 149 that tf-idf handled — a net 183, or 1.83 accuracy points. Build the disagreement 2×2 every time you swap a model, because the aggregate delta hides which documents you gave up.
• The encoder’s advantage is concentrated where you would expect. tf-idf’s error rate is flat across document length (4.86%–5.82%, no trend); BERTurk’s falls from 4.50% on the shortest reviews to 3.10% on 200–399 characters. It is not uniformly better — it is better at long documents, because that is where negation, concession and scope live.
• Sort your errors by confidence, not at random. The top of that list is where label errors concentrate. Ours includes a review that says the product broke in two months and explains that it is giving five stars so the complaint appears at the top of the listing — labelled satisfied, because the labels evidently come from star ratings. When the model looks more right than the label, stop tuning and go re-read Section 5.6; Frénay and Verleysen (2014) is the survey.
• Some of your test set is unlabelable. 204 documents defeat both models and 137 of them are complaints — 21.7% of every complaint in the test set. One of them is a review whose entire text is the word “downside” followed by a colon. No architecture reaches those, and their existence caps the score you can honestly chase.
• Suspect your harness before your model — especially when the number is good. In our timing sweep one inference run came back 27× slower than the other seven and one training epoch was 14× slower than its two siblings. Neither was a property of the model. But the expensive failure ran the other way: an LLM scored 93.75% accuracy and 48.39 macro-F1, which is the majority-class baseline, because a reasoning model spent its 24-token budget inside a thinking field the parser never read, and every unparseable answer fell through to the majority label. A bad number gets investigated; a good one gets published. Design against that asymmetry — never impute a label on a parse failure, abort on a low parse rate, smoke-test before the overnight queue — because your own scepticism will not fire when the result is what you hoped for.

Bucher, Martin Juan José, and Marco Martini. 2024. Fine-Tuned “Small” LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification. https://arxiv.org/abs/2406.08660.

Chawla, Nitesh V., Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. “SMOTE: Synthetic Minority over-Sampling Technique.” Journal of Artificial Intelligence Research 16: 321–57. https://doi.org/10.1613/jair.953.

Davis, Jesse, and Mark Goadrich. 2006. “The Relationship Between Precision-Recall and ROC Curves.” Proceedings of the 23rd International Conference on Machine Learning (ICML ’06) (Pittsburgh, Pennsylvania, USA), 233–40. https://doi.org/10.1145/1143844.1143874.

Edwards, Aleksandra, and Jose Camacho-Collados. 2024. “Language Models for Text Classification: Is in-Context Learning Enough?” Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (Torino, Italia), 10058–72. https://aclanthology.org/2024.lrec-main.879/.

Elkan, Charles. 2001. “The Foundations of Cost-Sensitive Learning.” Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI’01) (Seattle, Washington, USA) 2: 973–78.

Frénay, Benoît, and Michel Verleysen. 2014. “Classification in the Presence of Label Noise: A Survey.” IEEE Transactions on Neural Networks and Learning Systems 25 (5): 845–69.

Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.” Proceedings of the National Academy of Sciences 120 (30): e2305016120. https://doi.org/10.1073/pnas.2305016120.

Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, Proceedings of machine learning research, vol. 70: 1321–30. https://proceedings.mlr.press/v70/guo17a.html.

Hsieh, Cheng-Yu, Chun-Liang Li, Chih-Kuan Yeh, et al. 2023. “Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes.” Findings of the Association for Computational Linguistics: ACL 2023, 8003–17.

Huyen, Chip. 2021. Introduction to Machine Learning Interviews Book. https://huyenchip.com/ml-interviews-book/.

Kumar, Ananya, Percy Liang, and Tengyu Ma. 2019. “Verified Uncertainty Calibration.” Advances in Neural Information Processing Systems.

Lin, Stephanie, Jacob Hilton, and Owain Evans. 2022. “Teaching Models to Express Their Uncertainty in Words.” Transactions on Machine Learning Research. https://openreview.net/forum?id=8s8K2UZGTZ.

Niculescu-Mizil, Alexandru, and Rich Caruana. 2005. “Predicting Good Probabilities with Supervised Learning.” Proceedings of the 22nd International Conference on Machine Learning (ICML ’05) (New York, NY, USA), 625–32. https://doi.org/10.1145/1102351.1102430.

Nixon, Jeremy, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. 2019. “Measuring Calibration in Deep Learning.” CVPR Workshops.

Platt, John C. 1999. “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods.” In Advances in Large Margin Classifiers, edited by Alexander J. Smola, Peter L. Bartlett, Bernhard Schölkopf, and Dale Schuurmans. MIT Press.

Roelofs, Rebecca, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. 2022. “Mitigating Bias in Calibration Error Estimation.” Proceedings of the 25th International Conference on Artificial Intelligence and Statistics.

Schweter, Stefan. 2020. BERTurk — BERT Models for Turkish. V. 1.0.0. Zenodo, released April. https://doi.org/10.5281/zenodo.3770924.

Tunstall, Lewis, Nils Reimers, Unso Eun Seo Jo, et al. 2022. Efficient Few-Shot Learning Without Prompts. https://arxiv.org/abs/2209.11055.

van den Goorbergh, Ruben, Maarten van Smeden, Dirk Timmerman, and Ben Van Calster. 2022. “The Harm of Class Imbalance Corrections for Risk Prediction Models: Illustration and Simulation Using Logistic Regression.” Journal of the American Medical Informatics Association 29 (9): 1525–34. https://doi.org/10.1093/jamia/ocac093.

Warner, Benjamin, Antoine Chaffin, Benjamin Clavié, et al. 2025. “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference.” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. https://arxiv.org/abs/2412.13663.

Xiong, Miao, Zhiyuan Hu, Xinyang Lu, et al. 2024. “Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.” The Twelfth International Conference on Learning Representations (ICLR 2024). https://openreview.net/forum?id=gjeQKFxFpZ.

Zadrozny, Bianca, and Charles Elkan. 2002. “Transforming Classifier Scores into Accurate Multiclass Probability Estimates.” Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’02) (New York, NY, USA), 694–99. https://doi.org/10.1145/775047.775151.

Zhao, Tony Z., Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. “Calibrate Before Use: Improving Few-Shot Performance of Language Models.” Proceedings of the 38th International Conference on Machine Learning (ICML 2021), Proceedings of machine learning research, vol. 139: 12697–706. https://proceedings.mlr.press/v139/zhao21c.html.