13  Sequence Labeling and Information Extraction

Chapter status: drafted. Every number on this page was measured on WikiANN Turkish (PAN-X.tr) — the corpus built by Pan et al. (2017) and the balanced 20,000/10,000/10,000 split defined by Rahimi et al. (2019), which is what load_dataset returns and the split the leakage in Section 13.2 lives in. Used exactly as it ships, so that anyone can reproduce these numbers in one call. Section 13.2 reports what auditing that corpus turned up, including a leakage problem that changes how the rest of the chapter reads its own scores. Software: scikit-learn 1.9.0, sklearn-crfsuite 0.5.0, transformers 5.15.1, PyTorch 2.13.0, datasets 5.0.1, seqeval 1.2.2 on Python 3.14.5. Encoder fine-tuning: batch size 16, learning rate 3e-5, three epochs, seed 42, on an Apple M-series GPU through the MPS backend — one run per budget, no hyper-parameter search. Local LLM: a 4-bit quantised qwen3.5:9b served by ollama 0.24.0 at temperature 0 with thinking disabled. Treat the timing ratios below as portable and the absolute seconds as not.

Learning objectives. After this chapter you can:

13.1 One architecture, three tasks

Chapter 12 asked one question per document: which class is this? This chapter asks one question per token, and the questions are no longer independent — the answer for token 4 constrains the answer for token 5.

That single change of granularity covers most of what the field calls information extraction:

Three tasks, one shape
task the question at each token typical labels
POS tagging what part of speech is this word? NOUN, VERB, ADJ, …
Chunking is this word inside a noun phrase? B-NP, I-NP, O
NER is this word part of a name, and what kind? B-PER, I-ORG, O, …

named entity recognition (NER) — finding the mentions of people, organisations and places in text and saying which is which. The archetypal sequence-labelling task, and the one this chapter measures. ↩︎

The architecture is identical in all three: run the sentence through something that produces one vector per token, put a classifier on top, read off one label per token. That part is token classification, and it is genuinely the easy half.

token classification — one label per token instead of one label per document. The architecture NER, POS tagging and chunking all share; what differs between them is the label set, not the model. ↩︎

The hard half is that what you actually want is not labels, it is spans — “Türkiye Büyük Millet Meclisi is an organisation”, not four separate token verdicts (B-ORG I-ORG I-ORG I-ORG). Turning one into the other is a decoding problem, and this chapter spends more time on decoding and measurement than on models, because that is where the real defects are.

The corpus

Everything in this chapter is measured on one dataset, so it is worth two minutes of your attention before any of the numbers arrive.

from datasets import load_dataset

ds = load_dataset("unimelb-nlp/wikiann", "tr")
train        20,000 sentences   149,786 tokens   median length 6, mean 7.5
             26,482 entities    PER 8,833   ORG 7,970   LOC 9,679
validation   10,000 sentences    75,930 tokens
             13,517 entities    PER 4,374   ORG 4,129   LOC 5,014
test         10,000 sentences    75,731 tokens
             13,587 entities    PER 4,519   ORG 4,154   LOC 4,914

Three entity types, PER, ORG and LOC, and a striking absence: every sentence in this corpus contains at least one entity. Real text does not work that way — most sentences in a news feed or a ticket queue mention nobody. Keep that in mind when you read the scores; it makes the task easier than the one you will actually have.

The floor, before anything else

Chapter 12 insisted on computing the floor first, and the habit pays here more dramatically than anywhere else in the book. A model that emits O for every token — no features, no parameters, no training:

everything-is-O      token accuracy 61.36%      span-level F1 0.00

Token accuracy is the share of tokens whose label is right. Span-level F1 counts a whole entity as found only when its type and both of its boundaries are right; Section 13.7 defines it properly.

Sixty-one percent accurate, and it has never found a single entity. The gap between those two numbers is not a curiosity; it is the reason Section 13.7 exists. Any paper, model card or colleague reporting “94% accuracy” on a tagging task like this one is reporting a number whose floor, on this corpus, is 61.

What we will look at. First the corpus, audited before anything is measured on it, because what that audit turns up changes how every later score reads. Then the format that turns per-token labels into spans, and the sequences a model emits that the format forbids. Then the alignment step between words and subwords, where the silent bugs live. Then three ways of building the thing — a dictionary, a CRF (a conditional random field, the classic statistical sequence model, introduced in Section 13.5), a fine-tuned encoder — at six label budgets, because the question is not which one wins but when. Then why the metric everyone reports is the wrong one. And finally what happens if you skip all of it and describe the schema to an LLM instead.

13.2 What is actually in the corpus

Chapter 5 argued that you audit a labelled corpus before you trust a number computed on it. WikiANN was built automatically, by projecting Wikipedia’s internal links through cross-lingual links (Pan et al. 2017), which makes it enormous, free, available in 282 languages — and unreviewed by anyone. So here is the audit.

It is not the first. Lignos et al. (2022) examined WikiANN’s contents for lower-resourced languages and concluded that it overstates zero-shot cross-lingual transfer; the point below is that argument re-run on Turkish, with the numbers in front of you, plus one finding that neither the corpus paper nor the audit paper prepares you for.

These are mostly not sentences.

ends with . ! ?       6,896   34.5%
does not             13,104   65.5%
3 tokens or fewer       172    0.9%

Two thirds are fragments clipped out of Wikipedia markup. That is a fact about the domain, and it will not transfer to prose.

Some mentions are not mentions.

pure punctuation or digits    160   0.60%
containing wiki markup         63   0.24%

  ORG: '1966–67'
  ORG: '2012'
  ORG: '2013'
  ORG: '2000'

Bare years typed as organisations. These counts are over the training split’s 26,482 mentions, and the test split was built the same way. A small fraction, but it puts a ceiling on what any model can score, because some of the test set is simply wrong.

The same string gets labelled two ways.

distinct surface forms                16,591
forms carrying more than one label       116   (0.7%)
mentions affected                      1,397   (5.3%)

  'Türkiye'      {LOC: 82, ORG: 4}
  'Almanya'      {LOC: 80, ORG:  2}
  'Fransa'       {LOC: 69, ORG:  5}
  'İtalya'       {LOC: 55, ORG:  1}

Some of this is real ambiguity — Türkiye beat İtalya is the national football team, not the landmass — and some is annotation noise. Either way, for 5.3% of mentions the same string has appeared with another label, so the model has to guess which one applies here.

The part that changes how you read every score

Everything so far has been a nuisance: fragments instead of sentences, a handful of bad labels, some ambiguity the model has to guess through. Annoying, survivable, and roughly what you expect from an automatically built corpus. This next one is a different kind of problem.

13.06% of the test set is in the training set.

test sentences identical to a train sentence   1,306 / 10,000   13.06%
  of those, carrying an identical gold tagging  1,305
  carrying a different gold tagging                 1

validation sentences also in train             1,265   12.65%
test sentences also in validation                846    8.46%
duplicate sentences inside train itself        1,565
duplicate sentences inside test itself           540

Not similar sentences. Byte-identical token sequences, compared on the token lists exactly as datasets returns them, with no case folding, no Unicode re-normalisation and no whitespace repair. Looser matching could only find more, so 13.06% is a lower bound on the overlap rather than an upper one. And 1,305 of them carry byte-identical gold labels. A model that does nothing but memorise its training data gets those 1,306 sentences right for free.

This is the kind of thing Elangovan et al. (2021) warned about, and it is worth being precise about who it flatters. Later in this chapter three systems are scored separately on the leaked and the clean halves of the test set:

Span-level F1, split by whether the test sentence was in training. The last column divides the gap by the room the clean half left above it
system leaked 1,306 clean 8,694 gap gap / headroom
Gazetteer (pure memorisation) 77.54 51.04 +26.50 54%
CRF with hand features 99.81 83.56 +16.25 99%
Fine-tuned BERTurk 98.69 91.04 +7.65 85%

The CRF scores 99.81 on the leaked half. It is not a better model there; it is a lookup table that has seen the answer.

Read that gap column downwards, because it is the most useful thing in this section. The three systems are ordered by how much they memorise — a dictionary, a feature-based linear model, a pre-trained encoder — and the leakage bonus falls with them in the same order: +26.50, +16.25, +7.65.

Read the last column too, though, because the raw gap is bounded above by whatever headroom the clean half leaves: a system scoring 91.04 clean cannot gain more than 8.96 no matter how much it memorises. Normalised, the ranking changes: the gazetteer drops from first to last. The CRF’s +16.25 uses 99% of the 16.44 points available to it, the encoder’s +7.65 uses 85% of its 8.96, and the gazetteer’s larger +26.50 uses only 54% of its 48.96. The two readings agree on the practical point and disagree on the ranking, which is exactly why the gap is a diagnostic rather than a memorisation score: a large gap means look harder, not that you have measured how much your model memorises. Three systems on one corpus is as far as this evidence goes.

What both readings do support: a benchmark with 13% leakage does not just inflate scores. It inflates them unevenly, and the systems it flatters most are the ones that will disappoint you in production.

Go back to the table: one number in it deserves a second look. A dictionary that has memorised every training string ought to score near 100 on sentences it has seen verbatim, and it scores 77.54. The missing twenty-two points are the two nuisances from earlier in this section arriving together: longest match fires on strings that are entities somewhere in training but not here, and when a surface form carries two labels — the 5.3% of mentions in the table above — the dictionary has to pick one. Even pure memorisation does not get a leaked test set for free. It just gets most of it.

Every headline number in this chapter is computed on the full test set, because that is what the literature reports and comparisons have to be like-for-like. But the clean-half column is the one to believe.

The honest fix is to deduplicate against training before scoring, and to say so. The cheaper habit, and the one worth carrying out of this chapter: when a result looks too good, hash your test inputs and check how many of them your model has already seen. It takes four lines and it has saved more projects than any modelling trick in this book.

13.3 Tagging schemes and span decoding

A tagger emits one label per token. You want spans. The bridge is a tagging scheme, and the standard one is BIO: B-X opens a span of type X, I-X continues it, O is everything else.

One clarification before the grammar, because it decides what counts as illegal. The scheme this chapter uses — and the one datasets ships WikiANN in — is IOB2: every span opens with B-, without exception. The original CoNLL-2003 scheme, usually called IOB1, is a different thing: there B- appears only to separate two adjacent spans of the same type, so a span normally opens with I- and O → I-PER is perfectly legal. Everything below about forbidden transitions is a statement about IOB2. Run an illegal-transition counter over CoNLL-2003 as distributed and it will report thousands of violations that are not violations.

BIO tagging — B- marks a span’s first token, I- a continuation, O everything outside. The trick that lets a per-token classifier express multi-token spans, including two of the same type back to back. ↩︎

BIO turns spans into per-token labels — and creates a grammar
WHAT THE ANNOTATOR DREW — SPANS PER LOC Mustafa Kemal 1919 yılında Samsun'a çıktı . WHAT THE MODEL PREDICTS — ONE LABEL PER TOKEN B-PER I-PER O O B-LOC O O B- opens a span · I- continues one · O is outside any span which is why two adjacent people are B-PER B-PER, and stay two people THE GRAMMAR THIS CREATES — WHAT I-PER MAY FOLLOW B-PER legal I-PER legal O forbidden B-LOC forbidden a CRF learns to avoid the red ones only an explicit constraint rules them out

The span annotation on top is what a human drew; the label row underneath is what the model is actually trained to predict. Everything in Section 13.3.1 follows from the fact that those two are not the same object.

The B/I distinction is not decoration. Without it, two adjacent entities of the same type would merge into one: Ali Veli as PER PER is one span or two, and you cannot tell. With BIO, B-PER B-PER is unambiguously two.

BIOES adds E- for the last token of a span and S- for a single-token span. It gives the model more specific classes to learn and the decoder more redundancy to check, at the cost of nearly doubling the label set: with \(K\) entity types BIO has \(2K+1\) labels (7 here) and BIOES \(4K+1\) (13). Ratinov and Roth (2009) compared them carefully and found BIOES (which they call BILOU) ahead on their setup — which is worth knowing mostly so you can say it depends on the corpus with a citation behind it, rather than repeating whichever one your first tutorial used.

The sequences that should not exist

BIO has grammar. I-PER may follow B-PER or I-PER. It may not follow O, and it may not follow B-LOC. A tagger that scores each token independently does not know that.

Here is how often the gold data and three systems break the rule, across the 10,000 test sentences:

                              I- after O    I- after a different type
gold test set                          0                            0
gazetteer (longest match)              0                            0
CRF                                    0                            0
fine-tuned BERTurk                    49                           90

The CRF, at the one budget it was trained on (20,000 sentences), emitted no illegal transition; the fine-tuned transformer — the best model in this chapter, at 91.92 span F1 — emits 139 of them. That inversion — the weaker model is the well-formed one — surprises people, and the reason is structural rather than a matter of quality.

A softmax tagger picks the highest-scoring label at each position independently. Nothing in the arithmetic couples position 4 to position 5, so O followed by I-ORG is available whenever the scores happen to favour it. A CRF (Lafferty et al. 2001) scores whole sequences: it learns a transition score for every label pair it saw in training and leaves the pairs it never saw — O → I-ORG among them — without a supporting feature at all. (That is CRFsuite’s default, all_possible_transitions=False in sklearn-crfsuite. An unseen transition therefore scores 0 rather than being penalised: nothing pushes the model towards it, while the legal transitions carry the weights they learned from data. Setting the option to True creates features for the unseen pairs too, so they can learn negative weights.) Then Viterbi decoding (Viterbi 1967) returns the best-scoring path rather than the best label at each step, so an illegal transition has to pay for itself out of the emission scores of the whole rest of the sentence. In this run it never did: across 10,000 test sentences, the CRF emitted zero. That is a measured outcome, not a theorem. The scores are finite, so the path is improbable rather than impossible; if you need the guarantee instead of the empirical zero, “Making the grammar structural instead of learned” below is how to get it.

Viterbi decoding — a dynamic program that finds the single highest-scoring label sequence, instead of taking the best label at each position separately. What lets a CRF price a forbidden transition against the whole sentence rather than against one token. ↩︎

This is exactly why the classic neural NER architecture was BiLSTM-CRF (Lample et al. 2016) rather than BiLSTM alone: the encoder supplies the features, the CRF layer supplies the grammar. Whether that layer still earns its cost on top of a transformer is an open question this chapter does not settle: Section 13.6 measures the plain softmax head only. The transition mask below is the cheap alternative to finding out.

What a decoder does with a broken sequence

So your model emits O I-ORG I-ORG. What is that? There is no right answer, only conventions, and they disagree:

Three readings of the same predictions
decoder O I-ORG I-ORG becomes rationale
lenient (seqeval, conlleval (Tjong Kim Sang and De Meulder 2003)) one ORG span over tokens 2–3 a stray I- opens a span; recover what you can
strict nothing the sequence is invalid; refuse to guess
repair-then-decode rewrite to O B-ORG I-ORG, then decode fix the grammar first, score after

The point is not which to pick. The point is that your reported F1 depends on a decoder you probably did not choose deliberately, and the difference is invisible unless you look for it. On the CRF’s predictions the choice happens to cost nothing, because there is nothing to disagree about:

same CRF predictions, two decoders
  lenient (a stray I- opens a span)    F1 85.45
  strict  (a stray I- is dropped)      F1 85.45

On the transformer’s 139 malformed transitions the two decoders return different numbers of spans, so their F1 diverges; we did not measure that pair — measure it on your own model. When you compare your numbers against a paper’s, the decoder is one of the things that has to match.

Making the grammar structural instead of learned

You do not have to add a CRF layer to get the well-formedness a learned model only supplies empirically. A transition mask over the softmax logits, decoded with Viterbi, gives it at inference time — no extra parameters, no retraining. Two rules make up the grammar: I-X may only follow B-X or I-X, and a sentence may not start with I-X (the start behaves like an O).

One note on the constant below: \(-10^4\) is a large finite penalty rather than \(-\infty\). At inference either works. The finite value matters if you reuse the same mask inside a training objective: there a log-sum-exp over paths can turn \(-\infty\) into nan in edge cases, for example when every path through a position is ruled out. The finite penalty makes an illegal path lose in any realistic score range; it does not make it mathematically impossible. If you want the true constraint, remove the invalid transitions and start states from the search space rather than pricing them. The code below is the finite-penalty version; for the true constraint at inference, set NEG = float("-inf") and the rest runs unchanged. In practice the two agree: with log-probability scores, any illegal path can be turned into a legal one by rewriting the offending I-X as B-X, at the cost of one token’s log-probability gap — tens in realistic models, not \(10^4\).

import torch

NEG = -1e4
mask = torch.zeros(L, L)                    # L = number of labels
for a, la in enumerate(labels):
    for b, lb in enumerate(labels):
        if lb.startswith("I-"):             # I-X may only follow B-X or I-X
            if la not in (f"B-{lb[2:]}", f"I-{lb[2:]}"):
                mask[a, b] = NEG
start = torch.tensor([NEG if l.startswith("I-") else 0.0 for l in labels])

def viterbi(logits, mask, start):           # logits: (T, L) log-probabilities
    dp, bp = logits[0] + start, []          # no sentence may open with I-
    for t in range(1, logits.shape[0]):
        score = dp[:, None] + mask + logits[t][None, :]
        dp, best = score.max(0)
        bp.append(best)
    path = [int(dp.argmax())]
    for back in reversed(bp):
        path.append(int(back[path[-1]]))
    return list(reversed(path))

A two-token example shows what the dynamic program buys. Take Merkez Bankası (“Central Bank”) with only three labels, and suppose the softmax head gives these probabilities:

A toy softmax output; the greedy choice at each position is in bold
token O B-ORG I-ORG
Merkez 0.6 0.3 0.1
Bankası 0.2 0.2 0.6

Taking the best label at each position gives O I-ORG, an illegal sequence. The lenient decoder turns it into a one-token ORG on Bankası alone (a boundary error); the strict decoder drops it. Viterbi instead scores whole paths, a product of probabilities (a sum of log-probabilities). With O → I-ORG and a sentence-initial I-ORG ruled out, the legal candidates include O O \(= 0.6 \times 0.2 = 0.12\), O B-ORG \(= 0.12\) and B-ORG I-ORG \(= 0.3 \times 0.6 = 0.18\). The last one wins: the decoder revises the first token’s label because of evidence at the second, and returns the full two-token organisation. A per-token argmax can never do that.

That took the 139 illegal transitions to 0 in this run, for the price of a matrix and a dynamic program. Not by construction: a finite penalty is still a price, and an emission score large enough would still buy an illegal step. Drop the invalid entries from the search space instead of pricing them and the guarantee is real. Whether it helps is a separate question and one you should measure on your own schema rather than assume: it converts malformed spans into well-formed ones, and the lenient decoder was already recovering some of them. We did not run that comparison, which is the gap this subsection leaves open — but the mask is what to reach for before an extra trained layer.

13.4 The subword alignment trap

Your labels are per word. A transformer’s inputs are per subword. Every sequence-labelling pipeline therefore contains an alignment step, and it is the single most common place to introduce a silent bug — silent because the model still trains, the loss still falls, and the score is merely worse than it should be.

Turkish makes the problem vivid. Here is BERTurk’s tokenizer on entity words from this corpus:

Türkiye'nin      -> ['Türkiye', "'", 'nin']
Takımı'nı        -> ['Takımı', "'", 'nı']
Mississippi'de   -> ['Miss', '##iss', '##ip', '##pi', "'", 'de']
Baudelaire'in    -> ['Ba', '##ude', '##la', '##ire', "'", 'in']
Erdemir          -> ['Erdem', '##ir']
İstanbul         -> ['İstanbul']

One word, six subwords. And the fragmentation is not evenly spread — it concentrates on exactly the words you are trying to label:

BERTurk (dbmdz/bert-base-turkish-cased)
  entity words   1.98 subwords/word   53.8% split into more than one piece
  other words    1.45 subwords/word   30.9% split
  -> entity words are 1.36x more fragmented

mBERT (bert-base-multilingual-cased)
  entity words   1.90 subwords/word   47.3% split
  other words    2.00 subwords/word   45.5% split

Names are rarer than common words, so the tokenizer has less reason to keep them whole — the mechanism Section 8.3 described, arriving here as a concrete cost.

The mBERT row is worth reading carefully, because it is easy to over-read. It fragments everything in Turkish at 2.00 subwords per word, which is Chapter 8’s argument for language-specific tokenizers: more tokens for the same text means more cost and less effective context, permanently.

What it is not is evidence that mBERT will score worse on this task. Tokenization is what was measured here; accuracy is not. Chapter 16 fine-tunes both models on this exact dataset and finds the multilingual one slightly ahead — a useful reminder that a tokenizer tax is a tax on tokens rather than a prediction about F1.

The alignment itself

The convention is to label the first subword of each word and mask the rest with -100, the index PyTorch’s cross-entropy ignores. Here is that convention on three of the entity words above — and, underneath it, what happens when you skip the alignment and zip the labels on by position.

One label per word, many subwords per word — and the off-by-a-word that still trains
WHAT YOU HAVE — ONE GOLD LABEL PER WORD B-LOC B-ORG B-LOC İstanbul Erdemir Mississippi'de WHAT THE MODEL GETS — THE SAME THREE WORDS, IN THE SAME COLUMNS [CLS] İstanbul Erdem ##ir Miss ##iss ##ip ##pi ' de [SEP] CORRECT — LABEL EACH WORD'S FIRST SUBWORD, MASK EVERY OTHER POSITION -100 B-LOC B-ORG -100 B-LOC -100 -100 -100 -100 -100 -100 THE BUG THAT STILL TRAINS — THE THREE LABELS ZIPPED ON BY POSITION B-LOC B-ORG B-LOC — — — — — — — — Every label lands one word early. B-LOC trains on [CLS], and Mississippi'de is never labelled at all. Nothing raises an error. The loss falls, the run finishes, and the score is merely worse than it should be.

The three tokenizations are BERTurk’s, printed verbatim above. Nothing connects the rows except the columns: each word sits over exactly the subwords it was split into, which is what word_ids() gives you and what a positional zip throws away.

In code:

enc = tok(batch["tokens"], is_split_into_words=True, truncation=True)

labels = []
for i, tags in enumerate(batch["ner_tags"]):
    word_ids = enc.word_ids(batch_index=i)
    prev, row = None, []
    for wid in word_ids:
        if wid is None:            # [CLS], [SEP], padding
            row.append(-100)
        elif wid != prev:          # first subword of a word: label it
            row.append(tags[wid])
        else:                      # continuation: ignore in the loss
            row.append(-100)
        prev = wid
    labels.append(row)

Three ways this goes wrong, all of which still run:

  1. Forgetting word_ids() and zipping labels to subwords positionally. The moment any word splits, every subsequent label is attached to the wrong token. With 53.8% of entity words splitting, the labels are scrambled almost immediately. Training proceeds; F1 lands somewhere near noise.
  2. Propagating the word’s label to every subword. Defensible, and sometimes done — but if you propagate B-PER rather than converting continuations to I-PER, you have taught the model that a person’s name starts four times.
  3. Forgetting that [CLS] and [SEP] exist. An off-by-one that shifts every label by one position.

If a sequence-labelling model scores far below expectation, check the alignment before you touch the model. Print one tokenized sentence beside its aligned labels and read it. It takes a minute and it is the highest-yield minute in this chapter.

13.5 Baselines before transformers

Chapter 12’s discipline: build the cheap thing first, and make the expensive thing beat it.

A gazetteer

Memorise every entity string in the training data, then tag test sentences by longest match. No learning, no features, no generalisation whatsoever.

gazetteer (16,591 memorised phrases, longest match)
  token accuracy 75.73%
  span   P 59.73   R 51.41   F1 55.26      (0.1s to tag 10,000 sentences)

Fifty-five F1 from a Python dictionary. That is the number a model has to justify itself against, and it is higher than most people guess — because names repeat. In this corpus 52.2% of test mentions appear verbatim in the training data, which is simultaneously the gazetteer’s ceiling on recall and a restatement of the leakage problem from Section 13.2.

gazetteer — a memorised list of known entity strings, matched against text directly. Fast, exact, completely unable to recognise a name it has never seen. ↩︎

A gazetteer is also the right answer more often than its reputation suggests. If your entities are a closed set — the parts in your catalogue, the drugs in a formulary, the 81 provinces of Turkey — then the list is the ground truth and a model can only approximate it.

A CRF

The gazetteer cannot recognise a name it has never seen, because it has no notion of what a name looks like. So give a model that notion by hand. Capitalisation, suffixes, the shape of the word, what its neighbours look like.

A linear-chain CRF scores a whole label sequence \(y = (y_1, \dots, y_T)\) for a sentence \(x\) as a sum of two kinds of term — one that looks at the token, one that looks at the label pair:

\[ s(x, y) \;=\; \sum_{t=1}^{T} \underbrace{\mathbf{w}^\top \mathbf{f}(x, t, y_t)}_{\text{emission}} \;+\; \sum_{t=2}^{T} \underbrace{A_{y_{t-1},\, y_t}}_{\text{transition}} \]

and turns that score into a probability by normalising over every label sequence, which is what makes it a model of the sequence rather than of each token:

\[ p(y \mid x) \;=\; \frac{\exp s(x, y)}{\sum_{y'} \exp s(x, y')} \]

Two things follow from that denominator. Training maximises the log-likelihood of the gold sequence — the c1 and c2 in the code below are \(L_1\) and \(L_2\) penalties on \(\mathbf{w}\) and \(A\) — and prediction takes \(\arg\max_y s(x, y)\). Both the sum in the denominator (needed for training) and the argmax (needed for prediction) range over all \(L^T\) label sequences for \(L\) labels and \(T\) tokens, far too many to list. Because each score only couples adjacent labels, dynamic programming computes both in \(O(T L^2)\): the forward algorithm for the sum, Viterbi for the max.

The name for what the conditioning buys is worth having. A hidden Markov model (HMM), the classic sequence model before CRFs, tells a story in which each label generates its word and each label generates the next label. It is generative: it models the joint \(p(x, y)\), so it has to describe how the words were produced, and that forces strong independence assumptions on them — you cannot casually add “the previous word is title-cased” and “this word ends in -nin” as overlapping evidence without the model double-counting. A CRF is discriminative: it models \(p(y \mid x)\) directly and never has to explain where \(x\) came from, so the feature dictionary below can contain whatever correlated, hand-designed, overlapping signals you like. That freedom is the whole reason the feature template is written by hand, and the reason a CRF beat the alternatives on this shape of task in 2001 (Lafferty et al. 2001).

conditional random field (CRF) — a discriminative model that scores a whole label sequence at once, adding a learned transition score for each adjacent label pair to the per-token feature scores. Models \(p(y \mid x)\) rather than \(p(x, y)\), which is what lets its features overlap freely. ↩︎

Over twenty thousand sentences:

def features(tokens, i):
    w = tokens[i]
    f = {
        "bias": 1.0,
        "w.lower": w.lower(),
        "w[-3:]": w[-3:], "w[-2:]": w[-2:],        # Turkish suffixes
        "w.isupper": w.isupper(), "w.istitle": w.istitle(),
        "w.isdigit": w.isdigit(),
        "w.hasapos": ("'" in w or "’" in w),        # Türkiye'nin
        "w.shape": shape(w),                        # Xxxxx, 9999
        "w.len": min(len(w), 12),
    }
    for off in (-2, -1, 1, 2):                      # a window of context
        j = i + off
        if 0 <= j < len(tokens):
            v = tokens[j]
            f[f"{off:+d}:w.lower"] = v.lower()
            f[f"{off:+d}:w.istitle"] = v.istitle()
            f[f"{off:+d}:w[-3:]"] = v[-3:]
        else:
            f[f"{off:+d}:BOS_EOS"] = True
    return f
CRF (sklearn-crfsuite, lbfgs, c1=0.1, c2=0.1, 200 iterations)
  fit 18.5s        predict 0.2s for 10,000 sentences

  token accuracy 94.41%      token-level micro-F1 88.73
  span   P 86.27   R 84.64   F1 85.45      macro-F1 85.34

     PER   P 88.64   R 88.56   F1 88.60    (n=4,519)
     LOC   P 84.09   R 86.06   F1 85.06    (n=4,914)
     ORG   P 86.34   R 78.70   F1 82.34    (n=4,154)

85.45 span F1 in eighteen seconds on a CPU. No GPU, no pre-training, no downloads. This is the number the rest of the chapter has to beat, and it is a genuinely hard target.

Two things in that table are worth reading rather than skimming. ORG recall is 78.70, ten points below PER and seven below LOC — organisations are long, compositional and open-ended (Slovenya Millî Basketbol Takımı), so the five-token window around a long name’s first word may not reach its last word, which is often the one that says “organisation” (Takımı, “team”). And the w.hasapos feature is doing real work: in Turkish orthography an apostrophe separates a suffix from a proper noun (or from a number or abbreviation, as in 1919’da), so it is a strong signal that the preceding characters are a name — a piece of orthographic luck few languages offer so regularly.

13.6 Fine-tuning an encoder

Same task, same test set, AutoModelForTokenClassification on top of BERTurk, first-subword alignment as above, three epochs, learning rate 3e-5. The interesting question is not “does it win” but “at what label budget does it start winning”, so the run is a curve rather than a point.

budget    token acc   span P   span R   span F1   macro   illegal   train s
   200        63.43     9.02     1.90      3.14    2.95      1561       5.6
   500        85.04    51.25    58.11     54.47   54.05      3739      12.6
 1,000        92.59    74.99    80.44     77.62   77.26       915      24.6
 2,000        94.68    82.74    85.78     84.23   84.04       538      48.9
 5,000        95.96    87.44    89.24     88.33   88.16       304     121.3
20,000        97.20    91.49    92.35     91.92   91.82       139     476.6

CRF, all 20,000 sentences   94.41    86.27    84.64     85.45   85.34    0    18.5

One run per budget, seed 42, no hyper-parameter search — and the two small budgets are the volatile ones. At 200 sentences, whether a randomly initialised head escapes the all-O solution at all can depend on the seed, so read 3.14 as single digits rather than as a number with two decimals behind it. There is also a floor under every comparison in this table: span F1 near 85% over the test set’s 13,587 gold spans has a standard error of about 0.3 points from test-set sampling alone, a 95% interval of roughly \(\pm 0.6\). The 84.23-against-85.45 gap that fixes the crossover is 1.22 points, about four such standard errors, so the test set is not what makes it uncertain. The single training run is: we do not know how far a different seed would move 84.23. Three seeds per budget is the measurement this table is missing.

At the full budget the encoder reaches 91.92 span F1 against the CRF’s 85.45 — six and a half points, for 477 seconds of GPU training instead of 18 seconds of CPU, and 10.3 seconds of inference over the test set instead of 0.2.

Read the top row again. At 200 labelled sentences, a fine-tuned transformer scores 3.14 span F1 — while a dictionary of memorised strings scores 55.26.

The two sides are not matched: that dictionary holds 16,591 phrases built from all 20,000 sentences. Someone with only 200 labelled sentences could build a list of roughly 265 entities, and it would score below 55.26; we did not measure where. What this row really stands for is having a ready-made list (provinces, a product catalogue).

The encoder’s 3.14, on the other hand, is not a bug, and it is the most useful thing in this chapter. At 200 sentences the model has seen roughly 265 entities spread over six entity tags plus O; the classification head is randomly initialised; and at batch size 16, three epochs is 39 gradient updates.

Note what that last number does to the diagnosis. Every row in this table uses the same three epochs at 3e-5 — a schedule tuned for the full budget, which starves the small ones. So some of that 3.14 is the label budget and some of it is the schedule, and this run does not separate them. The standard remedy at 200 examples is ten to twenty epochs, possibly at a higher learning rate; running that is the first thing to try before drawing the conclusion, and we did not. So do not read 3.14 as the ceiling of what 200 sentences can buy. With 500 sentences and only 96 updates the same recipe already reaches 54.47, and a longer schedule at 200 may close much of the gap to the dictionary — this run cannot tell you. The raw number is a schedule artefact as much as a data-budget one; what survives is the practical point that, with a default recipe and no tuning, a small label budget can leave a fine-tuned encoder behind a list of strings.

Token accuracy is 63.43% — barely above the everything-is-O floor of 61.36% — which tells you precisely what it learned. It learned to say O. It picks up almost no entities at all (recall 1.90), and the few it does are mostly wrong.

The crossover against the 20,000-sentence CRF sits between 2,000 and 5,000 labelled sentences: at 2,000 the encoder is still a point behind (84.23 against 85.45), and by 5,000 it is nearly three ahead. So somewhere around two and a half thousand labelled sentences of BERTurk buy what twenty thousand sentences of hand-engineered CRF buy. Call it a fourfold to tenfold reduction in annotation (\(20{,}000 / 5{,}000\) to \(20{,}000 / 2{,}000\)). That is the range the measured crossover actually supports, it is the honest case for pre-training on this task, and it is smaller than the folklore suggests. Be exact about what that comparison is: a label-efficiency claim, in which the encoder’s budget varies while the CRF’s stays at 20,000. It is not a matched-budget comparison: the CRF was trained only on 20,000 sentences, so this table does not say whether a 2,000-sentence CRF would lead or trail the encoder. What it says is that below that crossover the 20,000-sentence classical pipeline is not a fallback; it is the better model. Training the CRF at the same six budgets is the table’s second missing measurement.

The illegal column is worth a second look, because it says something the F1 column does not. Well-formedness is learned here, not guaranteed: illegal BIO transitions peak at 3,739 once the model starts emitting entity tags at all, then fall steadily as it picks up the grammar from data.

And they never reach zero. Even at 20,000 sentences and 91.92 F1, the tagger emits 139 sequences that IOB2 forbids. The 20,000-sentence CRF emitted none, because Viterbi prices the whole sequence and a transition with no supporting feature never earned its place in the winning path here. If you want the guarantee rather than the empirical zero, mask the forbidden transitions before decoding (Section 13.3).

Notice which way the errors run. The CRF is precision-leaning (86.27 P against 84.64 R); every encoder row from 500 up is recall-leaning. Feature templates fire when they match and stay silent otherwise, while a fine-tuned encoder will guess. Which failure you prefer is a product decision, and Section 12.7 is the machinery for making it.

Label budget against span-level F1 — and what the CRF costs to beat

13.7 Token accuracy is the wrong metric

Three numbers, all computed on the same CRF predictions over the same 10,000 test sentences:

token accuracy                94.41
token-level micro-F1          88.73      (entity tags only, O excluded)
span-level F1                 85.45      (exact match on type, start and end)

They differ because they ask different questions, and only the last one asks the question you care about.

Token accuracy counts every token, including the 61% that are O. Predicting O everywhere already scores 61.36, so the informative range of this metric is the thirty-five or so points between 61 and the high 90s — and it gets worse, not better, on realistic corpora where most sentences contain no entity at all. It is the metric that makes a model look finished when it is not.

Token-level F1 over entity tags drops the O class, which fixes the floor. But it still gives partial credit: get three tokens of a four-token organisation right and you score three quarters of a hit. No downstream system can use three quarters of an organisation name.

Span-level F1 requires the type, the start and the end to all be right. It is the only one of the three that corresponds to “did we extract the record correctly”, which is why it is what NER papers report.

Two independent choices hide inside the phrase “span-level F1”, and separating them saves an argument later. Exact match is about boundaries: does the predicted span’s type, start and end all match a gold span? Strict versus lenient is about what the decoder does with a malformed tag sequence before any matching happens. seqeval is exact-match in both of its modes; its default mode is the lenient one, and mode="strict", scheme=IOB2 is the other.

Every span-level number in this chapter is computed in the lenient convention — seqeval’s default, and conlleval’s. The scorer used throughout this chapter is written out rather than imported, so that the decoding it assumes is visible. It was checked against seqeval on deliberately corrupted predictions and agrees to four decimal places — P 46.2332, R 75.2042, F1 57.2629 from both. If you write your own, run that comparison; the disagreements are always in the malformed sequences.

span-level F1 — a prediction counts as correct only if its type, its first token and its last token all match a gold span. Partial overlaps score zero, which is the point: half an entity is not half useful. ↩︎

Most of the gap between token-level 88.73 and span-level 85.45 is boundary errors — spans where the type was right and an edge was wrong. (The two metrics also weight spans differently, so the gap is not a pure error count.) Those errors concentrate in long names built from ordinary words, which is ORG, as the next subsection shows.

Turkish adds a subtler boundary problem that these scores cannot see: the true end of the entity in Samsun’a is inside the word, before the case suffix. WikiANN labels whole whitespace tokens, so the suffix is simply counted as part of the name, and neither metric penalises it. It comes back later, when the extracted strings have to be counted or linked (Section 13.11).

Exact match is the convention, not a law. If your downstream consumer is a human reviewing highlighted text, a span that is off by one token is nearly as useful as a perfect one, and reporting only exact match will make you reject models your users would have preferred.

The alternatives are the MUC-style partial-credit schemes (Chinchor and Sundheim 1993) and the strict / exact / partial / entity-type breakdown that nervaluate implements. Use them when they match your consumer, and say which you used — a paper reporting 90 under partial credit is not beating a paper reporting 88 under exact match.

Where the errors are

A single F1 number tells you how much is wrong and nothing about what. Split it by entity type and the same two systems look rather different:

                CRF                       BERTurk @ 20,000
        precision  recall      F1     precision  recall      F1   support
  PER       88.64   88.56   88.60         94.19   94.73   94.46     4,519
  LOC       84.09   86.06   85.06         91.16   93.22   92.18     4,914
  ORG       86.34   78.70   82.34         88.93   88.73   88.83     4,154

ORG is last in both columns, and its recall is the outlier — 78.70 for the CRF, and still the lowest of the three at 88.73 for the encoder six points further up. Six points of extra model did not move ORG off the bottom.

The reason is in the spans themselves. Organisation names are long — mean gold span 2.68 tokens for ORG against 2.26 for PER and 1.61 for LOC — compositional, and frequently built out of ordinary words (Türk Silahlı Kuvvetleri, Eurovision Şarkı Yarışması). A model with a five-token window cannot see all of a long one from its first word, and a model that can still has to decide where it stops.

Report macro-F1 alongside micro when performance is this uneven across types. Here they are close (85.34 macro against 85.45 micro) only because the three types happen to be similarly sized; in a schema with one rare type and one common one they diverge sharply, exactly as Chapter 12 showed for document labels.

One boundary on all of this, since the chapter opened with three tasks and measured one. Everything above about spans, BIO and span-level F1 applies to tasks whose output is a span: NER, chunking, slot filling. POS tagging is not one of them — every token carries a label, no token is O, and there is nothing to decode — so token accuracy is a perfectly reasonable metric there, and its floor is the most-frequent-tag baseline (each word gets the tag it carried most often in training; around 90% on English, lower on Turkish, where many word forms never appear in training) rather than 61%. The architecture is shared; the evaluation is not. If you came here for POS on Turkish, the label set is Universal Dependencies’ and the corpora are the UD Turkish treebanks (BOUN, IMST) — but Section 13.4 is the part of this chapter you still need, because the subword problem does not care what the labels mean.

13.8 Schema-guided extraction with an LLM

The 2026 alternative to training a tagger is to describe the schema and ask. No labelled data, no training run, an afternoon of prompt iteration. It works well enough to be a real option and it fails in ways a tagger never does, so it needs measuring rather than believing.

The setup is deliberately favourable: the annotation guidelines go in the prompt, the output is constrained to a JSON schema, temperature is 0.

SCHEMA = {
  "type": "object",
  "properties": {
    "entities": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "text": {"type": "string"},
          "type": {"type": "string", "enum": ["PER", "ORG", "LOC"]},
        },
        "required": ["text", "type"],
      },
    }
  },
  "required": ["entities"],
}

The trap that costs a day

Before any results, a bug worth more than the results. The same schema, the same model, the same server — two endpoints:

prompt: "What is the capital of France?"
schema: {"answer": integer}

POST /api/generate   with format=<schema>
  -> {"answer": 1}

POST /api/chat       with format=<schema>
  -> "The capital of France is **Paris**.\n\nLocated in the north-central
      part of the country along the Seine River, Paris is..."

The schema said the reply must be an object with an integer answer. On /api/generate the constraint was enforced. On /api/chat, with ollama 0.24.0 on Apple Silicon and thinking disabled, the format field was silently ignored and the model returned prose. No error, no warning, no field in the response saying the constraint had been dropped.

Be careful which variable you blame, though, because we did not isolate it. This is a known class of bug rather than a property of /api/chat: the upstream tracker carries several reports of format being dropped without warning, and they point at the runner and the flags rather than at the endpoint — one backend dropping the constraint while a different build of the same model enforces it, and think: false disabling it on some models. Our setup sits at exactly that intersection. The lesson survives either diagnosis, and it is the practical one: pin your server version, and assume the constraint is off until a request has proved it is on.

Run that inside an extraction pipeline with a try/except json.JSONDecodeError around the parse, and you get an extractor that returns zero entities for every document and a metrics table full of zeros you will spend an afternoon blaming on your prompt.

Note also what the constrained call returned: {"answer": 1}. Paris is not an integer, so grammar-constrained decoding produced something schema-valid and semantically worthless. Nor did the schema offer a legal way to say “this question has no integer answer” (a nullable field, say); the model was forced to produce a number. A schema guarantees shape, never truth. Chapter 18 is where the machinery behind this lives — how grammars constrain sampling, and what they cost.

The general rule this stands for: verify that your structured-output constraint is actually being applied, by sending one request whose schema the model would otherwise violate. A schema you believe is enforced and is not is worse than no schema, because it removes the malformed output that would have told you.

Getting the answer back onto the text

A tagger returns positions. An LLM returns strings. Nothing in {"text": "Samsun", "type": "LOC"} says which tokens that was, and span-level F1 is defined over token offsets — so every LLM answer has to be aligned back onto the sentence before it can be scored at all.

This step is usually skipped in write-ups and it is where the interesting failures are:

def align_span(tokens, text):
    want = text.split()
    for i in range(len(tokens) - len(want) + 1):      # 1. exact token run
        if tokens[i:i + len(want)] == want:
            return i, i + len(want)
    ...                                               # 2. ignore whitespace
    ...                                               # 3. allow a dropped suffix
    return None                                       # not in the sentence

Three things can come back:

  • A mention that is there — align it, score it.
  • A mention with the Turkish suffix stripped. The model returns Samsun; the token is Samsun'a. The guidelines asked for an exact copy and it helpfully normalised. Fallback 3 exists for this and it is doing real work.
  • A mention that is not in the sentence at all. A paraphrase, an expansion, an inference. There is no honest way to score it, and counting it as a hit would be scoring the model on a claim about text it invented.

It is tempting to call every unalignable mention a hallucination and move on. That turns out to be mostly wrong, and finding out why is worth the section.

Across 1,000 sentences the model returned 1,419 typed mentions, of which 31 would not align. Sorting those 31 by cause:

aligned directly                              1,388   97.82%
recovered by normalising apostrophes              7    0.49%   ' vs ’
recovered by stripping wiki markup               20    1.41%   ''Gorran ''
genuinely not in the sentence                     4    0.28%

Twenty-seven of the thirty-one failures were the aligner’s fault, not the model’s. Twenty came from the corpus’s own Wikipedia markup, seven from the model normalising a curly apostrophe to a straight one. Had this been reported without looking, the headline would have been “the model hallucinated 31 entities” — a claim about the model that is really a claim about preprocessing.

And the second half of the lesson, which is less comfortable: fixing the aligner barely moves the score, from 60.57 to 60.29 F1. Recall rises (60.82 to 61.11) and precision falls (60.33 to 59.50), because the recovered mentions were mostly wrong anyway. So check the alignment — and do not expect it to be the explanation for a disappointing number.

State the protocol behind these numbers, too: unalignable mentions were left out of the precision denominator — not silently, since their rate is reported above (the second option in question 6 of Section 13.12). That protocol is also why precision falls: the 27 recovered mentions enter the denominator for the first time, and most are wrong. Under the other protocol, counting unalignable mentions as false positives, back-calculating from these numbers gives roughly 59.9 F1 for the naive aligner and 60.2 for the corrected one. The difference stays under a point, but its direction flips: the recovered mentions were already counted as false positives there, and four of them become true positives.

The four genuinely unalignable mentions are all suffix-boundary confusions rather than invention: the model returned Çin Halk Cumhuriyeti where the text has Çin Halk Cumhuriyetini with the accusative fused on and no apostrophe to split at, and Niksar'ın where the text has Niksar'da — the right name, the wrong case.

The results

Same 1,000 test sentences for every system, 1,373 gold entities, scored the same way:

qwen3.5:9b against the taggers, on the identical slice. The LLM row uses the naive aligner; the corrected one scores 60.29
system labels used token acc span P span R span F1
Gazetteer 20,000 sentences 76.12 60.28 50.40 54.90
LLM, zero-shot none 82.87 60.33 60.82 60.57
CRF 20,000 sentences 94.42 84.94 83.39 84.16
BERTurk fine-tuned 20,000 sentences 96.98 91.11 91.77 91.44

Note which of the two numbers is in that row, because a chapter about auditing should not quote the flattering one silently. 60.57 comes from the aligner before the apostrophe and wiki-markup fixes; the corrected aligner gives 60.29. The gap is 0.28 and changes nothing here — but that is exactly the situation in which a headline number gets carried forward without anyone checking which alignment produced it.

Two readings, and both matter.

With no training data at all, the LLM beats the gazetteer by six points. That is a real capability and it did not exist in 2019. If you have a schema and no labels, this is a working extractor by tomorrow.

It also loses to an eighteen-second CRF by 24 points, and to the fine-tuned encoder by 31. Zero-shot extraction is where you start, not where you stop, and the gap is far larger than the marketing around structured output suggests.

The per-type breakdown says where it goes wrong:

             LLM              BERTurk @ 20,000
        P      R     F1        P      R     F1
  PER 64.63  82.00  72.29    92.97  95.32  94.13
  LOC 59.01  59.24  59.13    92.12  93.93  93.02
  ORG 54.67  40.10  46.26    90.46  87.97  89.20

ORG recall of 40.10 is the headline failure: the model misses three organisations in five. This is the same weakness every system in this chapter has, amplified — but it also has a cause the taggers do not share. The model is not failing to find the strings; it is disagreeing about whether Değişim Hareketi is an organisation, and about where its name stops. A tagger learns that boundary convention from 20,000 examples of it. A prompt has to state it, and one page of guidelines cannot encode what annotators converged on over a corpus.

PER shows the mirror image: recall 82.00 against precision 64.63. The model finds people eagerly and over-commits, tagging descriptions and titles that the guidelines exclude.

What it costs

So far the LLM has been judged on accuracy alone, where it lost by 31 points. The second axis is worse for it, and by a margin that is easy to underestimate until you see it written down:

                              LLM (qwen3.5:9b)     BERTurk fine-tuned
per sentence                        1.26 s              0.00103 s
1,000 sentences                    1,265 s                    1.0 s
generated tokens per sentence          26.4                    n/a
malformed JSON responses                  0

The LLM is roughly 1,200 times slower per sentence — and that is a 9B model quantised to 4 bits, running locally, with thinking disabled and a schema that keeps the output to 26 tokens. A hosted frontier model would probably be more accurate, but each request would still take on the order of a second, not a millisecond; you can buy throughput by sending requests in parallel, but you pay for every one of them.

For a thousand sentences that difference is nothing. For the ten million sentences an extraction pipeline can easily have to get through, 1.26 seconds each is 146 days of wall clock on one machine. The decision rule from Section 12.9 applies unchanged: prompt at low volume, distil into a tagger when the volume arrives.

The right middle option is missing from that table on purpose, because it deserves naming rather than a row: use the LLM to label, then train the tagger on its output. At 60.57 F1 the model is a poor extractor and a serviceable annotator — it gives you thousands of noisy labels overnight, which Section 13.6’s curve says is exactly the regime where a fine-tuned encoder starts to work — though that curve was measured with gold labels, and a tagger trained on 60-F1 labels inherits their errors. Chapter 5 is the chapter on doing that without fooling yourself.

The option this section skipped

The comparison above is between a general-purpose LLM and a trained tagger, and it leaves out the family built precisely for the gap between them. GLiNER (Zaratiana et al. 2024) is a BERT-sized encoder trained to match free-form type names against candidate spans, so it does zero-shot extraction over an arbitrary label set at encoder latency rather than generation latency. UniversalNER (Zhou et al. 2024) takes the instruction-tuned route to the same target, and NuNER (Bogdanov et al. 2024) pre-trains a 125M encoder on LLM-generated annotations. Published comparisons put a 90M GLiNER in the same zero-shot range as a 13B UniversalNER.

So the honest shape of the zero-label decision is three options, not two:

The zero-label options
option labels latency when
prompt an LLM under a schema none ~1.3 s/sentence prototyping, long-tail fields, conditional schemas
GLiNER-family zero-shot none tagger-scale any flat entity schema, at volume
fine-tune an encoder 2,000+ tagger-scale a stable schema you can afford to annotate

This chapter measured the first and the third. The middle row is the obvious next measurement, and the one to run before you accept the 1,200× latency penalty as the price of having no labels.

13.9 Where BIO stops working

BIO assigns one label per token. That single sentence contains the format’s whole set of limitations, and real schemas hit all of them.

Nested entities. Türkiye Cumhuriyeti Merkez Bankası is an ORG, and Türkiye inside it is a LOC. One token, two correct labels, and BIO has room for one. Practically everyone drops the inner one and never mentions it; biomedical and legal schemas, where nesting is the norm rather than the exception, cannot.

Overlapping entities. Two spans that share tokens without one containing the other — common in relation-heavy schemas and in clinical text.

Discontinuous entities. sol ve sağ ventrikül (“left and right ventricle”) mentions two structures, and the first, sol … ventrikül, is a span with a hole in it. Ordinary in clinical notes, inexpressible in BIO.

The alternatives all give up the one-label-per-token constraint:

Beyond one-label-per-token
approach idea cost
Span enumeration score every candidate span up to length k, classify each \(O(nk)\) candidates, which is \(O(n^2)\) only when k grows with n; a strong, simple baseline
Biaffine / dependency-style score every (start, end) pair with a biaffine layer (Yu et al. 2020) handles nesting natively; the standard strong result
MRC framing ask “where are the organisations?” as a question per type (Li et al. 2020) one forward pass per type; brings the type description into the model
Word-word grid tagging classify the relation between every token pair — next-neighbour, tail-head (Li et al. 2022) handles nesting and discontinuity in one formulation
Generative (sequence-to-sequence) emit the records directly as text no span algebra at all — and back to the alignment problem above

GLiNER-family models (Section 13.8) also return overlapping spans natively, which makes the zero-shot route the cheapest way to find out whether your schema needs a nesting-aware trainer at all.

How much you stand to lose is measurable before you decide: in the corpora where nesting is annotated at all — GENIA, ACE — somewhere between a sixth and a third of mentions are nested, so dropping the inner one is not a rounding error. If your schema has nesting and your pipeline uses BIO, you have already decided to lose those entities. The decision is usually accidental, inherited from a tutorial. Make it deliberately.

13.10 Relations and events

Extracting Ali and Acme is half a record. The other half is that Ali works at Acme.

The standard framing is pairwise classification: take every pair of mentions in a sentence, and classify the relation between them, including a no relation class that dominates the label distribution. It works, and with \(m\) mentions in a sentence it means \(m(m-1)\) ordered pairs to classify — \(O(m^2)\), and one forward pass each if, as is common, you mark the two mentions in the input text. The cost grows with pairs for the same reason span enumeration in Section 13.9 grows with candidate spans: every candidate is scored separately.

The three things worth knowing before starting:

  1. no relation is the overwhelming majority class, so everything Section 12.5 said applies here with more force. Report micro-F1 excluding no relation, and expect the raw accuracy to be uninformative.
  2. Annotation cost dominates everything else. Entity annotation is roughly linear in tokens; relation annotation is quadratic in entities, and the guidelines are far harder to make reproducible — two annotators who agree on PER and ORG routinely disagree on whether a sentence asserts employment. TACRED (Zhang et al. 2017) is the reference dataset and the reference cautionary tale: re-annotating its development and test sets moved reported F1 by roughly eight points (Alt et al. 2020), which measures the annotation noise rather than any model.
  3. Two limits worth stating out loud. The pairwise framing above is sentence-level, and a large share of the relations in real documents cross sentence boundaries — that is the document-level setting, and it is meaningfully harder. And the framing assumes the mentions are already found, so every entity error upstream becomes a relation error downstream.

Event extraction adds a trigger word and a set of typed arguments, which is the same machinery with more roles and worse agreement. For most applied work in 2026 the honest advice is to try schema-guided generation first (Section 13.8), measure it, and only build the pairwise classifier if you can afford the annotation to beat it.

13.11 Linking and normalization

Finding Türkiye’nin is recognition. Knowing that it refers to the same thing as Türkiye’de, Türkiye’deki and Türkiye is entity linking, and in an agglutinative language it starts one step earlier than the textbooks assume.

entity linking — mapping a mention to a canonical identifier, so that different surface forms of the same thing collapse to one record. Separate from recognising that the mention is an entity at all. ↩︎

Measured on this corpus’s training half:

entity mentions                             26,482
  carrying an apostrophe suffix              4,099   15.5%

distinct surface forms                      16,591
distinct after stripping the suffix         14,876   10.3% collapse

most common suffixes on mentions
  -da   587    -nın  352    -de   328    -ın   233
  -in   227    -a    201    -nin  195    -ya   182

One mention in six carries a suffix (mostly a case suffix) attached with an apostrophe. And the surface forms multiply per entity:

base forms appearing in more than one surface form   1,122

  ABD       -> ABD, ABD'de, ABD'deki, ABD'li, ABD'nin, ...        13 forms
  Fransa    -> Fransa, Fransa'da, Fransa'daki, Fransa'dan, ...    12 forms
  Türkiye   -> Türkiye, Türkiye'de, Türkiye'deki, ...             12 forms

Count mentions of ABD by exact string and you get thirteen separate “entities”, each with its own smaller count, instead of one entity with their total. Every aggregate you build on top — a frequency ranking, a co-occurrence graph, an alert on “mentions of X this week” — is wrong in a way that looks plausible.

The Turkish-specific first step is cheap and worth doing before anything clever: split on the apostrophe and keep the stem. It is not a general solution — plenty of inflected mentions carry no apostrophe, and the apostrophe convention is not universally followed — but it cuts the number of distinct surface forms by 10.3% for one line of code.

Beyond that, the pipeline is the standard one: generate candidate entries from a knowledge base by string similarity, then rank the candidates with context. Classical systems did this with a graph over co-occurring mentions (Hoffart et al. 2011); the modern version encodes mention-in-context and entry-description into the same vector space and retrieves (Wu et al. 2020) — which is Chapter 3’ machinery, pointed at a different problem. Chapter 19 builds that retrieval step properly.

The evaluation trap here mirrors Section 13.2’s. Linking accuracy measured on mentions whose surface form appears in your knowledge base is not linking accuracy; it is dictionary coverage. Report the two separately, and report how often the correct entry was not a candidate at all — that number is a ceiling nothing downstream can raise.

13.12 Intuition checks

Answer these before reading the answers.

You cannot tell, and the number is designed to stop you asking. On this corpus a model that outputs O for every token scores 61.36%, so 96% represents 35 points above the floor, not 96 points of competence. On a realistic corpus where most sentences contain no entity at all, the floor is higher still — 97% or more is routine — and a model can reach it while finding nothing.

Ask for span-level F1. If the answer is “we don’t have that number”, the model has not been evaluated.

Boundaries. Token-level F1 gives partial credit: three correct tokens out of a four-token organisation score three quarters. Span-level requires the type, the first token and the last token all to match, so that same prediction scores zero.

Do not read the gap as a boundary-error rate, though. It mixes several things: partial overlaps, spans of different lengths counted token by token, and type errors on long entities, all micro-averaged so that a four-token mistake weighs four times a one-token one. It is a signal that boundary errors are present, not a measurement of how many. If you want the number, count the error types directly — exact, boundary-only, type-only, spurious, missed — which is a twenty-line script and a far more actionable table. Report span-level either way, because no downstream consumer can use three quarters of a company name.

Nobody’s — it is the expected behaviour of a softmax tagger, which picks the best label at each position independently with nothing coupling position 4 to position 5. The best fine-tuned BERTurk in this chapter emits 139 such transitions across 10,000 sentences, and one trained on only 500 sentences emits 3,739.

What is your decision is what happens next. seqeval will silently recover an ORG span; a strict decoder will drop it; a repair pass will rewrite the first I- to B-. All three are defensible and they give different F1. The bug is not the illegal transition, it is not knowing which of the three your evaluation is doing.

Worth knowing how the count moves: it peaks at 3,739 at the 500-sentence budget, falls to 139 at the full budget, and never reaches zero. Well-formedness is something a softmax tagger learns approximately from data. A CRF scores whole sequences, so it rarely breaks the scheme, but a finite transition score is not a guarantee. Removing the forbidden transitions from the Viterbi search at decode time is.

Measured, on this corpus: fine-tuning on 200 sentences with the full-budget recipe (three epochs, 39 updates) gives 3.14 span F1. A dictionary of memorised strings gives 55.26 — but that dictionary was built from all 20,000 sentences; one built from your 200 would hold a few hundred phrases and score lower, which this chapter did not measure. A longer schedule would do better; how much better is the measurement to make before you decide.

At that budget the encoder has seen a few hundred entities across six entity tags plus O, with a randomly initialised head, and what it learns is to say O — its token accuracy of 63.43% is barely above the 61.36% floor. The crossover against a CRF trained on the full 20,000 sentences arrives somewhere between 2,000 and 5,000 labelled sentences: at 2,000 the encoder is still a point behind (84.23 against 85.45) and by 5,000 it is nearly three ahead.

So: gazetteer first if you have a ready-made list, and fine-tune once you are past a few thousand labelled sentences. In between, a CRF is cheap and the first thing to try, but this chapter did not measure a CRF trained on a small budget; check that ordering on your own data. The general lesson is that “just fine-tune a transformer” is advice with a budget attached, and the budget is rarely stated.

Check for leakage before you celebrate. In WikiANN Turkish, 13.06% of test sentences appear verbatim in the training set, 1,305 of them with identical gold labels. A memorising model gets those free.

The diagnostic takes four lines: hash your test inputs, intersect with your training inputs, and score the leaked and clean halves separately. In this chapter the CRF scored 99.81 on the leaked half against 83.56 on the clean half. If your gap looks like that, you measured your dataset, not your model.

It counts as a false positive, and the important part is that it must count as something. Span metrics are defined over token offsets, and a string with no offsets has nothing to compare against a gold span — which makes it tempting to drop the row from the evaluation entirely. Do not: silently discarding unalignable output prices invention at zero and flatters the extractor. Either score it as a false positive against precision, or report an explicit invalid-output rate alongside F1. Say in the protocol which you did.

But before you write “hallucination” in the report, check whose bug it is. Measured here: of 1,419 returned mentions, 31 would not align — and 27 of those were the aligner’s fault, 20 from wiki markup in the corpus and 7 from a curly apostrophe normalised to a straight one. Only 4 in 1,419 (0.28%) were genuinely not in the text, and even those were case-suffix confusions rather than invention.

So: count them, report the count, and split it by cause. It is a failure mode a tagger structurally cannot have — a tagger can only label tokens that exist — but most of what looks like it is preprocessing.

13.13 Interview questions

On sourcing — the same caveat as Chapter 2. Pages claiming to list “the NER questions Google asks” are search-engine content with nothing behind the claim. What follows are questions that recur in the two credible open sources this book uses throughout — Chip Huyen’s Introduction to Machine Learning Interviews (Huyen 2021) and the AIMLInterviews repository — plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the task

Q: Why is NER not just token classification?

Because the output is spans and the model produces labels, so there is a decoding step between them — and the decoding step is where the ambiguity lives. Say what BIO buys (adjacent same-type entities become separable), what it cannot express (nesting, overlap, discontinuity), and what a decoder does with a malformed sequence. A candidate who says “it’s just classification per token” has not shipped one.

Q: What is the CRF layer for, and do you still need it on top of BERT?

The CRF models dependencies between labels — it learns a transition score for every label pair and decodes with Viterbi, so it returns the best whole sequence rather than the best label at each position. That makes an illegal transition like O → I-ORG costly, or at least unsupported, against the whole sentence — say costly, not impossible, because the transition scores are finite.

On top of a strong pre-trained encoder the gain is small, because the encoder already sees the whole sentence and the transitions are mostly learnable from context. Reported differences on top of strong encoders are typically under a point on CoNLL-style benchmarks, but measure it on your own schema rather than take that on trust — we did not measure it here. And say what the cheaper option is: a transition mask over the logits plus Viterbi (Section 13.3) rules out the illegal transitions at decode time with no extra parameters and no retraining — and note that a guarantee needs those transitions removed from the search space rather than merely penalised.

Q: How would you evaluate an NER system?

Span-level micro and macro F1, exact match, reported per type. Then say what you would check before believing it: the everything-is-O floor, the overlap between test and training inputs, and which decoding convention the score assumes. Anyone who answers “F1” and stops has evaluated nothing.

Testing whether you have shipped one

Q: Your NER model does well offline and badly in production. Walk through it.

Ordered by cost, and the first four are not the model:

  1. Tokenization mismatch. Offline you evaluated on pre-tokenized gold tokens; production tokenizes raw text with a different splitter, so the spans do not line up. This is the most common cause and the easiest to miss.
  2. Alignment. Labels are per word, model inputs are per subword. Print one production sentence beside its aligned labels and read it.
  3. Domain. Wikipedia fragments are not support tickets. 65.5% of the sentences this chapter trained on do not even end in punctuation.
  4. Leakage in the offline number. See above; it inflates the memorising parts of your model most.
  5. Only then, the model.

Q: You need to extract 30 fields from contracts and have no labelled data.

Start with schema-guided generation, because the alternative is months of annotation before you learn whether the task is feasible. Then say how you would know it works: hand-label a couple of hundred documents as an evaluation set (evaluation data is far cheaper than training data), align every returned string back onto the source, and count the ones that will not align. Move to a trained extractor for the fields that are high-volume and stable; keep generation for the long tail.

The answer to listen for is the evaluation set. Candidates who describe the prompt and stop have not run one of these in production.

Q: How do you handle an entity type that appears twice in your training data?

You do not train it. Write a gazetteer or a rule, ship that, and collect examples until there is something to train on. The measurement in this chapter — 3.14 F1 at 200 sentences — is the general shape of the answer: below a certain budget, learned models are worse than lists, and knowing where that line is for your task is the actual skill.

13.14 The 2026 toolbox

Everything above ran on one laptop with the versions below.

The stack these numbers came from. Python 3.14.5, Apple Silicon
library version used what it is for here
datasets 5.0.1 loading WikiANN in one call, and the audit in Section 13.2
sklearn-crfsuite 0.5.0 the linear-chain CRF and its Viterbi decoding
transformers 5.15.1 BERTurk, AutoModelForTokenClassification, and word_ids() for the alignment
torch 2.13.0 the MPS backend the fine-tuning ran on
seqeval 1.2.2 the reference span-level scorer, used to cross-check the explicit implementation above
ollama 0.24.0 serving qwen3.5:9b for Section 13.8

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
spaCy 3.8.16 the pragmatic default for production NER pipelines: fast, has a Turkish language class and tokenizer (no official trained Turkish pipeline; trained ones are community-built), and ships tokenizer, tagger and entity ruler as one deployable object
spacy-llm 0.7.4 the schema-guided path of Section 13.8 inside a spaCy pipeline, so you can mix a prompt-driven component with a trained one
GLiNER 0.2.28 zero-shot NER from type names rather than a prompt (Zaratiana et al. 2024) — encoder latency instead of generation latency; see Section 13.8
nervaluate 1.2.1 the MUC-style partial-credit breakdown when exact match is the wrong contract for your consumer
Label Studio / doccano 1.23.0 / 1.8.4 span annotation interfaces; the bottleneck in every project in this chapter is labelled data, not modelling
pytorch-crf 0.7.2 a CRF layer to put on top of a transformer, when you want the label grammar learned inside the model — its transition scores are free parameters, so constrained decoding is still what guarantees well-formedness
outlines / jsonformer 1.3.3 / 0.12.0 grammar-constrained decoding you control, rather than trusting a server flag that may be silently ignored (Chapter 18)
zeyrek 0.1.3 Turkish morphological analysis, for the normalization step of Section 13.11 when the apostrophe heuristic is not enough

GLiNER deserves the second look its position in that table does not give it — Section 13.8 gives it one. It is a small encoder trained to match type descriptions against spans, so you get zero-shot extraction over an arbitrary label set at roughly tagger latency rather than LLM latency. We did not measure it, which is the reason this chapter’s zero-shot section has a 1,200× number in it.

13.15 Lab: extraction on a budget you actually have

The premise of this lab is the measurement in Section 13.6: at small label budgets the learned model loses to a list built from 20,000 sentences; whether it loses to one built from your own training half is what this lab measures. The lab is therefore not “fine-tune a transformer” — it is “find out what your budget buys, before spending it”.

Pick a domain where you can produce labelled sentences yourself: support tickets, job ads, match reports, a bibliography. Choose three to five entity types that a second person could label the same way.

Step 1 — write the guidelines before labelling anything

One page. For each type: a definition, three positive examples, and three near-misses that are not that type. Chapter 5’s argument in one sentence: if the rule is not written down, the labels are not reproducible, and the model is learning whatever mood you happened to be in on the day you labelled.

Step 2 — label 300 sentences, and set aside 150 of them as test

Not 30. The evaluation set is the part you cannot skip, and it is worth more than the training half at this budget — you can improve a model without more training data, but you cannot measure one without evaluation data.

Label the test half last, and do not look at it again until step 6.

Step 3 — compute the floor and the gazetteer

Everything-O, then longest-match over the strings in your training half. Two numbers, half an hour, and they are the two numbers every later result has to beat. Expect the gazetteer to be embarrassingly competitive.

Step 4 — train a CRF, and check whether the features are doing anything

Use the feature template from Section 13.5. Then ablate: drop the suffix features, drop the context window, drop the shape feature, and record the F1 each time. The point is not the final score; it is knowing which part of your model is load-bearing.

Step 5 — fine-tune an encoder at three budgets

50, 100 and 150 sentences from your training half. Plot span F1 against budget. You are reproducing Section 13.6’s curve on your own data, and the thing to look for is where — or whether — it crosses the CRF.

Step 6 — the schema-guided extractor, honestly

Put your guidelines in the prompt, constrain the output to a schema, and verify the constraint is really applied by sending one request whose schema the model would otherwise violate. Align every returned string back onto the source. Report three numbers: span F1, the fraction of returned mentions that would not align, and seconds per document.

Step 7 — open the test set once

Score all five systems on the 150 held-out sentences. One table, one reading.

What to hand in

The table, plus a paragraph naming which system you would ship and what it would cost per thousand documents. If the gazetteer wins, say so — that is a result, and on 150 labelled sentences it is a likely one.

13.16 Exercises

  1. Hand-decode a broken sequence. Take the tags O B-PER I-PER I-LOC O I-ORG B-ORG over a seven-token sentence. Write out the spans a lenient decoder produces, then a strict one. Which tokens do the two disagree about, and how many spans does each return?

  2. Token versus span F1. Take one system’s predictions and compute both. Explain the gap in terms of boundary errors, and construct a prediction that scores well on the first and zero on the second.

  3. Break the alignment on purpose. Take the alignment code from Section 13.4 and remove the word_ids() call, zipping labels to subwords positionally instead. Train on 2,000 sentences both ways and report both scores. Then find the shortest sentence in your data where the two labelings differ, and print it.

  4. Measure the leakage yourself. Hash the token sequences of any NER corpus you use — CoNLL-2003 (Tjong Kim Sang and De Meulder 2003), your own, or WikiANN in another language — and report the test/train intersection. Then score a memorising baseline separately on the leaked and clean halves.

  5. The suffix ablation. Remove w[-3:], w[-2:] and w.hasapos from the CRF features and re-fit. How much of the 85.45 was Turkish orthography? Predict the answer before you run it.

  6. Normalize the mentions. Take the extracted mentions from any system, collapse them on the apostrophe, and count how many distinct entities you have before and after. Then find a mention the heuristic gets wrong — they exist, and describing one is the exercise.

  7. Nesting, on paper. Write five sentences from your own domain that contain a nested entity. Encode them in BIO, then in a span-enumeration format, and state precisely what BIO lost.

13.17 Where to go deeper

Before anything else: check your benchmark for train/test overlap. It takes a few lines, it is the first thing this chapter did, and it changed how every later number should be read.

Beyond the papers
resource what it is
seqeval the span-level scorer everyone uses; read it to see exactly what “F1” means here, because it is not token accuracy
flair strong sequence-labeling models with a readable training loop, and a good source of baselines
spaCy course free, interactive, and the fastest route to a working rule-plus-statistics extraction pipeline
CMU CS11-711 Advanced NLP lecture materials covering structured prediction with the modelling assumptions kept explicit
Hugging Face token-classification chapter the fine-tuning path for the encoder used in this chapter, including the label-alignment step that causes most bugs
CS224N lecture series — Stanford free lectures covering sequence models and structured prediction, with the assumptions made explicit at the board
The primary sources
topic source
the structured model that scores whole label sequences Lafferty et al. (2001); Viterbi (1967)
the tagging scheme, and the alternatives Ratinov and Roth (2009); Tjong Kim Sang and De Meulder (2003)
neural sequence labeling as it settled Lample et al. (2016)
the multilingual benchmark used here, and its construction Pan et al. (2017); Rahimi et al. (2019)
entities that nest, which BIO cannot express Yu et al. (2020); Li et al. (2020)
memorisation versus generalisation in NER Elangovan et al. (2021)
linking a mention to an actual entity Hoffart et al. (2011)
relation extraction, the task after this one Zhang et al. (2017)
scalable entity linking with dense retrieval Wu et al. (2020)

13.18 What to remember

  • Compute the floor. Everything-is-O scores 61.36% token accuracy and 0.00 span F1 on this corpus, and higher on a realistic one where most sentences contain no entity. Every token-accuracy number in sequence labelling should be read as a distance from that floor, and the distance is usually smaller than the headline suggests.

  • Report span-level F1. Token accuracy 94.41, token-level F1 88.73, span-level F1 85.45 — the same predictions, three answers. Only the last one asks whether the record came out right.

  • Audit the corpus before you trust the score. WikiANN Turkish has 13.06% of its test set verbatim in training, and the raw leakage bonus falls as a system’s tendency to memorise falls: +26.50 for a gazetteer, +16.25 for the CRF (99.81 leaked against 83.56 clean), +7.65 for the fine-tuned encoder. Normalised by the headroom above the clean half, the order changes (the CRF tops it at 99%). That gap is a diagnostic, not a measurement of memorisation: a large one means look harder. Hash your test inputs — it is four lines.

  • The classical baseline is not a formality. A gazetteer built from 20,000 sentences scores 55.26 from a Python dictionary; a CRF with hand features scores 85.45 in 18.5 seconds on a CPU. Fine-tuned BERTurk needs 2,000 to 5,000 labelled sentences to match the CRF trained on 20,000, and reaches 91.92 on the full budget — a real saving, and a real threshold below which the transformer is worse than the 20,000-sentence CRF. A CRF trained at the same small budget was not measured.

  • With the default recipe, 200 labelled sentences bought single-digit span F1 — 3.14 in our single run of three epochs (39 updates), a schedule that starves small budgets, so read the order of magnitude rather than the decimals. At that budget the fine-tuned encoder learned to say O, and a list memorised from 20,000 sentences beat it by fifty points (one built from 200 would lead by less, if at all; we did not measure it). “Just fine-tune a transformer” is advice with a budget — and a training schedule — attached; ask what they are.

  • Decoding is a decision you are making whether or not you know it. A softmax tagger at full budget emitted 139 illegal BIO transitions across 10,000 sentences — and 3,739 at the 500-sentence budget. The 20,000-sentence CRF emitted zero, because Viterbi scores whole sequences and a forbidden transition with no supporting features did not make the winning path. That is an empirical zero, not a guarantee; remove the transitions from the search if you need one. What your evaluation does with those 139 changes your reported F1, and the convention is not stated in most papers.

  • Alignment is where the silent bugs live. Entity words fragment into 1.98 subwords each in BERTurk against 1.45 for ordinary words — names are rare, so tokenizers break them. Get word_ids() wrong and the model still trains, the loss still falls, and the labels are attached to the wrong tokens.

  • A schema guarantees shape, never truth. ollama 0.24.0 silently ignored a JSON schema on /api/chat while enforcing it on /api/generate, and the enforced call happily returned {"answer": 1} for the capital of France. Send one request the model would otherwise fail, and check.

  • Zero-shot extraction is where you start, not where you stop. On the same 1,000 sentences: LLM 60.57 span F1 with no labels at all, gazetteer 54.90, CRF 84.16, fine-tuned encoder 91.44 — and the LLM is about 1,200× slower per sentence. It beats the dictionary and loses to eighteen seconds of CRF by 24 points. Its worst class is ORG recall at 40.10, because a prompt cannot encode the boundary convention that 20,000 annotated examples teach.

  • An LLM returns strings; a metric wants offsets. Every mention has to be aligned back onto the source, and that step is where you can quietly misattribute your own bugs: 27 of 31 alignment failures here were the aligner’s, not the model’s. Fixing it moved F1 by 0.28 — so check it, and do not expect it to explain a bad number.