6  Language Models: From N-grams to Neural Networks

Chapter status: drafted. Every number on this page was produced by running the code written for this chapter, of which only the core pieces (the bigram counts, the neural model and its loss) are shown, on one machine, with PyTorch 2.13.0, NumPy 2.5.2 and pandas 3.0.5 on Python 3.14.5. The n-gram models are plain Python dictionaries and need no libraries at all; the neural model trained in about two minutes on this machine’s laptop GPU. The corpus and the test split are those of Chapter 4; 2,000 of its 40,000 training reviews are held out here as a development set, and the counts come from the remaining 38,000. Its classification scores and the perplexities here are different measures: what the two chapters share is the data, not the numbers. While this chapter was being reviewed, the code was rewritten from the text and run again; the development split could not be rebuilt exactly, so that run’s numbers differ in the third or fourth digit. Numbers marked as coming from the re-run are from it.

Learning objectives. After this chapter you can:

6.1 The question chapter 4 could not answer

Chapter 4 ended on a pair its unigram representations could not tell apart. Two Turkish reviews, four words each, the same words, opposite meanings:

a = "kalite iyi fiyat kötü"      (quality good, price bad)
b = "kalite kötü fiyat iyi"      (quality bad, price good)

cosine, averaged fastText   1.0000
cosine, tf-idf              1.0000

Those are the unigram representations. Both sentences contain the same set of words, so both get the same vector, and no classifier built on top can tell them apart. Chapter 4’s word-bigram model does better on this particular pair — iyi fiyat and fiyat iyi are different columns — but it is still an unordered bag, now of pairs, and it buys that sensitivity by multiplying the vocabulary. Say that carefully: those bigram columns really do encode local order — this is not a representation that has no access to it. But that classifier was trained to predict a label, not the probability of a word sequence or the distribution of the next word. That takes a model with a different objective, and this chapter builds one.

Here is the same pair scored by the models in this chapter — the probability each assigns to the whole sentence, meaning the four words and the </s> marker that says the review has ended, trained on nothing but raw review text:

### log10 P(sentence), Kneser-Ney smoothed
                  log P(a)   log P(b)   P(a)/P(b)
unigram            -12.340    -12.340       1.000
bigram             -12.297    -12.764       2.933
trigram            -12.481    -12.835       2.259

The unigram model reproduces chapter 4’s failure exactly: ratio 1.000, because a unigram model multiplies independent per-word probabilities, so reordering the words cannot change the product. Like a bag of words, it never sees order. The bigram model breaks the tie and prefers sentence a by a factor of about three (2.933).

That factor of three is a small instance of what Part II is about. The new thing is not that order has become visible — bigram features made it visible in Chapter 4. It is that an ordered sequence now has a probability, which is the same object you can use to rank two sentences and to generate the next word, and it came from one structural change: conditioning each word on the one before it. From here to Chapter 10 the models change, and the same question keeps coming back: conditioned on what, exactly, and how is that context represented?

Be precise about what the bigram model actually knows, though, because it is less than it looks. Here are the six counts doing the work:

a: kalite iyi        7     b: kalite kötü       1
   iyi fiyat       112        kötü fiyat        0
   fiyat kötü        0        fiyat iyi        25

Before celebrating, look at why it got them apart.

The model has never seen kötü fiyat or fiyat kötü in 858,000 tokens, so it has no direct evidence about bad prices at all — both sentences lean entirely on the interpolated lower-order term for one bigram. What tips the balance is not one phrase but a tally of collocations, which the bigram surprisals in the widget below spell out: kalite iyi (7 sightings against 1 for kalite kötü) and iyi fiyat (“good price”, 112 against 0 for kötü fiyat) each favour a by about five bits, while the 25 sightings of fiyat iyi — against none of fiyat kötü — win about seven bits back for b, and the end-of-review token another 0.8. The net is about 1.6 bits: the factor of three in the table. (The widget’s values are rounded to one decimal, so these contributions are approximate, and they belong to this model only.)

So what explains the preference is collocation, not meaning. The model prefers sentence a because that word order sounds like the reviews it read. That is a real and useful signal, but it does not show that the model understood what the two reviews say about quality and price. Measuring that would take a separate task.

Hold on to that distinction. It is what the rest of this chapter measures.

What we will look at. First what a language model is, which takes one line. Then we build one by counting and measure how zero probabilities break it on held-out data, and fix it with smoothing — a form of regularisation. Then we ask what the fixed model actually knows, and what perplexity does not measure. Finally we replace counting with a small neural network and compare the two on the same data; the answer turns out to depend on how well each side is tuned.

6.2 What a language model is

A language model assigns a probability to a sequence. That is the entire definition, and it is worth resisting the urge to make it sound harder.

\[ P(w_1, w_2, \ldots, w_n) \]

One detail: as written, that is a distribution over sequences of length \(n\). To give complete reviews of different lengths a probability, the model also has to predict that the sequence ends; below, the </s> marker does that.

chain rule — the rule that lets you swap “how likely is this whole sentence” for “how likely is each word, given the words before it”. As long as the preceding words have non-zero probability, nothing is lost in the swap; the two questions have the same answer. ↩︎

You cannot estimate that directly — the number of possible four-word Turkish sentences exceeds the number of four-word Turkish sentences that have ever been written by many orders of magnitude, so counting them is hopeless. But the chain rule of probability rewrites it without any approximation at all:

\[ P(w_1, \ldots, w_n) = \prod_{i=1}^{n} P(w_i \mid w_1, \ldots, w_{i-1}) \]

This is an identity, not a model. It says a joint distribution over sentences is the same thing as a rule for predicting the next word given everything before it. That is the move that makes the field workable, and it is why most of the rest of this book is about next-token prediction: for a model that generates left to right (an autoregressive model), predicting one word at a time is not a simplification of language modelling, it is language modelling, exactly. It is not the only objective — models such as BERT in chapter 8 are trained to predict masked parts of the text — but every model that generates text uses this factorisation.

The difficulty just moved rather than disappeared. \(P(w_i \mid w_1, \ldots, w_{i-1})\) conditions on a context that grows without bound, and almost every long context you will ever see is one you have never seen before. The n-gram answer is to truncate:

\[ P(w_i \mid w_1, \ldots, w_{i-1}) \approx P(w_i \mid w_{i-n+1}, \ldots, w_{i-1}) \]

Markov assumption — the decision to look at only the last few words and throw the rest of the sentence away. For natural language it is wrong — distant words can matter — and we make it anyway, because it is what turns an impossible count into a possible one. ↩︎

Only the last \(n-1\) words matter. This is the Markov assumption, and for natural language it is false — a good guess for the blank in “the phone I bought from the shop near the station last March finally ___” is helped by the word phone, twelve words back (the things a failing phone does), and a model that looks at the last two words cannot see it. The question is not whether it is false but how much it costs, and that turns out to be measurable.

Three steps from an impossible quantity to a table of counts
P(whole sentence) CANNOT BE COUNTED — EVERY SENTENCE IS NEW EXACT chain rule ∏ P(wᵢ | w₁…wᵢ₋₁) AN IDENTITY. NOTHING LOST. LOSSY Markov assumption P(wᵢ | wᵢ₋ₙ₊₁…wᵢ₋₁) FORGET EVERYTHING OLDER count c(h,w) / c(h) TWO DICTIONARIES THE MODELLING ASSUMPTION IS THE THIRD BOX; THE LAST BOX ESTIMATES IT FROM DATA. n-grams truncate the context. Neural models keep the truncation but learn a smooth function of it. Recurrent models replace it with a running state. Transformers (chapter 7) let every position look at every earlier one within a finite window.

The chain rule is free; the Markov assumption is where the modelling happens. The last box is an estimate: an n-gram model is fitted by dividing one integer by another (the maximum likelihood estimate). That is learning from data too, with counts instead of gradients, and because the counts are finite the estimates are noisy. It is why n-gram models stayed a strong, cheap standard for many years on hardware on which other models were too expensive to train.

Shannon was already doing this in 1951, by hand, with a book and a pencil, to estimate the entropy of English (Shannon 1951). The core of this chapter’s mathematics — n-gram statistics and entropy — is his; the corpus size is the most obvious thing that has changed.

6.3 Counting on 38,000 reviews

That is the theory. Now the smallest possible version of it: count how often each word follows each context, divide, and see what comes out.

The corpus is the Turkish product-review set from Chapter 4, and the 10,000-review test set is deliberately the same one, so that everything you learned there about the data still applies here. There is one addition. Smoothing has hyperparameters, and hyperparameters chosen by looking at the test set are not hyperparameters, they are leakage. So the 40,000 training reviews are split again: 2,000 become a held-out dev slice the model never counts, and the counts come from the remaining 38,000:

smoothing — taking a little probability away from the word combinations you did see and giving it to the ones you did not, so that “never seen” stops meaning “impossible”. ↩︎

held-out data — text the model was not allowed to count. It is where a score measures generalisation: a score on the training text rewards memorisation too, although set beside the held-out score it shows overfitting. The dev set is for choosing settings, the test set for the final evaluation. ↩︎

### the language-modelling setup
reviews counted            38,000
reviews held out (dev)      2,000     smoothing constants chosen here
reviews held out (test)    10,000     read exactly once, at the end
tokens counted            857,987
distinct word forms        72,087
vocabulary (count >= 2)    29,773

Two of the notes in that table state an intention, and neither was kept exactly, so say it up front. The test set was not read “exactly once, at the end”: the test values of the first neural attempt and of the neural sweep are printed below too, and those places are marked. Of the smoothing constants, add-k’s \(k\) and the interpolation weights really were chosen on the dev set; Kneser-Ney’s discount was not, and was kept at the textbook value of 0.75 (the Kneser-Ney section measures what that costs).

That 29,773 is the number this chapter’s count-based sections mean by \(V\): the size of the set of symbols the model can assign probability to, the set its probabilities have to sum to one over. That set is the distribution’s support. The row’s label undersells it: the number adds two symbols to the 29,771 word types that pass the threshold — <unk>, into which the words seen once are folded, and </s>, which marks the end of a review. The count models predict </s> too; the test table below counts 10,000 of those events. The chapter’s own numbers show it: further on, with the count threshold at 1, the vocabulary is 72,089, exactly two more than the 72,087 distinct forms.

The neural model later in the chapter reports a 29,774-way softmax. The extra class is not </s> — that is already inside the 29,773 — but <s>. The network’s input table needs a vector for <s>, and because the same \(V\) sizes the output layer, <s> also appears as an output class that is never a target. The parameter count confirms it: \(161 \times 29{,}774 + 8{,}320 = 4{,}801{,}934\) (per class, 32 embedding weights, 128 output weights and one bias; the hidden layer’s \(2 \times 32 \times 128 + 128\)). The model learns to give a class that is never a target almost no probability, so it does not affect perplexity; it does take up parameters.

The lesson: define the support once, write down which special symbols are in it, and use that same definition for smoothing, for the softmax and for the count of prediction events. Different definitions in one pipeline are one of the most common ways a published perplexity becomes incomparable with anyone else’s.

Two housekeeping decisions do real work later, so they are worth stating plainly.

Every review is padded and terminated. A <s> marker is prepended so the first real word has a context, and a </s> marker is appended so the model has to predict that the review ends. Without the terminator the probabilities do not sum to one over sentences of all lengths, and generation stops only at a length limit imposed from outside, because the model has not learned when to stop.

Words seen once are replaced by <unk>. Chapter 4 measured 58.6% hapax on its own 40,000-review split — a different sample from this one, so take it as the scale of the problem rather than as this corpus’s number. Here the threshold takes 72,087 distinct forms down to 29,771 words, so more than forty thousand forms would otherwise have had a probability estimated from a single observation each. Mapping them to a single “unknown word” symbol gives the model a way to assign non-zero probability when a word outside the vocabulary arrives — but it is the probability of the <unk> class. The model cannot tell which unknown word it is (uyuyakaldı, or anything else). It also has a consequence we will come back to, at some length, because it is the most common way published perplexity numbers are made meaningless.

### the test set, through that vocabulary
test tokens                229,055
test tokens -> <unk>        17,003     7.42%
distinct test forms         32,573
  not in the vocabulary     15,352     47.1%
prediction events          239,055     229,055 words + 10,000 </s>

Not being in the vocabulary is not the same as never having been seen: some of those 15,352 forms occurred once in training and were folded into <unk> at the threshold. The forms never seen at all are, in the re-run, 11,151 of the 32,573 (34.2%).

Now the model itself. A bigram model is two dictionaries and a division:

from collections import Counter

ctx, ngr = Counter(), Counter()
for s in train_seqs:                       # each already padded
    for i in range(1, len(s)):
        ctx[s[i - 1]] += 1
        ngr[(s[i - 1], s[i])] += 1

def p_mle(prev, w):
    # prev is always <s> or a vocabulary word (including <unk>); each occurs
    # as a context in training, so ctx[prev] > 0 and the division is defined
    return ngr[(prev, w)] / ctx[prev]

That is the core of statistical language modelling as it stood around 1990, and it is worth looking at what it produces. Here is the model’s distribution over the word that follows kargo (“shipping”):

### P(next | kargo), bigram MLE
ve             167   0.0676   (and)
için           163   0.0659   (for)
hızlı          105   0.0425   (fast)
bedava         103   0.0417   (free)
da              93   0.0376   (also)
çok             83   0.0336   (very)
</s>            82   0.0332   (review ends here)
ile             81   0.0328   (with)

context count 2,472; 516 distinct continuations observed

Nobody wrote a grammar. The model has discovered that kargo is followed by function words, by adjectives about speed and cost, and — 3.3% of the time — by the end of the review. It is a description, extracted by counting, of how the reviewers in this corpus write; it is not direct evidence about Turkish grammar or Turkish usage in general.

Because otherwise the model is not a distribution over sentences.

Suppose you drop </s>. Then \(P(\text{"kargo hızlı"})\) and \(P(\text{"kargo hızlı geldi"})\) are both products of conditionals, and the shorter one is necessarily larger, since the longer is the shorter multiplied by another probability below 1. Both are really the probabilities of prefixes, not of finished sentences. Sum over all sequences of all lengths and you get infinity, not one. The model has no way to express “and then it stopped”.

Practically it shows up the moment you generate: without a terminator, unless you impose a length limit from outside, your sampler runs until you cut it off, and every sentence it writes is a fragment wherever it stops. </s> is what a generation loop is waiting for; the end-of-turn token of a modern chat model does a similar job.

6.4 Perplexity

We now have a model that assigns probabilities to sentences. How do we say whether it is any good?

We need a single number, and it should not require a downstream task, because the whole point is that the model was trained without one. Plain next-word accuracy is a poor choice too: many continuations are acceptable, and accuracy only asks whether the model’s top guess was right and makes no use of how much probability it gave to the word that actually came. What we can ask instead is how surprised the model was by text it had not seen, and there is a standard way of putting a number on that.

perplexity — the exponential of the average negative log probability per token on held-out text. Lower is better. The intuitive reading: a perplexity of 10 means the model gives the observed words as much probability as someone picking uniformly from a shortlist of ten would; not that there are really ten options at every step. ↩︎

The natural quantity is the probability the model assigns to text it has never seen. Held-out probability is a product of hundreds of thousands of small numbers, so we work with the average log instead, and then exponentiate it back into something interpretable:

\[ \mathrm{PPL} = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(w_i \mid h_i)\right) \]

Here \(N\) is the number of prediction events and \(h_i\) is the context (history) before event \(i\). The events are the words and the </s> at the end of each review; on this chapter’s test set \(N = 239{,}055\). The formula is the geometric mean of \(1/P(w_i \mid h_i)\) — the reciprocal of the geometric mean of the probabilities the model gave to the events that actually came.

The base does not matter as long as you are consistent: \(\mathrm{PPL} = \exp(H_e) = 2^{H_2}\), where \(H\) is the average negative log probability per token — which is the model’s cross-entropy against the held-out text. So minimising perplexity, maximising held-out log-likelihood and minimising cross-entropy are three names for one operation, and the pretraining loss of the language models in the rest of this book has the same form before the exponential: the average negative log probability of the next token. What changes is the text it is measured on and the tokenizer; a training loss and a held-out value, or two values measured with different tokenizers, cannot be compared directly. The widget below reports \(H_2\) in bits, one token at a time; the tables report \(\exp(H_e)\).

Two words in that paragraph deserve a line each. Cross-entropy \(H(p, q) = -\sum_x p(x) \log q(x)\) measures how costly it is to describe data that really comes from \(p\) (here, Turkish reviews) using a model \(q\); the held-out average above is simply its estimate from a sample. It can never be lower than the entropy of \(p\) itself. As long as the next word of a review is genuinely uncertain, which it is, even a perfect model keeps a perplexity above 1; a perplexity of 1 is possible only in a language where every next event is determined. A bit is the information in one fair yes/no choice: \(H_2 = 7\) bits per token means that, on average, the model is as uncertain as if it had to answer seven fair yes/no questions to pin down each word, i.e. choose among \(2^7 = 128\) equally likely options. That is an analogy too: the real distribution is not uniform like that at every position.

Read it as an effective branching factor. A model with perplexity 100 gives the observed words, on average (a geometric average), as much probability as a model choosing uniformly among 100 words at every position would. The real model is not like that: almost certain at some positions, spread over thousands of options at others; perplexity averages them. A fair six-sided die has perplexity 6. A model that assigns the correct next word probability 1 every time has perplexity 1. Lower is better, and the units are “how many equally likely options am I effectively choosing between”.

Take a three-token sequence and a model that assigns

\[ P(w_1 \mid h_1) = 0.5, \quad P(w_2 \mid h_2) = 0.25, \quad P(w_3 \mid h_3) = 0.125 \]

The average negative log probability is

\[ -\tfrac{1}{3}\left(\ln 0.5 + \ln 0.25 + \ln 0.125\right) = -\tfrac{1}{3}\ln(0.015625) = 1.386 \]

and \(\exp(1.386) = 4.0\). So the perplexity is 4, which is the geometric mean of 2, 4 and 8 — exactly the reciprocals of the three probabilities.

Two properties fall out of this that matter in practice.

Perplexity is a geometric mean, so one catastrophic token dominates a short sequence. Replace the third probability with \(10^{-6}\) and the perplexity jumps from 4.0 to 200 — the geometric mean of 2, 4 and \(10^{6}\). Two tokens out of three were still predicted well, and the score moved by a factor of fifty. On a long test set a single small probability is divided by \(N\): over 239,055 events one event at \(10^{-6}\) moves perplexity by less than a tenth of a percent. Zero is different: replace it with 0 and the perplexity is infinite, regardless of how well the model did on every other token in the corpus. The next section is entirely about that fact.

Perplexity is per-token, so it depends on what a token is. A model that splits the same text into characters instead of words predicts different events from different contexts, and \(N\) changes too; the number it produces shares no common scale with the word-level one. This is not a subtlety; it is the reason two perplexity figures from two papers usually cannot be compared at all.

Where did the model get surprised?

Each bar is one token’s surprisal, \(-\log_2 P(w \mid h)\), measured on the models built in this chapter. The models are Kneser-Ney smoothed (smoothing comes a few sections later), which is why a bigram never seen in training, such as fiyat kötü, gets 13.9 bits rather than infinity. A bar of 10 means the model gave that word a probability of about 1 in 1,000. The bars are rounded to one decimal; the sentence perplexity was computed from unrounded values. Switch the context length: most bars shrink, a few grow, and that is what the extra context gained and lost.

Three things are worth hunting for. In fiyatına göre… the word göre drops from 8.2 bits to 0.2 once the model can see fiyatına — in Turkish göre is a postposition that almost has to follow that word, and one word of context is enough to know it. In kalite kötü fiyat iyi the surprisal rises with more context, because that order is rare in the reviews of this corpus — which shows that the model has seldom seen it, not that Turkish does not use it. And in the last sentence uyuyakaldı is not in the vocabulary at all, so the model scores <unk> instead and finds it unsurprising, at 2.5 bits: that is the score of the class of unknown words, not of uyuyakaldı. Hold on to that.

6.5 The catastrophe: unseen n-grams and infinite perplexity

So far everything has worked on a sentence we chose. The test set contains sentences nobody chose. Let us see whether the same model hits zero when it scores them.

maximum likelihood estimation (MLE) — the estimate that makes the observed data as probable as possible; for an n-gram model with a seen context, one raw count divided by another, with nothing added. It fits the training text as tightly as possible and gives every unseen continuation zero, and on held-out data those zeros are the problem. ↩︎

Now run the maximum-likelihood model, the one defined by nothing but \(c(h,w)/c(h)\), on the held-out reviews.

### test perplexity, maximum likelihood
unigram      1,303.6
bigram             ∞
trigram            ∞
4-gram             ∞
5-gram             ∞

Not “large”. Infinite, and infinite for a mundane reason: some test bigram was never seen in training, so the model assigns it probability exactly zero, so the log is \(-\infty\), so the average is \(-\infty\). One unseen word pair anywhere in 239,055 predictions is enough to destroy the score for all of them.

And it is not one. It is this many:

### test predictions the MLE model assigns probability zero
bigram        73,225 of 239,055     30.6%
trigram      161,544 of 239,055     67.6%
4-gram       201,599 of 239,055     84.3%
5-gram       214,122 of 239,055     89.6%

A trigram model trained on 858,000 words of Turkish reviews gives zero probability to two-thirds of the test events, each after its own two-word context. That says not that those words are impossible in Turkish but that those triples were never counted in training; to MLE, the two are the same thing.

The reason is not a bug, and not a gap that more data will close. More data would make some of these particular triples visible, but new text keeps bringing new ones, because the number of possible n-grams grows like \(V^n\) while your corpus grows like \(N\), so the moment \(n\) increases the counts you are dividing by become tiny. Look at what happens to the model as \(n\) grows:

### how the count table behaves, 895,987 training events
          distinct    a new type      seen exactly
n         n-grams     every…          once
1           29,773     30.1 tokens      0.0%
2          383,579      2.3 tokens     78.9%
3          674,067      1.3 tokens     91.8%
5          818,339      1.1 tokens     98.0%

By \(n = 5\), 98% of the n-grams in the training data occur exactly once, and 91.3% of the tokens you read introduce a five-gram you have never seen before. At that point the high-order count table looks less like a model that generalises and more like a record of pieces of the corpus; a little further down we measure what that does to sampling. (Unigrams show 0.0% singletons only because we already mapped those to <unk>.)

More context, less evidence — the trade every n-gram model makes
OF 239,055 TEST PREDICTIONS, HOW MANY HAD BEEN SEEN BEFORE? 0%25% 50%75% 100% n=2n=3 n=4n=5 context seen the model has an estimate at all whole n-gram seen the estimate is not zero MEANWHILE THE TABLE YOU MUST STORE KEEPS GROWING n=2383,579 n=3674,067 n=5818,339 ENTRIES, AGAINST 895,987 TRAINING EVENTS — THE TABLE IS APPROACHING THE CORPUS ITSELF The bind Longer context is more informative and less observed, at the same time. Every n-gram model is a choice of where on that curve to stand; the dev set picks the place. SMOOTHING STANDS IN TWO PLACES AT ONCE

The teal line is the fraction of test predictions for which the model has any estimate; the rose line is the fraction for which that estimate is non-zero. Both fall off a cliff. Everything above the rose line is what smoothing has to cover: in the gap between the lines the context is known but the continuation is new, and above the teal line even the context is new. At \(n=5\) the full five-gram was seen in training for only 10.4% of the events; for the rest, a smoothed model relies on the lower orders’ estimates.

6.6 Smoothing is a form of regularisation

Every statistician reading this has already diagnosed the problem: an unbiased estimator with catastrophic variance, fitted to sparse counts. The cure is the usual one. Give up a little bias to buy a large reduction in variance, and above all stop the estimate from ever reaching zero. That is exactly what smoothing does, which is why you can read it as a form of regularisation. The two words are still not synonyms: smoothing is a family of regularisers specific to count tables.

Add-one, the textbook first idea, and why it is a disaster on this corpus. Pretend you saw every possible continuation once before you started:

\[ P_{+1}(w \mid h) = \frac{c(h, w) + 1}{c(h) + V} \]

For the statistically minded: it is the posterior mean under a symmetric Dirichlet prior with \(\alpha_i = 1\) for every word — a flat prior over all possible next-word distributions, which on average treats every word as equally likely. Because each of the \(V\) words gets one pseudo-count, the total concentration is \(V\), not 1, and the MAP estimate under that prior would be the unsmoothed MLE instead. The \(V\) here is the whole support: 29,771 words, <unk> and </s>, 29,773 in all. It is perfectly respectable, and on this corpus it does worse than a model that uses no context at all:

### test perplexity, add-one
unigram      1,313.8
bigram       3,216.5
trigram     13,221.2
5-gram      22,234.0

The bigram model with add-one smoothing is two and a half times worse than ignoring context entirely. The arithmetic shows why. After kargo, with \(c(h) = 2{,}472\), \(V = 29{,}773\) and only 516 continuations ever observed, add-one leaves those 516 just \((2{,}472 + 516)/32{,}245 = 9.3\%\) of the probability mass and hands the remaining \(29{,}257/32{,}245 = 90.7\%\) to word pairs that were never seen (\(29{,}257 = 29{,}773 - 516\); the 516 observed continuations include </s>). (The familiar \(V/(c(h)+V) = 92.3\%\) is the total pseudo-count mass added, part of which lands back on observed continuations; the number that matters is the 90.7%.) The model collected 2,472 examples of what follows kargo, and add-one instructs it to spend nine tenths of the mass elsewhere. With longer contexts \(c(h)\) is smaller still and the damage is total.

Tuning the pseudo-count down helps, and the tuning must happen on dev:

### add-k, k chosen on the dev set
order   best k     dev      test
2       0.003     714.6     728.5
3       0.001   4,011.3   4,058.6
5       0.0003 16,280.4  16,141.8

Better at \(n=2\), still catastrophic beyond it. On this corpus, with the values of \(k\) tried, adding a constant to every count did not work beyond bigrams, and the reason is general: the problem is not that unseen events need a little mass — it is that we have no idea which unseen events deserve it, and a constant says they all deserve the same.

interpolation — blending the estimates of contexts of different lengths (three-word, two-word and one-word, say) into a single number, with non-negative weights that sum to one. In this chapter the weights are chosen on the dev set. ↩︎

The idea that works: ask a shorter model. If you have never seen kargo çok hızlı, you have probably seen çok hızlı, and you have certainly seen hızlı. Mix the orders together by interpolation:

\[ P_{\text{int}}(w \mid h) = \lambda_3 P(w \mid w_{-2}, w_{-1}) + \lambda_2 P(w \mid w_{-1}) + \lambda_1 P(w) \]

with the \(\lambda\)s summing to one and fitted on the dev set by EM (Jelinek and Mercer 1980). Now an unseen trigram falls back on a bigram estimate rather than on a uniform guess, and the fallback is graded by how much each order can be trusted. One detail does not show in the formula: if a context was never seen in training, that order’s MLE term \(c(h,w)/c(h)\) is undefined (\(c(h)=0\)). An implementation either counts the term as zero or drops it and rescales the remaining weights, and the choice changes the result: in the re-run the trigram’s test perplexity is 408.3 with the first and 397.2 with the second. Which one the code behind the table below did was not recorded; the re-run gives lower numbers than the table with either choice, and weights very close to the table’s.

What EM decides is more interesting than the perplexity it produces:

### interpolation weights, fitted on dev
                unigram  bigram  trigram  4-gram  5-gram
2-gram model     0.4204  0.5796
3-gram model     0.4228  0.4834   0.0938
4-gram model     0.4237  0.4878   0.0882  0.0003
5-gram model     0.4234  0.4900   0.0865  0.0001  0.0000

Nobody told EM that 4-grams and 5-grams add almost nothing on this corpus; it worked it out. Given a free choice of how much to trust each order, it puts 0.03% of the weight on 4-grams and rounds 5-grams to zero (both are exactly zero in the re-run). This is the sparsity figure above, rediscovered from the other direction, and one of the clearest model-selection results in the chapter. Know its limit too: it is a single global weight for every context. Higher orders may still help in some contexts; this mixture cannot tell those apart.

Kneser-Ney, and one genuinely clever idea. One of the strongest classical smoothers — its modified version came out best in Chen and Goodman’s broad comparison (Kneser and Ney 1995; Chen and Goodman 1999) — subtracts a fixed discount \(d\) from every observed count and redistributes what it took. The formula is for a context seen in training (\(c(h) > 0\)):

\[ P_{\text{KN}}(w \mid h) = \frac{\max(c(h,w) - d,\ 0)}{c(h)} + \lambda(h)\, P_{\text{KN}}(w \mid h') \]

Here \(h'\) is the context \(h\) with its oldest word dropped, so the second term is the same formula one order down, applied recursively. For a context never seen (\(c(h) = 0\)) the model goes straight to \(P_{\text{KN}}(w \mid h')\).

backoff — when the long context’s estimate is not good enough (in the classic form, when the word has never been seen after it), drop a word off the front and ask the shorter context instead, rescaling its share so the probabilities still sum to one. Interpolation always blends; backoff only falls back when it has to. ↩︎

The variant written there is interpolated Kneser-Ney: it always adds a share of the shorter-context estimate. Keep that apart from backoff in the strict sense, which consults the shorter context only when the longer count is zero.

The clever part is not the discount, it is what the lower-order distribution counts. A naive smoother asks “how often does this word occur?” Kneser-Ney asks “how many different contexts does this word occur in?” The standard example is English, but the effect exists in every language: the word Francisco is frequent, but it appears almost exclusively after San, so it is a terrible guess after any other word. Frequency says common; continuation count says specialised. Kneser-Ney uses the second.

Two quantities the formula hides. First, \(\lambda(h)\) is not a free parameter — it is exactly the mass the discount removed, so that the distribution still sums to one:

\[\lambda(h) = \frac{d \cdot N_{1+}(h\bullet)}{c(h)},\]

where \(N_{1+}(h\bullet)\) counts the distinct words seen after \(h\). A context with many different continuations gives more weight to the lower order; a context with one repeated continuation gives almost none. Second, the lowest-order distribution is not a unigram frequency but the continuation distribution,

\[P_{\text{KN}}(w) = \frac{N_{1+}(\bullet w)}{N_{1+}(\bullet\bullet)},\]

the number of distinct words that \(w\) has ever followed, over the total number of distinct bigram types. That is the Francisco rule, written down: Francisco has followed almost only one distinct word, so its continuation probability is tiny however often it occurs. The same substitution is made at every order below the highest, not only the lowest: in the lower steps of the recursion, continuation counts replace the raw counts (for example \(N_{1+}(\bullet\, h\, w)\) in place of \(c(h, w)\)); the unigram level is simply where the idea is clearest. (The discount \(d\) is a constant between 0 and 1, tuned on the dev set or estimated from how many n-grams occur once and twice, as \(n_1/(n_1 + 2n_2)\). Modified Kneser-Ney uses separate discounts for counts of one, two and three-or-more at each order. 0.75 is a common starting value, not a value tuned for this corpus.)

### test perplexity, everything together
order      MLE    add-1    add-k*   interpolated   Kneser-Ney
1      1,303.6  1,313.8   1,303.6            —            —
2            ∞  3,216.5     728.5        427.8        351.9
3            ∞ 13,221.2   4,058.6        426.4        333.5
4            ∞ 20,147.7  11,240.8        438.1        344.7
5            ∞ 22,234.0  16,141.8        447.6        353.8

The best model in the table is a Kneser-Ney trigram at 333.5, and three things about that table deserve to be said out loud — but first a warning about its Kneser-Ney column. It uses \(d = 0.75\) at every order; the discount was not tuned on the dev set. The re-run shows what that costs:

### Kneser-Ney test perplexity in the re-run, by how d was set
order    d = 0.75    d = n1/(n1+2n2) per order    one d chosen on dev
2           352.3                        350.5
3           336.5                        320.2                 320.7 (d = 0.9)
4           347.7                        319.5
5           354.2                        319.5

The first column reproduces the table’s numbers to within one percent. Estimating the discount from the counts, or choosing it on the dev set, improves the trigram by about 5% and brings the 4-gram and 5-gram level with the trigram. The three observations below should be read with those two columns in mind.

Smoothing is not a detail. It is the difference between an infinite perplexity and 333.5, on identical counts. The estimator matters more than the model class here.

Going beyond trigrams does not help. Under every smoothing scheme in the table, 4-grams and 5-grams are worse than trigrams; with a tuned discount the harm disappears but no gain arrives (319.5 against 320.2). That is the sparsity curve doing what it promised: the extra context is real information, and on this corpus the model cannot reach it.

And a trigram at 333.5 means that after reading two words of a Turkish review, the model gives the observed words, on average, as much probability as picking at random from 333 equally likely words would. It is not a ceiling — tuning the discount alone gets the re-run to 320 — but it is where the counting methods tried here end up on this corpus, and getting there took decades of work.

Kneser-Ney was not beaten by cleverness so much as by scale. Google’s machine-translation systems of the late 2000s used stupid backoff (Brants et al. 2007) — an unnormalised scheme that is not even a probability distribution. Brants and colleagues reported that as the data approached a trillion tokens this simple scheme caught up with Kneser-Ney in translation quality; at that scale and for that task, the engineering decided (Heafield 2011). That does not mean smoothing stops mattering for perplexity: stupid backoff gives no probability, so its perplexity is not even defined. The idea of falling back to a shorter context when evidence runs out is older still (Katz 1987); Kneser-Ney’s contribution was what to fall back to.

We cannot measure that directly, because there is no second corpus of the same size, but we can measure the reverse. In the re-run we kept the vocabulary and the test set fixed and took the counts from a random half of the 38,000 reviews (\(d = 0.75\)):

### Kneser-Ney test perplexity, 19,000 -> 38,000 reviews (re-run)
order    19,000    38,000    change
2         410.0     352.3    -14.1%
3         401.4     336.5    -16.2%
4         413.9     347.7    -16.0%
5         421.0     354.2    -15.9%

In absolute terms the number that moves most is indeed the 5-gram’s (66.8 points), but in relative terms the three higher orders improve at almost the same rate; the one that stands apart is the bigram, which improves less. The expectation “the most starved 5-gram gains most” does not hold at this scale: the 5-gram stays worse than the trigram at both sizes. With the discount estimated from the counts the picture is the same: trigram and 5-gram both improve by 15.8%.

The sparsity table hints at why. Doubling the data turns some singleton 5-grams into repeated observations, but the new text brings new singletons at the same time, and for 5-grams of Turkish reviews most of them are new. You cannot read off the corpus size how much an order will gain; you have to measure it.

The transferable version: measure where more data will help, with a learning curve over a fixed vocabulary and test set, rather than by intuition. Here the bigram lagging fits the intuition that orders with few counts gain a little more, but the higher orders do not separate. The scaling laws of Chapter 8 measure curves like this at large scale; one reason the models there are trained on trillions of tokens rather than millions is that a big model on a small corpus can drift into memorising.

It also has a limit worth predicting now. Doubling the corpus improves the 5-gram model but does not make 5-grams sufficient, because English and Turkish both contain dependencies longer than five words. No amount of data fixes a wrong assumption; it only reduces the variance around it.

6.7 What the model actually knows

Perplexity is a summary, and summaries hide things. A language model has a second, more revealing interface: you can sample from it. Start with the padding, draw a word from the conditional distribution, shift the context, repeat until </s>.

The samples and copying rates below come from unsmoothed (MLE) n-gram models, not from the Kneser-Ney models of the perplexity tables. Sampling from MLE is well defined: the model can only produce n-grams seen in training, so every context it reaches is one it has seen. What sampling from Kneser-Ney gives comes a little further down.

This is pure sampling: each word is drawn from the model’s distribution as it is. Decoding choices such as greedy selection, temperature or top-k change these outputs — and the copying rates below — and they are the subject of Chapter 18.

One more difference. When perplexity is measured, the context always comes from the real test text, the true prefix; in neural network training, conditioning the model on the true prefix is called teacher forcing. When sampling, the model conditions on the words it drew itself (free-running): one odd early choice can push every later context somewhere training never went. Part of why the samples below fall apart after a few words may come from this. The degradations that come from a model conditioning on its own output (repetition loops and degeneration) persist in large models too; we return to them in Chapter 18.

### sampling, seed 7

unigram
  kullanım <unk> tavsi
  sadece telefon
  ederi bu göre

bigram
  ürünü tercih edilebilir
  makina sessiz ve tadi oluyor ben çok kullanışlı ve masaj yapacak alan…
  en iyi bir ürün kolay

trigram
  arkadaşlar ürün günde <unk> ulaştırdıkları için siteye teşekkür ediyorum
  ofise aldık çok beğendi teşekkürler hepsiburada
  ürünü yeni kullanmaya başladım gayet güzel ve kalıcı fıs yetiyor…

5-gram
  ürünü yeni aldım gayet kullanışlı tavsiye ediyorum
  bu ürünü uzun süredir kullanıyorum şunu diyebilirim ki ses yapiyor…
  yıllardır kullandığım ürün en uzun gece kampanyası ile tl ye uygun…

The unigram output is word salad. The bigram output has local grammar and no global sense. The trigram output contains whole clauses that read like Turkish. The 5-gram output reads like a product review.

It reads like a product review because it is one:

### of the 8-word spans the model generates, how many appear verbatim
### in the training corpus?
unigram      0.0%
bigram       0.0%
trigram      5.8%
5-gram      91.6%

Nine out of ten eight-word spans that the 5-gram model “writes” occur word for word in the training data. The fluency is largely retrieval dressed up as generation. Note what the measure says: whether the eight-word windows of the generated text occur verbatim in the training text. It does not say that whole reviews are copied, and the rate depends on the sampling method and the window length. In the re-run, with three seeds (200 samples each), the MLE 5-gram gives 91.8%, 92.6% and 93.9% and the MLE trigram 7.0%, 6.8% and 7.2%; the table’s numbers belong to one seed, but they are no accident. The sparsity table predicted this: since 98% of training 5-grams occurred once, conditioning an MLE model on four words usually leaves it exactly one choice — the word that followed last time.

Does smoothing change that? The same measurement, with the Kneser-Ney models of the perplexity tables (re-run, \(d = 0.75\)):

### 8-word spans found verbatim in the training corpus (re-run, 200 samples per seed)
                         seed 7   seed 8   seed 9
trigram, MLE               7.0%     6.8%     7.2%
5-gram,  MLE              91.8%    92.6%    93.9%
trigram, Kneser-Ney        0.0%     0.0%     0.0%   (1 of 9,262 spans)
5-gram,  Kneser-Ney        2.4%     1.9%     1.8%

5-gram, Kneser-Ney, seed 7:
  sürekli yazı <unk> düştü hem toplu alabilirsiniz
  ürün güzel gösterecektir kargo güvenle eşi m kullaniyor

It does: the Kneser-Ney 5-gram hardly copies at all. But look at its samples: the fluency has gone too, and the output is as scattered as the bigram’s. The cause is the same sparsity. In a context seen only once, Kneser-Ney leaves the single observed continuation a quarter of the mass (\(1 - 0.75\)) and hands the rest to shorter contexts; with 98% of 5-grams singletons, the model is in effect sampling from the lower orders at most steps.

This is the bind that motivates everything after this chapter. Fluency needs long context. Long context, estimated by counting, either copies or falls apart. In these measurements no n-gram model was both fluent and original: the MLE 5-gram is fluent because it copies; the Kneser-Ney 5-gram is original but scattered. Smoothing reduces the copying; it does not create a new generalisation, it redistributes probability to shorter contexts.

You will meet this pair of failure modes again in a modern guise — verbatim memorisation of training data is a live problem for large language models too (Carlini et al. 2021), and Chapter 5’s warnings about benchmark contamination are the same phenomenon seen from the evaluation side.

6.8 What perplexity does not tell you

Before leaving the counting era, one warning that will save you from misreading a great many published numbers, including some you will be asked to reproduce.

Perplexity depends on the vocabulary. Every word outside the vocabulary becomes <unk>, and <unk> is a frequent symbol, so it is easy to predict. Shrink the vocabulary and you replace hard predictions with easy ones. Here is the same method (a Kneser-Ney trigram), the same code and the same raw test reviews, with one setting changed: the frequency threshold at which a word is kept. That setting changes a lot, though — which words of the training text become <unk>, and so the counts and the fitted model itself. The table is not eight measurements of one model but eight models, refitted with eight vocabularies, on eight different prediction tasks:

### one method, one raw test set, eight vocabularies, eight refitted models
min_count    vocab    test <unk>    perplexity
    1       72,089        5.21%          752.5
    2       29,773        7.42%          333.5
    3       20,090        9.00%          281.6
    5       12,839       11.36%          222.6
   10        7,275       15.17%          157.8
   25        3,460       21.56%           94.3
  100        1,057       34.58%           38.0
  500          199       54.83%           10.9

A model that cannot name more than half the words in the test set reports a perplexity of 10.9. It looks like a triumph and it is a side effect: for 54.8% of the test words, the “prediction” is the single easy guess <unk>. That is the widget’s last sentence in numeric form — uyuyakaldı scored 2.5 bits not because the model understood anything but because it had been allowed to stop trying. Not all of the effect comes from <unk>: in the re-run, even with the <unk> events left out, perplexity is 417.8 at threshold 2 and 74.5 at threshold 500. The words that remain are frequent words with few rivals; shrinking the vocabulary makes the whole task easier.

Note also the top row. Keeping every hapax word gives the highest number, 752.5 — but careful: that comparison crosses vocabularies too, so it is not by itself proof of a worse model. The genuine modelling reason to threshold is different: at min_count 1, <unk> never occurs in the training text, so the model has no counts from which to learn how often a new word turns up, and the 5.21% of test words it has never seen get almost no probability. The re-run shows how little: <unk>’s probability comes only from the uniform floor (\(1/V\)) at Kneser-Ney’s lowest order, and those events average 20.5 bits — about one in a million. That is why the top row is the highest number in the table. So there is a real reason to threshold as well as a way to cheat by thresholding. The two are not distinguishable from the perplexity alone. That is the point.

So: two per-token perplexities can be compared directly only if the two models share the test text, the vocabulary, the tokenisation and the event definition (is </s> counted?). Across tokenisers per-token perplexities cannot be compared, because the denominator \(N\) counts different things — this is why modern papers report bits per byte or report on a fixed benchmark with a frozen vocabulary (Chelba et al. 2014). Bits per byte divides the total \(-\log_2\) probability of the text by its length in bytes rather than in tokens, so the denominator is the same whatever tokeniser the model uses. That has a condition: both models must assign probability to the raw text itself. A model that turns a word into <unk> and loses its surface form gives no probability to that word’s bytes; switching the denominator to bytes does not fix that comparison on its own. When you meet a leaderboard in Chapter 22, this is the first question to ask of it.

The deeper version of the warning: perplexity measures how well a model predicts text, and that is not the same as whether the model is useful. A model can be excellent at predicting the next token of product reviews and unable to answer a question about one. Chapter 9 is entirely about the gap between those two things.

6.9 Neural language models

The failures so far share a root. kargo and teslimat mean almost the same thing, and to a count table they are two unrelated strings: nothing counted for kargo çok is written into the row for teslimat çok. Smoothing does draw on shared lower-order information (the continuation probability of the next word, say), but it does not know that the two words are alike. It is Chapter 3’ complaint, arriving now in a probabilistic setting: the model has no way to say that two contexts are similar.

Bengio and colleagues (Bengio et al. 2003) proposed the fix that later neural language models all descend from. Keep the Markov assumption — condition on the previous \(n-1\) words — but stop treating those words as symbols. Instead:

  1. Look up a dense vector for each context word. These are the embeddings of Chapter 3, except that nobody trains them separately: they are parameters of the language model, learned by backpropagation from the prediction objective.
  2. Concatenate them and push them through a hidden layer, so the model computes a smooth function of the context rather than looking up a cell in a table.
  3. Project to a softmax over the vocabulary, giving a proper distribution over the next word. A final linear layer produces one score \(z_v\) per vocabulary word, and the softmax (met in Chapter 3) turns the scores into probabilities: \(P(v \mid h) = e^{z_v} / \sum_{u} e^{z_u}\).

The idea of sharing evidence between related words was not new — Brown and colleagues (Brown et al. 1992) had already grouped words into hard classes so that counts could be pooled. What Bengio et al. changed is that the grouping is soft, learned, and continuous: no word has to be assigned to exactly one class, and similarity is a matter of degree.

The consequence is the entire point. The model is a continuous function of the context vectors; if kargo and teslimat end up with nearby vectors and the rest of the context is the same, the model tends to give the two contexts similar predictions. Generalisation across similar contexts is not a feature that was added; it is a consequence of replacing a lookup table with a smooth function. It is a tendency, not a guarantee: how much the nearness helps depends on the network’s weights and the other words in the context. Whether it helps is measured below.

In PyTorch the whole model is nine lines:

class NNLM(nn.Module):
    def __init__(self, V, ctx, emb=64, hid=256, drop=0.5):
        super().__init__()
        self.emb = nn.Embedding(V, emb)
        self.net = nn.Sequential(
            nn.Linear(ctx * emb, hid), nn.Tanh(), nn.Dropout(drop),
            nn.Linear(hid, V))

    def forward(self, x):              # x: (batch, ctx) word ids
        return self.net(self.emb(x).flatten(1))

The same V sizes both the input table and the output layer; here V = 29,774: the prediction support (29,773) plus the <s> the input needs (explained under Counting on 38,000 reviews).

nn.Dropout(drop) switches off a random fraction drop of the hidden units at each training step (and none at test time), which makes it harder for the network to lean too heavily on any single unit; it is a standard regulariser against overfitting.

Training minimises exactly the quantity the perplexity section defined: the average negative log-probability of the word that actually came next. The forward pass returns scores, and the cross-entropy loss applies the softmax and takes the log in one step:

logits = model(x)                          # (batch, V) scores
loss = F.cross_entropy(logits, y)          # mean of -ln P(y | context) over the batch
# dev perplexity: sum the loss over all dev events (reduction="sum"),
# divide by the number of events, take math.exp; not a mean of batch means

So the “train ppl” and “dev ppl” columns below are just exp of this loss on the two splits. Nothing about the objective is specific to this small model: the pretraining in Chapter 8 minimises the same next-token cross-entropy, only with a much longer context and far more data.

Shared parameters can carry evidence between contexts a table keeps apart
COUNT TABLE TWO ROWS, NO CONNECTION kargo teslimat ROW 1 — 2,472 OBS. its own distribution ROW 2 — ~610 OBS. its own distribution NO PATH NEURAL MODEL ONE FUNCTION, SHARED PARAMETERS EMBEDDING SPACE kargo teslimat NEARBY ≈ SIMILAR hidden tanh softmax 29,774 SHARED WEIGHTS LEARN FROM BOTH CONTEXTS. WHAT WE MEASURED — AGAINST KNESER-NEY WITH d = 0.75 The ranking depends on whether the two-word context was seen in training. Contexts seen in training: neural 265.9 beats Kneser-Ney 273.1. Contexts never seen: neural 554.4 loses to Kneser-Ney 526.7. A KNESER-NEY WITH A TUNED DISCOUNT LEADS IN BOTH GROUPS (RE-RUN). SEE BELOW.

The mechanism on the right is real, but nearby vectors make similar predictions easier, they do not guarantee them. The measured result on this corpus also does not fit the often-told expectation that neural models win on novel contexts, which is why it was worth measuring rather than asserting. The numbers at the bottom are against Kneser-Ney with \(d = 0.75\); against a Kneser-Ney with a tuned discount the picture changes again (below).

The first run was a failure

Trained with the settings above at drop=0.2, the model reached test perplexity 429.0 against the Kneser-Ney trigram’s 333.5. Worse, the training curve said exactly why:

epoch  train ppl   dev ppl
    2      456.6     432.7
    3      297.1     430.0   <- best dev
    4      219.8     448.4
    5      177.6     467.5   early stop

Training perplexity falls through the floor while dev perplexity turns around at epoch 3. That is textbook overfitting. One number describes the capacity: 9.6 million parameters against 896,000 training events, roughly eleven parameters per event. That ratio does not prove the cause on its own (the events are not independent either); the evidence is the two curves pulling apart. Publishing “neural language models lose to Kneser-Ney” on the strength of that run would have been a statement about my hyperparameter choices dressed up as a statement about architectures.

So the honest next step is a search, choosing on the dev split. A note in the same spirit: we had already read the first run’s test value (429.0), and the table below prints every configuration’s test value too. They were not used to choose, but that makes them exploratory results shown for audit, not an independent final evaluation.

### 7 configurations, ranked by dev perplexity (test shown for audit, not used to choose)
ctx   emb   hid   drop        params       dev      test
  2    32   128    0.5     4,801,934     329.8     332.4
  2    64   256    0.5     9,590,478     362.5     362.6
  3    64   256    0.5     9,606,862     366.0     365.3
  2    64   256    0.2     9,590,478     430.0     429.0
  2   128   512    0.5    19,216,718     432.1     436.4
  3   128   512    0.5    19,282,254     479.6     483.1
  5   128   512    0.5    19,413,326     541.4     548.8

Read that table from the bottom up, because the ordering is the lesson. In this sweep, apart from one row, every step down in capacity was a step up in quality. (The exception is the drop=0.2 run: it has exactly as many parameters as the second row, 9,590,478, and differs from it only in regularisation — raising dropout alone moved test perplexity from 429.0 to 362.6.) The table is not a one-variable sweep: context, embedding, hidden layer and dropout change together from row to row, so the ordering cannot be pinned on a single cause. The largest model tried, with five words of context and 19.4 million parameters, is the worst model in the sweep at 548.8 — worse than every interpolated and Kneser-Ney n-gram model in the smoothing table. The winner is the smallest model tried: two words of context, 32-dimensional embeddings, a 128-unit hidden layer.

This is the same lesson as Chapter 4’s, arriving through a different door. Thirty-eight thousand reviews is a small corpus. On a small corpus the binding constraint is often data, not capacity, and extra parameters may buy you nothing but a faster route to memorising the training set.

At first sight the ctx=5 row says “more context made the model worse”; but in that row the embeddings and the hidden layer grew along with the context. In the re-run we changed only the context, holding the smallest configuration (32/128/0.5) fixed:

### re-run: only the context changes (emb 32, hid 128, drop 0.5)
ctx        params       dev      test
  2     4,802,578     319.0     333.5
  3     4,806,674     315.5     327.0
  5     4,814,866     312.9     324.1

On the same small network, longer context helps. The sweep’s bad ctx=5 came not from the context but from the capacity that grew with it. It is a single-seed measurement and the sweep had only seven rows, but the table is a reminder that the winner of a sweep is not the best of the configurations nobody tried.

The result, stated honestly

                          test ppl    stored numbers
Kneser-Ney trigram           333.5         1,087,421 counts
neural, ctx=2 32/128         332.4         4,801,934 parameters

That is a tie, not a victory. A 0.3% difference is inside the run-to-run variation: retraining the same configuration, changing nothing, gave test 330.3 instead of 332.4. The run-to-run variance is larger than the margin of victory. Two runs do not give a distribution; in the re-run we tried five seeds:

### re-run: neural ctx=2 32/128, five seeds, against Kneser-Ney trigrams
seed                       0       1       2       3       4
neural test ppl        333.5   332.7   335.6   331.9   332.6
vs KN, d = 0.75 (336.5)  -0.9%   -1.1%   -0.3%   -1.4%   -1.2%
vs KN, tuned d (320.2)   +4.1%   +3.9%   +4.8%   +3.7%   +3.9%

The five seeds average 333.2 with a standard deviation of 1.4. A paired bootstrap over the test reviews (which covers the uncertainty of the test sample, not of the seed) shows the difference from the \(d = 0.75\) Kneser-Ney excluding zero for four of the five seeds: the neural model is ahead, by a small but consistent margin. Against the Kneser-Ney with a tuned discount it is about 4% behind for all five. So the “tie” is a tie against a textbook-tuned Kneser-Ney; tune the counting side too and the count model leads. Improving both sides does not change that: the small network with five words of context reaches 324.1 (one seed), the tuned Kneser-Ney 5-gram 319.5. Anyone reporting this as “the neural model wins” is claiming too much, and the cheap way to catch it is the one used here — train it several times, and tune the model it is compared against.

Note the second column too, because the usual claim that neural models are more compact is false here: the neural model stores four and a half times as many numbers as the count table it merely matches, and in this implementation takes two minutes rather than seconds to fit. These are item counts, not bytes: with 32-bit weights the network is about 19 MB, while what the count table takes depends on how you store its keys and counts (much more in plain Python dictionaries, much less in compressed n-gram structures). Almost all of those numbers sit at the two ends of the network: the embedding table (\(29{,}774 \times 32\)) and the output layer (\(128 \times 29{,}774\) weights plus one bias per word) together hold 4.79 of the 4.80 million parameters. The vocabulary size, not the hidden layer, sets the cost of a model like this.

Which raises the obvious question — what was the point? Two answers. First, the two models are good in different places, so they can be combined, and the combination is clearly better than either (the next two subsections). Second, the neural model has properties perplexity does not show: it produces embeddings, and when sampled it does not copy the training text. We will measure which of those really sets it apart from the count models.

Where the neural model’s advantage actually lives

Split the test events by whether the two-word context was ever observed in training. Both models condition on the same two words, so the cut gives both the same events and the comparison is fair. But the two models differ in both the estimator and the parametrisation; the cut does not, by itself, show why they differ:

                        events      KN    neural    mixture
all                    239,055   333.5     332.4      274.4
context seen           166,375   273.1     265.9      227.4
context unseen          72,680   526.7     554.4      421.8

Look at the last two rows, because they are the opposite of what you have probably been taught. On contexts the model has seen before, the neural model wins — 265.9 against 273.1. On contexts it has never seen, the neural model loses, and not narrowly: 554.4 against 526.7.

The generalisation story is backwards on this corpus. The smoothed count model is the one that handles novel contexts better. One plausible explanation: Kneser-Ney’s lower-order term is a carefully engineered answer to exactly that situation — when you have no evidence, fall back to how many distinct contexts a word completes — while the neural model, faced with a context it has never seen, leans on an embedding geometry that was itself learned from sparse and noisy counts. This experiment does not prove that this is the cause; that would take varying the regularisation, the amount of data and the context groups separately.

A Kneser-Ney with a tuned discount changes the split once more. In the re-run (\(d = 0.75\) gives 276.8 / 526.6, the neural model 265.7 / 561.2) the tuned Kneser-Ney gives 259.0 on seen contexts and 520.6 on unseen ones: ahead of the neural model in both groups. The “neural model wins” row on seen contexts, too, is an advantage won against an untuned discount.

You can watch it happen in the model’s own uncertainty:

### neural model, mean predictive entropy
context seen      7.42 bits  =  171.2 effective choices
context unseen    7.83 bits  =  228.0 effective choices

On unseen contexts the model spreads its mass over somewhat more words. That does not mean it knows how much trouble it is in. In the re-run we set the entropy beside the surprisal of the word that actually came, that is, beside the real loss:

### re-run, seed 0: mean predictive entropy against mean surprisal (bits)
                  entropy    surprisal    gap
context seen         7.58         8.05    0.47
context unseen       7.81         9.13    1.32

On unseen contexts the uncertainty rises by only 0.23 bits while the loss rises by 1.08. The model is overconfident in both groups, but far more so on novel contexts: its hedging covers a small part of its real difficulty. Entropy on its own is not evidence of good calibration; calibration has to be measured separately, as here.

Be careful about how far you carry this finding. It is a result about this corpus at this scale: 896,000 training events, a 29,774-word vocabulary, and a feedforward model with two words of context. At the scales of Chapter 8 neural models are clearly ahead, which is why the field went the way it did — and a stronger architecture can change it even at this scale: in Mikolov and colleagues’ setup of a few million words, recurrent models (From a window to a state, below) were already beating Kneser-Ney clearly (Mikolov et al. 2010). Where this particular split reverses was not measured here. What the result does establish, permanently, is that “neural models generalise better” is a claim to be measured on your data and not assumed from a textbook.

The move that made neural models stick

Here is the observation that follows from the split table. The two models are good at different things. That is a good reason to try both rather than pick one; it does not guarantee a gain in advance, because group averages do not say what happens event by event. Mix the two distributions and fit the single mixing weight on dev:

\[ P(w \mid h) = \lambda \, P_{\text{KN}}(w \mid h) + (1 - \lambda) \, P_{\text{NN}}(w \mid h) \]

### dev perplexity against lambda (weight on Kneser-Ney)
0.0   329.8      0.4   272.2      0.8   283.3
0.1   296.7      0.5   270.9  <-  0.9   296.4
0.2   283.6      0.6   272.1      1.0   327.9
0.3   276.2      0.7   276.0

On the 0.1-step grid the curve bottoms out at \(\lambda = 0.50\) — the two models get equal weight. That does not show that the two models are of equal value on their own; it says only that this weight best lowers the log loss of the probabilities they give together on this split. The re-run makes the point plainly: mixing with the tuned Kneser-Ney, which is 4% better than the neural model, the best weight is still 0.5. Reading the test set once, at the end:

Kneser-Ney trigram    333.5
neural                332.4
50/50 mixture         274.4      -17.4% against the better of the two

A 17% cut in perplexity from two models that individually tie. This is not a trick, and not the luck of one run either. In the re-run the gain is between 17.7% and 18.2% for each of five seeds; the paired bootstrap interval over the test reviews is 17.7% to 18.3%. Mixing with the tuned Kneser-Ney the gain falls to 15.6%, but it stays.

A control shows how much of the gain comes from combining two different model families: mixing two neural models trained with different seeds the same way (an ordinary ensemble) gains 9.1% in the re-run (from 333.5 and 332.7 to 302.5). The mixture with Kneser-Ney gains about twice that; so roughly half of the gain would come from averaging any two models, and the other half appears to come from two different families complementing each other.

Similar combinations were studied in the literature: Bengio et al. reported the same structure of finding in 2003 (Bengio et al. 2003), and many strong systems of the following years were a neural model interpolated with a count model (Mikolov et al. 2010). It was one of the reasons neural language models stayed in use in those years, when on their own they were merely competitive with a well-tuned n-gram model and more expensive to train.

The mixture also helps most exactly where the split table said the models disagree most: on unseen contexts it improves on the better single model by 20%, from 526.7 to 421.8. That is consistent with the probabilities complementing each other; which events the gain comes from is shown by the event-level log losses, not by a group average.

Two properties perplexity does not show

Perplexity does not show two properties of the model at all: the word representations it learns, and how much it repeats the training text when sampled. Both have to be measured separately; the first really does set the neural model apart from the count models, the second less than it seems.

First, the embeddings. Nobody asked for them. They are a by-product of training a language model, falling out of the first layer: the next-word objective updates the embedding table too. Nearest neighbours by cosine, with each word’s training frequency (nearness is not synonymy, and whether the vectors help on another task is a separate question to test):

güzel      (7547x)  harika, hoş, kaliteli, kaçırılmıyacak, kullanışlı
hızlı      (3856x)  zamanında, hizli, mikemmel, hizlı, ücretsiz
kargo      (2472x)  teslimat, gönderi, gönderim, kargolama, teslimatı
telefon    (1977x)  makina, alet, ürün, kavrayışı, pil
beğendim   ( 657x)  kaldım, getirebiliyorsunuz, memnunuz, memnunum
iade       ( 158x)  rica, ade, istop, verem, iptal
kalitesiz  ( 143x)  konsantre, lezzetlı, kapıdaydı, minicik, fileli

The kargo row is the promise of this section delivered literally: teslimat, gönderi, gönderim, kargolama, teslimatı — the model discovered the shipping vocabulary of Turkish e-commerce from nothing but next-word prediction. And look at hızlı: its neighbours include the misspellings hizli and hizlı (and mikemmel for mükemmel); tavsiye (“recommendation”), not shown in the table, has tevsiye, tafsiye and tasviye among its neighbours. The model gave a word’s misspellings vectors close to the correct spelling, because they occur in the same places. That does not mean it recognises them as the same word; being close is enough. A word-level count table cannot do that; to it hızlı and hizli are as unrelated as hızlı and kedi. (The counting side has its own routes: normalising the text first, or counting at the character level.)

Now read the frequency column downwards, because the honesty of this chapter depends on it. In these seven examples the neighbours are excellent at 7,547 occurrences, good at 2,472, thin at 657, and at 143 occurrences kalitesiz (“low quality”) returns konsantre, lezzetlı, kapıdaydı — noise. Frequency is one plausible cause (how many different contexts a word occurs in, and how many senses it has, play a part too); how many occurrences are enough cannot be read off a seven-row table. Embeddings are not magic; they are counts in a trench coat — counts in disguise — and they degrade with the counts underneath them exactly as Chapter 3 found. The sharing of statistical strength is real, and it has a floor.

Second, copying — but the difference is not where it seems.

### 8-word spans found verbatim in the training corpus
                      context    chapter (seed 7)        re-run, seeds 7 / 8 / 9
trigram, MLE          2 words     5.8%                    7.0% / 6.8% / 7.2%
5-gram,  MLE          4 words    91.6%                   91.8% / 92.6% / 93.9%
trigram, Kneser-Ney   2 words       —                     1 of 9,262 spans
neural, ctx=2         2 words     0.0% (0 of 2,989)       0 of 8,385 spans

Read the context column before the percentages. The 91.6% belongs to an MLE model conditioning on four words; the neural model conditions on two. Matched on context length, the MLE trigram copies 5.8% and the neural model nothing. But the third row, from the re-run, changes the comparison: with the same two words of context, the smoothed Kneser-Ney trigram hardly copies either. The copying comes from unsmoothed counting; the neural model’s not copying sets it apart from MLE, not from Kneser-Ney.

On the neural side, zero observations are a bound, not a guarantee: across three seeds none of 8,385 windows occurs verbatim in training; by the “rule of three” that says the verbatim-match rate is very probably below 0.04% — not that it is zero. Shorter matches, or copying of meaning, this measure does not see at all.

Sampling from the neural model, seed 7:

fiyatına göre süper bir makina hiç bir kalite herkese tavsiye ederim…
sevimli hediye olarak aldım çok güzel çıktı olduğunu göreceksiniz…
her gün bu <unk> bütün fonksiyonları bir verilen parayı hak ediyor…

That text is worse than the MLE 5-gram’s output — it wanders, and it loses the thread after a clause or two — but it is put together without repeating the training text. The Kneser-Ney trigram’s samples, with the same context, are not copies either, but to the eye they are more scattered; we did not measure that. What really sets the neural model apart from the count models in this chapter is not perplexity or the copying rate, but the embeddings and (below) what can improve with scale.

That is the trade this chapter has been walking towards, and it is why Part II exists. On this corpus the counted models were either fluent by copying or scattered without copying. The field bet on models that compose the sentence themselves and share their parameters across similar contexts; how those models improve with scale is what Chapter 8 shows. How much the count models would improve on far larger corpora this chapter did not measure.

Finally, the pair from Chapter 4, scored one last time:

### log10 P(sentence), neural model
kalite iyi fiyat kötü    -11.069
kalite kötü fiyat iyi    -11.262    ratio 1.561

It also prefers sentence a, more weakly than Kneser-Ney’s 2.933 (1.561). One pair is an illustration, not a benchmark: it measures neither an overall ranking nor how far either model grasps meaning. The question chapter 4 left open has an answer — an ordered sequence now has a probability — and the answer is less triumphant than the setup implied, which is the lesson.

6.10 From a window to a state

One limitation has survived everything in this chapter. Every model so far, count-based or neural, conditions on a fixed-size window. Widening it has a cost: in a feedforward network the input layer grows with the window, and an n-gram table grows sparser. On this corpus the small network could still afford a five-word window (the ablation above); the sweep’s bad ctx=5 row had grown its capacity along with the window. But however wide the window, it is fixed: nothing older is ever seen.

Recurrent networks (Elman 1990) remove the window. Instead of concatenating \(n-1\) embeddings, process one word at a time and carry a hidden state forward:

\[ h_t = \tanh(W_{h} h_{t-1} + W_{x} x_t), \qquad P(w_{t+1} \mid w_1 \ldots w_t) = \operatorname{softmax}(W_o h_t) \]

The conditioning bar now contains the entire prefix: there is no last-\(n-1\) limit, and every part of the past can reach \(h_t\) by some path. The Markov assumption — the modelling assumption in the very first figure of this chapter — is gone in that form. But note: the model does not store the prefix word for word, it summarises it in the fixed-size \(h_t\) (below). Applied to language modelling by Mikolov and colleagues (Mikolov et al. 2010), this beat the best n-gram systems of the day in their setup, and LSTMs (Hochreiter and Schmidhuber 1997) made long-range dependencies easy enough to learn to be practical.

There is a reason recurrent language models took twenty years, from Elman to Mikolov, to pay off: their gradients are hard to train. Differentiating \(h_t\) with respect to \(h_{t-k}\) multiplies \(k\) Jacobians together, and for a tanh unit each factor is \(\operatorname{diag}(1 - h_t^2)\,W_h\) — the recurrent weights, but scaled by the activation derivative along the trajectory the network actually took. Repeated products of those factors are what vanish or explode (Bengio et al. 1994). The recurrent weights set the scale; they do not settle it on their own. A scalar \(W_h = 2\) looks explosive until the unit saturates: at \(h_t = 0.99\) the local derivative is \((1 - 0.99^2)\times 2 = 0.0398\), and the gradient is shrinking. In practice you clip the gradient norm to stop the explosion (Pascanu et al. 2013), and you change the architecture to ease the vanishing — an LSTM’s cell state is added to rather than multiplied by a weight matrix, which gives the gradient a path with no repeated \(W_h\) on it. That path is not free of decay: it is still scaled by the forget gate at every step, so a gate that sits below one shrinks the gradient geometrically all the same. The architecture makes long-range learning possible, not automatic.

But look closely at what \(h_t\) has to be. One fixed-size vector, carrying everything relevant from an arbitrarily long past.

Removing the window did not remove the bottleneck. It moved it. The constraint used to read “you may look at the last four words”. Now it reads “you may remember as much as fits in a few hundred numbers (512, say), compressed one word at a time”.

And there is a second cost, which turned out to matter more. Because \(h_t\) depends on \(h_{t-1}\), the computation is inherently sequential: a thousand-token document takes a thousand steps that cannot be parallelised across positions. On GPUs, which are fast only when they do many things at once, that is a severe handicap.

Two bottlenecks, then: a compression bottleneck and a sequencing bottleneck. Chapter 7 describes the architecture that eases both: every position looks directly at every earlier position within a finite window, without squeezing them into one state, and in training the computation for all positions can run in parallel. It does not remove them outright: the window is still finite, the representations still have a fixed size, and when generating text each new token still waits for the one before. Everything in this chapter is the argument for why that step was worth taking.

6.11 Choosing

Two models, near-identical perplexity, and very different bills. Which one would you actually deploy?

What each model is actually for, on 38,000 Turkish reviews
n-gram + Kneser-Ney Feedforward neural LM
Test perplexity here 333.5 (\(d = 0.75\)); ~320 with a tuned discount (re-run) 332.4; 331.9–335.6 over five seeds in the re-run
Training time in this implementation seconds ~2 minutes, on a laptop GPU
Novel contexts better — 526.7 worse — 554.4
Seen contexts 273.1; 259.0 with a tuned discount (re-run) 265.9: ahead of \(d = 0.75\), behind a tuned Kneser-Ney
Repeats training text when sampled hardly ever with Kneser-Ney (trigram: 1 of 9,262 windows); yes with unsmoothed MLE (5.8% at \(n=3\), 91.6% at \(n=5\)) 0 of 2,989 windows (0 of 8,385 in the re-run); shorter matches not measured
Gives you embeddings not in this word-level setup yes, as model parameters; usefulness on another task to be tested
Adding data retrain in seconds retrain in minutes
Returns from more data ~16% better from 19,000 → 38,000 reviews (re-run); larger scales not measured not measured in this chapter; the evidence at scale is in Chapter 8
Debuggable inspect the counts and the backoff contributions no counts to read; event-level log losses, embedding neighbours, activations and samples can be inspected

The practical reading, on perplexity and cost alone: at this scale, use the n-gram model and tune its discount; use both if you can afford it — the mixture gains 16% even against the tuned Kneser-Ney. Actually deploying something would also take measuring task performance, latency, memory, and how much the output overlaps the training data.

The neural model does not earn its place by its perplexity on this corpus, nor by its copying rate, which only sets it apart from unsmoothed counting. It earns it in two other rows: it produces embeddings, and its returns from a hundred times more data are the ones expected to keep going (this chapter did not measure that; Chapter 8 shows the evidence at scale). That is a comparison of slopes, not a claim that counting stops improving. It is a bet on the future, and this chapter is the moment in the book where the bet is placed.

6.12 Intuition checks

“Same vocabulary?” Per-token perplexity can only be compared directly across models that share the test text, the vocabulary, the tokenisation and the event definition. The table in What perplexity does not tell you moved the same method from 752.5 to 10.9 by changing only the frequency threshold (and rebuilding the counts with it). If the neural model was trained with a 30k vocabulary and the baseline with 72k, the 40 points may be entirely a side effect of the <unk> rate, not a result. If the vocabulary matches, the next questions: were the </s> events and the loss average counted the same way, and was the baseline tuned? This chapter’s “tie” was against an untuned Kneser-Ney discount too.

Because 38,000 reviews is a small corpus, and the binding constraint is evidence rather than capacity. Even the 9.6-million-parameter first run, at roughly eleven parameters per training event, memorised the training set and its dev perplexity turned around at epoch 3; the 19.4-million-parameter model had roughly 22 per event and did worse still. Model capacity should be chosen against dataset size, and the sweep is how you find out. But this sweep does not say “keep stepping down”: context, size and dropout changed together from row to row. In the ablation where only the context changed, the smallest network did better with five words of context (333.5 to 324.1). The right next step is new dev runs that vary capacity, context and regularisation one at a time.

Because they are wrong about different things. The split by seen and unseen context shows the neural model ahead on familiar contexts (265.9 vs 273.1) and behind on novel ones (554.4 vs 526.7), so a mixture is exposed to whichever model is better on each kind of event. It does not know, event by event, which one to trust: it always averages, and the averaging is what pays. The group table is a good cut for looking into this, but it does not explain the 17% on its own: a mixture of two neural models with different seeds gains 9.1% too (re-run). Roughly half of the gain appears to come from plain averaging and the rest from the difference between the two families; separating them for certain takes the event-level log losses.

Be careful with the general claim, because the usual version of it is false. What interpolation guarantees is weaker: by Jensen’s inequality the mixture’s log loss is never worse than the \(\lambda\)-weighted average of the two models’ log losses, so it is never worse than the worse model. “Never worse” is the honest word: at \(\lambda = 0\) or \(1\), or when the two models put the same probability on every event, the inequality is an equality. Beating the better one needs their strengths to be genuinely complementary, and adding a weak model to a strong one at a badly chosen \(\lambda\) makes things worse. Fitting \(\lambda\) on held-out data is what protects you — it can collapse to 0 or 1 — but a weight chosen on development data is still a bet that the same trade-off holds on the test set, not a guarantee that the mixture beats its best component there. Here it came out at 0.50, but that does not show the two models are worth the same on their own: in the re-run the best weight is 0.50 with the tuned Kneser-Ney too, which is 4% better than the neural model. The weight says what best lowers the log loss of the probabilities the two give together.

Because 91.6% of the eight-word windows the (unsmoothed, MLE) 5-gram produces occur verbatim in the training data, against 5.8% for the trigram. The fluency is largely retrieval. Sparsity explains it: 98% of training 5-grams occur once, so a four-word context usually admits exactly one continuation — the one that followed last time. That is not a finding that whole reviews are copied, nor a legal judgement of plagiarism; but the model’s output should not reach a user without checking its overlap with the training data. Check the window length, the seed and longer matches too. Smoothing nearly removes the copying (about 2% for the Kneser-Ney 5-gram) but takes the fluency with it. How more data would shift that balance this chapter did not measure.

6.13 Interview questions

On attribution — the same caution as Chapter 2. These are questions that come up again and again across the two credible public sources, both free: Introduction to Machine Learning Interviews by Chip Huyen (Huyen 2021) and AIMLInterviews. Be wary of pages that list “the questions company X asks” without a source; attributing a question to a company takes a source you can check.

  1. Define perplexity, then say what it does not measure. The answer is the exponentiated average negative log-likelihood per token — the effective branching factor. What it does not measure: usefulness on any downstream task. And per-token values cannot be compared directly across a change of test text, vocabulary, tokenisation or </s> counting; if both models give the raw text a lossless probability, a common unit such as bits per byte can be used. Candidates who give the formula and stop have answered half.

  2. Why does add-one smoothing fail so badly? Because it moves mass in proportion to vocabulary size rather than to evidence. After kargo there are 516 observed continuations and 29,257 unobserved ones, so add-one hands 90.7% of the probability to things never seen. The failure is not that it smooths, it is that the implied prior is wildly wrong under these conditions: when the support is large (29,773) and the context count small next to it (2,472), one pseudo-count per type swamps the evidence.

  3. What does Kneser-Ney’s continuation count do that a raw count does not? It asks how many distinct contexts a word completes, not how often it appeared. Francisco is frequent but almost always follows San, so its continuation count is low and it is correctly a poor guess in a novel context. Raw frequency gets this exactly backwards.

  4. You have 10 million tokens. 5-gram or neural? The expected answer is a question back — what is the task? For perplexity alone, a tuned n-gram model is a strong, cheap baseline that trains in seconds; remember to tune its discount. But know the limits of this chapter’s result: it was a small feedforward model on under a million tokens. Well-regularised recurrent or transformer models beat Kneser-Ney at that size in most published comparisons, but measure it on your own data, with the same test definition and both sides tuned. When you need the embeddings, when text must be generated fluently without copying, or when you expect the corpus to grow by orders of magnitude, try the neural model as a strong candidate, and count latency and the compute budget too.

  5. Why did the field abandon RNNs for transformers? Two bottlenecks, and the second one is the answer interviewers are listening for. Compression: a fixed hidden state must hold an unbounded past. Sequencing: \(h_t\) depends on \(h_{t-1}\), so a document of length \(T\) costs \(T\) steps that cannot be parallelised even in training, which wastes exactly the hardware the field had started buying. A transformer computes the positions in parallel in training and looks directly at every position in a finite window; when generating text, though, new tokens are still chosen one at a time.

  6. How would you detect that a reported perplexity is not honest? Ask for the test text, the vocabulary size, the <unk> rate on the test set, the tokeniser, whether the </s> events are counted, and the total log loss. A perplexity of 10.9 reported with a 54.8% <unk> rate cannot be read as success at predicting surface forms: for more than half the test words what is predicted is <unk>, and the rest are frequent words in a shrunken vocabulary.

6.14 Where to go deeper

Beyond the papers
resource what it is
The spelled-out intro to language modeling: building makemore — Andrej Karpathy builds a character-level model from counts up, which is exactly this chapter’s arc, in code you can run
Large Language Models explained briefly — 3Blue1Brown a seven-minute video; a short visual introduction to “predict the next token”, not a substitute for this chapter’s probability arithmetic
Speech and Language Processing — Jurafsky & Martin chapter 3 free online; the canonical treatment of n-grams, smoothing and perplexity
Stanford CS224N the language-modelling lectures, with slides and assignments
kenlm a widely used library for building and querying n-gram language models fast; a good option when you need a fast, cheap probability of a string
  • Speech and Language Processing, chapter 3 (Jurafsky and Martin 2026). The canonical treatment of n-grams, smoothing and perplexity, free online, and the source most of this chapter’s vocabulary comes from.
  • An empirical study of smoothing techniques (Chen and Goodman 1999). The paper that compared the smoothers under the same conditions, running all of them properly; it is the comparison in which modified Kneser-Ney came out ahead. Read which data and measures its results hold for, too. A model of how to do an empirical comparison.
  • A neural probabilistic language model (Bengio et al. 2003). Worth reading now that you have built one: the argument in section 1 is exactly the sharing-strength argument above, made before there was hardware to demonstrate it convincingly.
  • Prediction and entropy of printed English (Shannon 1951). Seventy-five years old and still the clearest statement of what a language model is for.
  • Recurrent neural network based language model (Mikolov et al. 2010). One of the papers that showed recurrent language models clearly beating n-grams. Read it for the results table: in its setup the RNN wins outright, and it still wins by more when interpolated with an n-gram model, which is the 50/50 result of this chapter at a larger scale. N-grams stayed in use in many applications afterwards.
  • KenLM (Heafield 2011). If you ever need an n-gram model in production — and for on-device speech and keyboard prediction people still do — it is one of the first implementations to look at; weigh its memory, vocabulary and licence against your own use.

6.15 Exercises

  1. Reproduce the catastrophe. Build the bigram MLE model and find the first test bigram it assigns zero probability. Print the two words. Then explain, without using the word “smoothing”: more Turkish review data could make this particular bigram visible, so why do unseen combinations keep turning up in new test text?
  2. The add-one autopsy. For the context kargo, compute how much total probability add-one assigns to the 516 observed continuations and how much to the other 29,257; the support is \(V = 29{,}773\) (29,771 words, <unk> and </s>). Then find the value of \(k\) at which those two masses are equal, and compare it to the tuned \(k = 0.003\).
  3. Move the crossover. The interpolation weights put 0.0003 on 4-grams (0.0001 inside the 5-gram model). Retrain on 5,000, 10,000, 20,000 and all 38,000 reviews, with the same vocabulary and the same dev protocol, and plot how the fitted \(\lambda\)s and the dev perplexity change with corpus size, over several random-subset seeds. At what size does the change in the 4-gram weight separate from the seed-to-seed noise? Read the trend without assuming a single universal threshold.
  4. Break perplexity on purpose. Reproduce the vocabulary table, then write the two-sentence description of your model that would let a reader detect the trick. This is a writing exercise as much as a coding one.
  5. Measure the copying. Extend the verbatim-span measurement to 4-grams and 6-grams and plot copy rate against \(n\), for both MLE and Kneser-Ney models and over several seeds. The chapter reports 5.8% at \(n=3\) and 91.6% at \(n=5\) for MLE; find where the curve turns sharply upward, and set the location beside the singleton percentages from the sparsity table. The two moving together is not by itself proof of cause; vary the window length too and see how much the result changes.
  6. Reproduce the split. Recompute the seen-versus-unseen context table, but cut the test events by the frequency of the target word instead of by whether the context was seen; report the <unk> events as a group of their own. Does the neural model’s advantage live in frequent words or rare ones? Discuss whether the result is consistent with the claim that embeddings share statistical strength, and say why this cut alone cannot prove the mechanism.
  7. Break the mixture. The 50/50 interpolation cut perplexity by 17%; the chapter reports that mixing two neural models with different seeds gains 9.1%. Repeat that control with three to five seed pairs, in the same dev and test protocol as the Kneser-Ney + neural mixture, and report the single models’ perplexities and the mixing weights. Then show where the two mixtures differ in the event-level log losses. Explain why these two conditions cannot split the 17% cleanly into “plain ensembling” and “different model family” shares.
  8. A Turkish-specific probe. Turkish negation is a suffix, so beğendim and beğenmedim are different words. Measure how much probability mass each model puts on the negated form after a positive context, matching contexts and word frequencies, and read the differences in terms of word representation and suffix information. To compare with English you would need an equivalent English corpus, the same task and the same tokenisation; a single-language measurement cannot say “better or worse in Turkish”.
  9. The other direction. Train a character-level n-gram model on the same reviews. Report its perplexity, explain why that number cannot be compared directly with any number in this chapter, and then find a quantity that can be compared. Hint: bits per byte works only if both models give the raw text a lossless probability; if the word model loses surface forms to <unk>, write down how you handle that.