8  Pretraining and the Model Landscape

Chapter status: drafted. Every number and every code output in the subword tokenization section was produced by running the code shown, with tiktoken 0.13, sentencepiece 0.2.2 and tokenizers 0.22; the full environment is in Appendix A. The BPE tool in that section and the tokenizer-budget and scaling calculators on this page are interactive.

Learning objectives. After this chapter you can:

• draw the stages (pretraining, supervised fine-tuning, preference tuning) that turn a randomly initialised transformer first into a base model and then into an instruction-tuned assistant, and say what data and what objective each stage uses;
• follow a token from a web crawl into a training batch, naming what each filtering step removes and what it is protecting against;
• explain why the most aggressive deduplication in an open dataset made the data worse, and what that tells you about quality filters in general;
• estimate a pretraining run’s compute roughly from its parameter and token counts, work out the elapsed time and rental cost separately from an assumed GPU utilisation, and state where that estimate stops being true;
• read model FLOPs utilization as the gap between a hardware brochure and a datacentre, and attribute the gap to the right cause;
• explain how byte-pair encoding (BPE) works, and use a given set of starting units and merge rules to work out how a new word is split into tokens;
• measure what a tokenizer costs a language, and price that cost in context window, in latency, and in money;
• classify a model release by what it actually opens — weights, data, code, licence — and choose one for a task and a budget.

8.1 A model that can compute and knows nothing

At the end of Chapter 7 we had a machine with the right shape and no contents. Every matrix in it — \(W^Q\), \(W^K\), \(W^V\), \(W_1\), \(W_2\), the embedding table — is full of small random numbers. The architecture is finished. The model knows nothing. Ask a freshly initialised Llama-shaped network to continue kedi sütü and, because standard small initialisation leaves the scores close together, it hands back a token drawn roughly uniformly from 128,000 possibilities — meaningless as language.

Everything the model will ever know arrives through one procedure, repeated a few million times:

show it a stretch of text, ask it to predict the next token, measure how wrong it was, and nudge every parameter slightly in the direction that would have made the true token more likely.

That is pretraining’s mechanism; there is no second one. Grammar, the capital of France, how to close a Python bracket, the fact that coffee can be ready and robots take instructions — all of it arrives as a side effect of getting better at that one prediction, over enough text; how much of it arrives depends on the data, the architecture and the optimisation choices.

Next-token prediction is the objective of decoder-only models such as GPT and Llama, the family this chapter follows. The encoder models of Section 7.5 use variants of the same idea. BERT (and BERTurk, which appears in the tokenizer table below) hides about 15% of the tokens and predicts them from both sides: masked language modelling. T5 hides whole spans of text and generates them: span corruption. The recipe is the same — hide some text, predict it, measure the error.

This chapter is about what happens when you run that loop at industrial scale. Where the text comes from, what has to be thrown away before it is usable, what the run costs in hardware and hours and money, why a Turkish sentence can cost more than its English translation, and how to tell model releases apart by what they share and what their model cards show, rather than by their announcements.

Three stages, and the first one dominates the data bill

A deployable assistant is not the output of one training run. It is the output of a short pipeline, and the stages differ in what they are for, not just in how much data they consume.

The three stages, and the scale each one operates at

The two things that matter about each stage are what the data is and how much of it there is, and they point in opposite directions: the stage with the most data has the least curation, and the stage with the most careful curation has almost no data.

Put published numbers on that gap. GPT-3 was pretrained on 300 billion tokens (Brown et al. 2020). The InstructGPT work that turned a GPT-3-class base model into something you could talk to used about 13,000 supervised prompts — 12,725 in the training split, averaging 408 tokens each, so on the order of five million tokens of prompt text — plus 33,000 prompts for the reward model and 31,000 for the reinforcement-learning stage (Ouyang et al. 2022).

Pretraining saw roughly 58,000 times the prompt tokens that supervised fine-tuning did. Read the denominator carefully: those five million tokens are prompts only, and an SFT example is a prompt plus a response, with the loss usually taken on the response. The ratio against the complete training text is smaller, and neither paper reports it.

Even so, the order of magnitude is the most useful single fact about this pipeline, because it explains a pattern you would otherwise find baffling: a fine-tune often changes how a model answers strongly, and usually moves what it knows more modestly (how much depends on the data, the method and the evaluation). Facts can be taught later, and Chapter 9 measures both effects. But a few million tokens against three hundred billion is not the place to install a worldview, and retrieval is usually the easier way to keep facts current. The bulk of the knowledge was installed in stage 1. Stages 2 and 3 are, as an analogy, close to a coat of paint — an important one, applied by skilled people, on a building someone else spent tens of millions of dollars constructing.

The word pretraining is a historical accident. It was coined when the stage really was preliminary: you pretrained on unlabelled text, then trained properly on your labelled task. Today this first stage usually consumes the most data and compute, and the name never caught up.

This chapter is about stage 1. Chapter 9 takes stages 2 and 3 seriously, including LoRA, DPO, and what “alignment” actually buys.

Why the order cannot be swapped

It is worth being explicit about why you cannot skip to stage 2, since stage 2 is the one whose data you can actually afford.

The instruction data is far too small to teach a language.

Five million tokens is roughly forty novels. We would not expect a model trained from scratch to reach an assistant’s command of English syntax — let alone Python semantics and the geography of Anatolia — from forty novels. A model can certainly learn something from a corpus that size; broad general competence is not what we would expect it to acquire there, and a model with billions of parameters runs a high risk of mostly memorising them.

What five million carefully written tokens can do is teach a format. That a question is followed by an answer rather than by another question. That a request for a list produces a list. That “I don’t know” is a permitted output.

And format is cheap to learn precisely because the base model already has every ingredient. It simply had no reason to prefer one arrangement over another.

base model — what pretraining produces on its own: something that continues text plausibly, with no particular reason to answer a question rather than ask another one. ↩︎

That is the cause and effect worth carrying. The table has a fourth row that the three-stage picture above leaves out; the paragraph after the table explains it.

stage	supplies	costs	can it be redone?
pretraining	knowledge, grammar, world model	weeks to months for a large model, thousands of GPUs	for most teams, in practice no; rarely, with a large budget
fine-tuning	format, task shape, domain style	hours to days depending on data and model, often one node	often
preference tuning	tone, refusals, ranking of answers	days, plus annotation and candidate generation	usually per release
RL on verifiable rewards	reasoning traces, tool use	days to weeks, a real cluster	depends on infrastructure and budget

A fourth stage arrived after that picture was drawn. Since late 2024, models trained for reasoning add large-scale reinforcement learning against verifiable rewards — maths with a checkable answer, code with a passing test — and that stage does not fit the “coat of paint” description. Its token counts stay small, but its compute does not: rollouts are generation, and generation is the expensive half of RL. The 58,000× ratio above is a fact about data, not about dollars, and Chapter 10 is where the dollars go.

The last column is why this chapter matters even if you never train anything. The pretraining decisions are the ones you inherit and that are expensive to undo. If the tokenizer splits Turkish badly, every user of that model pays for it; changing the tokenizer means re-adapting the model. If the data cutoff is December 2023, later training can teach the model who won an election in 2025 — you then own the coverage, the reliability and the interference, which is why retrieval usually wins for facts that keep moving. If the corpus contained little Turkish, an adapter has little to build on; how much that limits the result has to be measured on the target task.

What we will look at. What pretraining produces and — more usefully — what it does not. Then the three decisions that consume the budget before any architecture question arrives: where the text comes from, how it is cut into tokens, and what the run actually costs. We finish with the landscape of models you can choose from, and how to read a claim about one.

8.2 Where the data comes from

The honest answer to “what is a large language model trained on?” is: the web, plus whatever else the lab could obtain, minus an enormous amount that had to be thrown away during text extraction, language selection, quality filtering and deduplication. The interesting part is the minus: what each step removes, and why.

The raw material

Most open pretraining corpora begin at Common Crawl, a nonprofit that has crawled the web since 2008 and publishes the results. It releases a snapshot every month or two; the FineWeb team worked from 96 of them (Penedo et al. 2024). A snapshot is on the order of hundreds of terabytes of raw HTTP responses, varying with the period and the file type downloaded.

Common Crawl — a nonprofit that has crawled the public web since 2008 and publishes each crawl free of charge. Most open pretraining corpora start here. ↩︎

Two file formats matter, and choosing between them is the first real decision. WARC files hold the raw crawl response — the full page HTML and the request metadata. WET files hold Common Crawl’s own plain-text extraction; using them saves a separate extraction step and its compute. FineWeb used WARC and ran their own extraction with trafilatura, for a reason they state plainly: the WET files “retained too much boilerplate and menu text” (Penedo et al. 2024). Navigation bars, cookie notices, and footer links are text, and a next-token objective will happily learn to predict them.

This is the first appearance of a pattern that runs through the entire pipeline: the model has no way to tell you its data was bad. It minimises the loss it is given, and next-token loss does not tell a cookie banner from the information you want. If cookie banners take up a lot of the corpus, part of the training budget goes to modelling them — say a tenth of the achievable loss reduction lives there, and the model gets good at cookie banners. What that share really is has to be measured on the data.

The funnel

Here is what survives, with the token counts FineWeb published at each stage. The global-deduplication branch at the side is not a further step of the main pipeline but an alternative route tried for comparison.

From 96 crawl snapshots to a usable corpus

Token counts are FineWeb’s own approximate figures, measured with the GPT-2 tokenizer (Penedo et al. 2024). The 4T branch is not a further step of the main pipeline but an alternative route tried for global deduplication; it is the result discussed below: more deduplication is not monotonically better. FineWeb-Edu is a subset selected from the published FineWeb; the 6,000 H100-hours is the cost of applying the classifier to the 15 trillion tokens.

What each filter is actually protecting against

A list of filters is boring. The reason each one exists is not.

filter	removes	the risk it aims to reduce
URL blocklist	pages on an adult-content list	training on material you cannot defend and cannot later remove (misses pages not on the list)
trafilatura extraction	navigation, ads, footers	a model fluent in “Accept all cookies” (useful content can be lost too)
fastText language ID, score \(\ge 0.65\)	pages not confidently English	a corpus whose claimed language is not its actual language
repetition filters	pages that repeat one line 400 times	degenerate loops at generation time (does not prevent them on its own)
quality heuristics	no terminal punctuation, absurd symbol ratios	keyword-stuffed spam no human ever read
MinHash deduplication	near-identical documents	memorisation, and compute spent twice on one text (does not remove memorisation entirely)
PII reformatting	e-mail addresses, public IP addresses	a model that can be prompted into leaking a real person’s details (missed detections and indirect leaks remain)

Two of those rows deserve more than a table cell.

Deduplication earns its place empirically. Models trained on deduplicated data memorise their training set substantially less, and are no worse downstream (Lee et al. 2022). The memorisation is not hypothetical: given the right prompt, a large model will emit long verbatim passages of its training data, including personal information (Carlini et al. 2021). Duplicates make this worse for a mechanical reason — a passage that appears a hundred times is a passage the gradient has rewarded the model a hundred times for reproducing.

The language filter is a policy, not a measurement. A 0.65 threshold on an English classifier does not merely detect language, it defines the corpus. Code-switched text, Turkish written without diacritics, English with heavy dialect features: all tend to score lower, and some of them fall below the threshold and get dropped. What share does should be measured on a sample. When a model is later found weak on some variety of language, this is usually where it happened — silently, months before anybody trained anything.

The deduplication surprise

Now a result that tests the obvious intuition about data pipelines.

Deduplication has an obvious knob: how widely do you look for duplicates? Within one snapshot, or across all 96? Across all of them is more thorough, and the obvious expectation is that it should be better.

deduplication — removing documents that are identical or near-identical to ones already kept, so the model does not spend compute on, or memorise, the same text repeatedly. ↩︎

FineWeb tried exactly that. The two strategies are alternative routes from the same starting point: run on the ~36 trillion tokens of base-filtered text, per-snapshot deduplication leaves 20 trillion and global deduplication leaves 4 trillion — a fifth as much, and roughly a ninth of what went in. It also hit the oldest snapshots hardest, removing “as much as 90% of the original base filtered data” from them, since those had 95 later crawls to be deduplicated against (Penedo et al. 2024).

Then they checked, which is the part most pipelines skip. Taking the 2013-48 snapshot in isolation, they built two pools: the ~31 billion tokens global deduplication had kept, and 171 billion tokens re-deduplicated out of the ~460 billion it had removed. They then trained one ablation model on a sample from each pool, at the same model size and the same token budget, so the only variable was which pool the tokens came from. The kept pool lost.

That equal-budget detail is what makes the result mean anything. Had one model simply seen more tokens than the other, the comparison would have measured data quantity and told you nothing about data quality, which is the whole claim.

Their explanation, after reading documents by hand, is what to memorise: the data surviving aggressive global deduplication “contains more ads, incoherent lists of keywords and generally badly formatted text than originally removed data” (Penedo et al. 2024).

Why this happens, and why it will happen to you. The mechanism is worth getting right, because the obvious explanation is wrong. Deduplication does not throw away good writing — it keeps one copy of everything. What matters is which copy it keeps. FineWeb deduplicated iteratively, starting from the newest snapshot and working backwards, so every document that also appeared in a later crawl was struck from the older ones. Good writing is exactly what gets quoted, syndicated and mirrored forward in time, so it survives in the recent snapshots and is stripped from the old ones. Spam is generated fresh and never reappears, so it stays where it was.

Good writing keeps at least one copy (its old-snapshot copy goes, its new one stays — though that is not guaranteed for every document). What changes is that within any one old snapshot the proportion of low-quality text went up, and the paper says so directly: global deduplication “upsamples lower-quality data in the last deduplicated crawl, resulting in worse performance of the retained data compared to the removed data” (Penedo et al. 2024).

The general lesson: a proxy — here, duplication — was standing in for something you actually wanted, and past a certain strength the proxy inverted. The FineWeb team caught it not by reasoning but by training two models and comparing; their account of the mechanism is a reading of the surviving documents’ composition. Whether the same happens on other corpora has to be tested separately.

This is why FineWeb deduplicates within each snapshot and stops there. It is also a preview of an argument that recurs throughout this book: when a data decision looks obviously right, the cost of checking it with a small training run is often lower than the cost of being wrong at scale; weigh the cost of the experiment against the cost of the decision.

Quantity is not weight

Having collected data, you must decide how much of each kind the model actually sees. These are different numbers, and conflating them is a classic misreading of a model report.

GPT-3’s table makes the distinction unusually clear (Brown et al. 2020):

dataset	tokens available	share of the training mix	times seen in 300B tokens
Common Crawl (filtered)	410B	60%	0.44
WebText2	19B	22%	2.9
Books1	12B	8%	1.9
Books2	55B	8%	0.43
Wikipedia	3B	3%	3.4

Read the last two columns together. Wikipedia is 0.6% of the available tokens and 3% of the training mix, so the model went through the whole of Wikipedia 3.4 times while reading less than half of the filtered web crawl. The authors say why in one sentence: the weights are deliberately not proportional to size, and they accept “a small amount of overfitting in exchange for higher quality training data” (Brown et al. 2020).

Two bookkeeping notes before you reuse this table. The mix column is the paper’s own rounded figures and sums to 101%, not 100%. And the last column is the paper’s reported epoch count, not something you can recompute from the other two: share × 300B ÷ tokens available gives 3.5 for WebText2 against the 2.9 printed, and 3.0 for Wikipedia against the 3.4 printed. These are the figures the original table reports; they do not reconcile exactly. Rounded shares or different counting bases are possible reasons, but the source does not say, and an exact recomputation would need the training sampling procedure. Read the two columns as approximate sampling weights. Read the table for the shape of the argument — quantity is not weight — and not as a set of numbers that close.

Four years later Llama 3 reports its mixture by topic rather than by source, which itself tells you what the field learned to care about: roughly 50% general knowledge, 25% mathematical and reasoning tokens, 17% code, and 8% multilingual (Grattafiori et al. 2024). Note the last figure. Eight percent, shared across every language that is not English, in a model explicitly marketed as multilingual. That total says neither what Turkish’s own share is nor how good the model is in Turkish; those need the language distribution and a measurement on the target task. Keep that number — it returns when we price what Turkish costs.

Code is not in there for the coding. Seventeen percent of a general-purpose model’s diet being source code looks odd until you consider what code is: long-range structured text in which a mistake thousands of tokens earlier makes the continuation invalid, with strict syntax and sometimes automatically testable output. Whether it is causally responsible for better reasoning is still argued; Llama 3’s 17% does not measure that, it only shows what the lab is betting on.

The last forty million tokens

Llama 3’s pretraining is itself three stages, and the proportions are startling (Grattafiori et al. 2024):

1. Initial pretraining. AdamW, peak learning rate \(8\times10^{-5}\), warmed up over 8,000 steps, decayed on a cosine schedule to \(8\times10^{-7}\) over 1.2 million steps. The batch size grows: 4M tokens at sequence length 4,096, doubled to 8M tokens at length 8,192 after 252M tokens, doubled again to 16M after 2.87 trillion tokens. Small batches early buy stability; large batches later buy efficiency.
2. Long-context pretraining. The context window is raised from 8K to 128K in six stages, using roughly 800 billion tokens — about 5% of the whole run spent purely on learning to use a longer window.
3. Annealing. Over the final 40 million tokens the learning rate is taken linearly to zero while the data mix shifts to upsample very high-quality sources. The released weights are then a Polyak average of checkpoints from this phase.

Forty million tokens is \(0.00026\%\) of 15.6 trillion. Can so little matter? The paper tested this in a separate experiment, not on the final 40M-token stage itself. Taking a pre-trained 8B model through an annealing phase on a mix that upweighted high-quality code and mathematics (including the GSM8k and MATH training sets) improved its GSM8k and MATH validation performance by 24.0% and 6.4% (the paper does not say whether these are relative gains or percentage points). On the 405B model the same treatment made a negligible difference, which the authors read as the larger model already having the capability and not needing the reminder.

annealing — taking the learning rate down to zero at the end of a run, usually while shifting the data mix toward the best material available. Borrowed from metallurgy, where slow cooling settles a structure. ↩︎

Where a token sits in the schedule changes what it teaches. A token seen at step 400,000 under a large learning rate contributes to a broad capability that the remaining 800,000 updates will partly overwrite. A token seen in the last 0.00026% of the run, as the learning rate goes to zero, meets very few updates afterwards, so the data mix of the final phase can shape the model’s output (the lasting contribution of one token cannot be read off the schedule). That is the plausible mechanism behind an otherwise mysterious industrial habit: save the best data for last.

How many times may you reuse a token?

Sooner or later every serious pretraining project runs out of the language it wants. This is already true for Turkish, and it will be true for English sooner than the raw token counts suggest. The obvious move is more epochs (complete passes over the training data) over what you have.

The measured answer is more generous than folklore expects (Muennighoff et al. 2023). Training on up to four epochs of repeated data “yields negligible changes to loss compared to having unique data”. Past that, the value of each further epoch decays — and eventually the value of adding compute decays to zero as well, because extra passes over data the model has already exhausted cannot help.

The practical shape of the advice:

your situation	what the evidence supports
more unique tokens than your compute can use	use unique data before repeating any
data-limited, modest compute overrun	try a few epochs of repetition; test the paper’s “up to about four, nearly free” finding against your own validation loss
data-limited, large compute overrun	split it between parameters and epochs, tilting toward epochs
far past both limits	compare the marginal value of another epoch with the cost of new data and compute; in the paper the returns approach zero

The third row is the one that surprises people, so take it from the authors directly: in the data-constrained regime, allocating new compute to “both more parameters and epochs is necessary, and … epochs should be scaled slightly faster” (Muennighoff et al. 2023). Their figure puts it more sharply still — the data suggest “training smaller models for more epochs, in contrast to what assuming Chinchilla scaling laws hold for repeated data would predict”. If you reach for Chinchilla’s ratio when you are short of data, it will tell you to buy parameters, and it will be wrong.

For a Turkish project this is encouraging. A corpus of 30 billion Turkish tokens need not cap you at 30 billion tokens of training: four passes give 120 billion processed tokens, and if the general finding holds, returns do not visibly bend before that. Whether it holds on your corpus — it depends on its diversity and on how it is repeated — is a question for your validation loss; if it does, it changes what model size is defensible.

The second route for projects that run out of data is synthetic data: tokens generated by a strong model, or existing text rewritten (rephrasing), take a growing place in new pretraining corpora. It has limits too: when model outputs are fed back into training generation after generation, without keeping the original real data, the rare tails of the distribution have been shown to erode step by step, and the model learns an ever narrower world. This is called model collapse (Shumailov et al. 2024). How large the risk is depends on the share of synthetic data, how it is generated and selected, and how much real data stays in the mix.

8.3 Subword tokenization: how a model sees text

Chapter 2 cut text into words and counted them; for bag-of-words and TF-IDF that was enough. A pretrained language model does not see text as words. It sees the units its own tokenizer produces, and those are usually smaller than a word. That is what “token” meant all through Chapter 7, where we did not ask where the tokens came from.

This section answers that question. First we build BPE step by step on a small corpus. Then we look at what changes when the starting alphabet is characters or bytes, and at why the same tokenizer cuts a Turkish text into more tokens than its English counterpart. WordPiece, which BERT uses, and Unigram LM, which T5 uses, are two relatives of the same idea; the “Deep dive” boxes at the end of the section run them, and Section 8.4.5 compares them alongside the models. The section after this one (Section 8.4) prices the same decision as a budget line at pretraining scale.

Subwords: the compromise that won

byte-pair encoding (BPE) — a way of learning a subword vocabulary from training data: start from chosen starting units (characters or bytes, depending on the implementation) and merge the most frequent adjacent pair, one merge at a time; the merge list is applied to new text in the same order. ↩︎

With word-level tokenization, a fixed vocabulary may not cover every word form met in new text. Character-level tokenization usually represents the same text with longer sequences, and the model has to learn word pieces and words from those small units. Byte-pair encoding (BPE) offers an option between the two by learning its units from data (Sennrich et al. 2016).

See the three options on one word. Suppose unbreakable never appeared in the tokenizer’s training text. In the comparison we assume that the word-level tokenizer has a fixed vocabulary and writes <UNK> (unknown) for words it does not know. The character-level tokenizer’s alphabet contains every character of the word.

One word at three unit sizes

Unit	How unbreakable looks	Result
Word	<UNK>	The whole word is not in the vocabulary, so its spelling is not kept; the unknown-word marker takes its place.
Character	u n b r e a k a b l e	The spelling of the word is kept, in eleven units. The model learns the patterns inside the word from the order of those units.
Subword	un break able	The word is represented with three units. These pieces occur in other words too: undo, breaking, readable.

The split in the last row is not made up: OpenAI’s cl100k_base tokenizer cuts this word exactly there. Which leaves a question: who chose un, break and able? Nobody picked them one by one. The tokenizer’s designers choose the starting units, the pre-tokenization and the vocabulary budget; the merge rules are found by counting over training data. The four-word example below shows that counting; cl100k_base’s pieces were learned from its own, far larger training data.

In this example BPE training starts by splitting the words into characters. To each word we append </w>, a symbol that marks the end of a word. We count adjacent pairs of symbols, weighting each word by its frequency in the corpus, and merge the most frequent pair into a new symbol. Then we update the counts and repeat. The code below makes ten merges. Which of several equally frequent pairs is chosen first depends on the implementation; this code takes the first one it met.

from collections import Counter

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
words = {tuple(w) + ("</w>",): n for w, n in corpus.items()}

def pair_counts(words):
    pairs = Counter()
    for symbols, freq in words.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs

def merge(words, pair):
    a, b = pair
    out = {}
    for symbols, freq in words.items():
        new, i = [], 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                new.append(a + b)
                i += 2
            else:
                new.append(symbols[i])
                i += 1
        out[tuple(new)] = freq
    return out

merges = []
for step in range(1, 11):
    best, count = pair_counts(words).most_common(1)[0]
    merges.append(best)
    words = merge(words, best)
    print(f"merge {step:>2}: {best[0]!r} + {best[1]!r} -> "
          f"{best[0] + best[1]!r}  (seen {count} times)")

merge  1: 'e' + 's' -> 'es'  (seen 9 times)
merge  2: 'es' + 't' -> 'est'  (seen 9 times)
merge  3: 'est' + '</w>' -> 'est</w>'  (seen 9 times)
merge  4: 'l' + 'o' -> 'lo'  (seen 7 times)
merge  5: 'lo' + 'w' -> 'low'  (seen 7 times)
merge  6: 'n' + 'e' -> 'ne'  (seen 6 times)
merge  7: 'ne' + 'w' -> 'new'  (seen 6 times)
merge  8: 'new' + 'est</w>' -> 'newest</w>'  (seen 6 times)
merge  9: 'low' + '</w>' -> 'low</w>'  (seen 5 times)
merge 10: 'w' + 'i' -> 'wi'  (seen 3 times)

The first two merges build est, which in these words coincides with the English superlative suffix. The third merges it with the end-of-word marker: est</w>. Later steps build low and new as well. The algorithm does not recognise these pieces as stems or suffixes; at each step it picks the most frequent adjacent pair of symbols. In this small corpus some pieces coincide with morphemes; BPE does not guarantee that they will, and this example does not show what the overlap buys a model.

BPE can also split words that are not in the tokenizer’s training corpus into learned pieces. To tokenize a new word in this example, we apply the merge rules in the order they were learned:

def apply_merges(word, merges):
    symbols = tuple(word) + ("</w>",)
    for pair in merges:
        symbols = tuple(merge({symbols: 1}, pair).keys())[0]
    return symbols

for w in ["lowest", "newer", "slow", "widest"]:
    print(f"{w:<8} -> {' '.join(apply_merges(w, merges))}")

lowest   -> low est</w>
newer    -> new e r </w>
slow     -> s low</w>
widest   -> wi d est</w>

lowest is not in the tokenizer’s training corpus, and yet it is represented with two units from the vocabulary: low and est</w>. This is the main reason subwords became common: in this BPE setup, as long as the characters of the starting alphabet remain available, a word never seen before can still be built from those characters and their learned merges. Some pieces, like the s in slow, can stay single characters. Using smaller pieces can raise the number of tokens; how that affects a language model’s performance has to be evaluated separately.

Notice the condition: every character of the new word has to be in the alphabet the merges were learned over. Ours came from four English words. What happens when a letter outside it arrives, a ç for instance? Section 8.3.2 takes that up after the figure.

What byte-pair encoding learns, and how it uses it

In training (top), the frequencies of symbol pairs are counted and an ordered list of merges is built. In this small corpus the first three merges build est</w>, which includes the end-of-word marker. That unit coincides with the superlative suffix of the example words, but the choice rests on the frequency of pairs, not on grammatical analysis. In inference (bottom), the learned list is applied to a new word. lowest, which is not in the training corpus, is split into the vocabulary units low and est</w> (Sennrich et al. 2016).

Byte-pair encoding — type a word and watch the merges apply

viewof bpeWord = Inputs.text({
  label: "Word",
  value: "lowest",
  placeholder: "try: lowest, newer, slow, widest, snow",
  width: 260
})

// Verbatim output of the merge loop at the top of this section — if the small
// corpus changes, regenerate both together.
bpeMerges = [
  ["e", "s"], ["es", "t"], ["est", "</w>"], ["l", "o"], ["lo", "w"],
  ["n", "e"], ["ne", "w"], ["new", "est</w>"], ["low", "</w>"], ["w", "i"]
]

bpeTrace = {
  const raw = bpeWord || "";
  const alphabet = new Set("deilnorstw");
  const invalid = [...new Set([...raw].filter(c => !alphabet.has(c)))];

  const trace = [];
  trace.invalid = invalid;
  if (!raw.length || invalid.length) return trace;

  let symbols = [...raw, "</w>"];
  trace.push({ step: 0, rule: "start (characters)", symbols: [...symbols] });
  bpeMerges.forEach(([a, b], k) => {
    const out = [];
    let i = 0, used = false;
    while (i < symbols.length) {
      if (i < symbols.length - 1 && symbols[i] === a && symbols[i + 1] === b) {
        out.push(a + b); i += 2; used = true;
      } else { out.push(symbols[i]); i += 1; }
    }
    symbols = out;
    if (used) trace.push({ step: k + 1, rule: `${a} + ${b}`, symbols: [...symbols] });
  });
  return trace;
}

html`<div style="font-size:0.9rem;">
  ${bpeTrace.invalid.length
    ? html`<p style="color:var(--fig-warm);">
        This input contains characters that are not in the training alphabet:
        <code>${bpeTrace.invalid.map(c => JSON.stringify(c)).join(", ")}</code>.
        The input was left as it is and no merges were applied. This example
        supports only the lowercase letters that occur in its four-word corpus:
        <code>d, e, i, l, n, o, r, s, t, w</code>.</p>`
    : bpeTrace.length === 0
      ? html`<p style="color:var(--fig-muted);">Type a word made of letters from the training alphabet.</p>`
      : html`<div>
        ${bpeTrace.map(t => html`<div style="display:flex; gap:0.8rem;
             align-items:baseline; padding:0.22rem 0; border-bottom:1px solid var(--fig-border);">
          <span style="min-width:8.5rem; color:var(--fig-muted); font-size:0.82rem;">
            ${t.step === 0 ? t.rule : `merge ${t.step}: ${t.rule}`}
          </span>
          <span>${t.symbols.map(s => html`<code style="background:var(--fig-chip);
            border:1px solid var(--fig-border); border-radius:0.35em; padding:0.05em 0.35em;
            margin-right:0.2em;">${s}</code>`)}</span>
        </div>`)}
        <p style="margin-top:0.7rem; color:var(--fig-muted);">
          Final: <b>${bpeTrace[bpeTrace.length - 1].symbols.length}</b> symbol(s)
          (a lone <code>&lt;/w&gt;</code> counts too; this is the toy example's
          symbol count, not a real model's token count).
          10 merges were learned from this small corpus. Large BPE
          vocabularies hold far more; GPT-2's tokenizer has 50,000 merges.
        </p>
      </div>`}
</div>`

Characters or bytes: what BPE starts from

The example above started from characters. Tokenizers that use byte-level BPE, such as GPT-2’s and cl100k_base, start from bytes. To see the difference, we explain the terms character, code point, character encoding and byte through ASCII, Unicode and UTF-8.

A character is an abstract unit that text is made of; letters, digits, punctuation marks and spaces are examples. ç, a, y, 7, ? and 😀 are visible ones. But a unit perceived as a single sign on screen can be represented in Unicode by one code point or by several.

Computers store text as sequences of bits. A character encoding determines how the units of text are represented by those sequences. ASCII is a 7-bit character encoding whose first version was published in 1963. Its 128 code values cover the English letters, the digits, punctuation, the space and control characters. The code value of the letter a is 97. ASCII has no ç, ş or ğ, no Arabic or Chinese script, and no emoji.

Unicode is the standard that aims to represent the characters of different writing systems in one shared system. The numerical positions in Unicode’s code space are called code points; encoded characters are assigned to them. A code point is usually written as U+ followed by the number in hexadecimal: the code point of ç is 231, that is U+00E7. The 128 ASCII characters keep their numbers in Unicode, so a is still 97.

A byte is the basic unit a computer stores data in. It is eight bits and holds a number from 0 to 255, so it can take only 256 different values. Unicode’s numbers go far beyond 255, so a code point does not always fit in one byte. An encoding decides which bytes store a code point. UTF-8, which is widely used on the web, represents every encodable Unicode code point with one to four bytes. Code points in the ASCII range are encoded as a single byte, and larger values as two, three or four bytes, using bit patterns that UTF-8 defines.

Here are all three side by side on the Turkish word çay (tea):

çay: three code points, and four bytes in UTF-8, when ç is written as U+00E7

	ç	a	y
Unicode code point	231 (U+00E7)	97 (U+0061)	121 (U+0079)
In the ASCII table?	no	yes	yes
UTF-8 bytes	195, 167	97	121

In this spelling çay is three letters and three code points, and takes four bytes when encoded in UTF-8. In Python len("çay") is 3 and len("çay".encode("utf-8")) is 4. Text that looks the same can also be written with a different sequence of code points: write c (U+0063) followed by the combining cedilla (U+0327, COMBINING CEDILLA) instead of ç, and the word looks the same but is four code points and five bytes. These numbers depend on the Unicode spelling and on the encoding; the token count can also change with whether the tokenizer applies Unicode normalization.

Why does a tokenizer care? Because the symbols BPE starts merging from are either characters or bytes, and there are very different numbers of each:

• A character alphabet can be large. Unicode’s code space is finite, but it holds a great many characters, and new versions define new ones. A tokenizer’s starting alphabet can be chosen from the training data; characters can be added to it, or its coverage limited. A character that is not in that alphabet needs an extra mechanism, such as an unknown token or a fallback to bytes.
• A byte alphabet can be complete with 256 values. If the starting alphabet contains every byte value, every byte of UTF-8 encoded text can be represented. The bytes of a new character then need no new starting symbol.

The rest of this section works through the two options and what each costs.

First a short note on history. The loop above — count adjacent pairs, merge the most frequent, repeat — is BPE. The name is older than its use in NLP: it comes from a 1994 data-compression algorithm that kept replacing the most frequent pair of adjacent bytes in a file with a single byte value the file did not use (Gage 1994). Sennrich and colleagues kept the loop and changed its purpose: the merges build a vocabulary instead of shrinking a file, and the merge list is kept so it can be replayed on new text (Sennrich et al. 2016).

That leaves one decision the loop does not make for you: what the starting symbols are. Do not confuse it with the choice of algorithm; the two are separate decisions. BPE is not the only way to learn a subword vocabulary. You will meet two relatives of it later: WordPiece, which BERT uses, and Unigram, which T5 uses. All three are used to build subword vocabularies, but they differ in how the vocabulary is learned and in how new text is split into pieces. Our main example here is BPE. WordPiece and Unigram come up in a “Deep dive” box in this chapter (Section 8.3.4), and the methods are compared in more detail where we look at BERT’s tokenizer (Section 8.4.5). What you need for now: the algorithm decides how units are learned. Characters or bytes decide what they are built from, and that question is the same for all three algorithms.

Starting from characters. This is what the example above did. The merge table can only combine symbols it has seen, and the alphabet this one saw came from four English words. apply_merges("çay", merges) returns ('ç', 'a', 'y', '</w>'). But ç, a and y are not in the starting alphabet built from this small corpus. The function only applies merge rules and does not check for symbols outside the vocabulary, so it leaves them in the output. A usable tokenizer has to handle this case separately: it can add the characters to its starting alphabet, use an unknown token (<UNK>), or fall back to bytes. That a character is rare does not mean it must become an unknown token.

Starting from bytes. The other option is to start from the UTF-8 bytes of the characters instead of the characters themselves. UTF-8 represents every encodable Unicode code point with one to four bytes. ASCII letters take one byte. In the spellings below, each of the Turkish letters ç, ş, ğ, ı, ö, ü and İ is two bytes; 世 is three and 😀 is four. Emoji and other combined spellings made of more than one code point can be longer than four bytes in total:

for ch in ["a", "ç", "ş", "İ", "世", "😀"]:
    b = ch.encode("utf-8")
    print(f"{ch}  U+{ord(ch):04X}  {len(b)} byte(s): {' '.join(f'{x:02X}' for x in b)}")

a  U+0061  1 byte(s): 61
ç  U+00E7  2 byte(s): C3 A7
ş  U+015F  2 byte(s): C5 9F
İ  U+0130  2 byte(s): C4 B0
世  U+4E16  3 byte(s): E4 B8 96
😀  U+1F600  4 byte(s): F0 9F 98 80

A byte has only 256 possible values, so an alphabet of 256 bytes is closed: any text encoded in UTF-8, in whatever language, is a sequence of them. Byte-level BPE runs exactly the same merge loop over byte sequences instead of characters (Radford et al. 2019). At each step the loop merges the most frequent adjacent pair of symbols. Some characters and word pieces can end up represented as single tokens that way. The two bytes of ç, for example, can become one token; whether that happens, and at which step, depends on the frequencies in the training data and on the number of merges made. But the merges are not bound to Unicode character boundaries: the algorithm does not know where a character ends, only which byte pair is frequent. So a token can stand for a whole character, for several characters, or for only part of one UTF-8 character. The next output shows all three.

Here is a byte-level tokenizer, OpenAI’s cl100k_base, showing the bytes behind each token. enc.encode returns the token ids as a list of integers. Text is passed to the model through those ids. decode_single_token_bytes shows the sequence of bytes each id stands for:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
for text in ["çay", "şişe", "ŞİŞE", "😀"]:
    pieces = [enc.decode_single_token_bytes(t) for t in enc.encode(text)]
    print(f"{text:<5} {len(pieces)} tokens  {pieces}")

çay   2 tokens  [b'\xc3\xa7', b'ay']
şişe  3 tokens  [b'\xc5\x9fi', b'\xc5\x9f', b'e']
ŞİŞE  6 tokens  [b'\xc5', b'\x9e', b'\xc4\xb0', b'\xc5', b'\x9e', b'E']
😀     2 tokens  [b'\xf0\x9f\x98', b'\x80']

In this output the two bytes of ç sit in a single token. ŞİŞE produces six tokens and şişe three. The two bytes of capital Ş are split over separate tokens, while İ and E stay whole. From this output alone we cannot tell how often Ş occurred in the training data. b'\xc5' is not a valid UTF-8 sequence on its own; it has to be joined with the next byte before the character can be read. 😀 is split into two tokens as well: the first holds three bytes and the second the last one. None of these examples needs an unknown token; but the same number of visible characters can produce different numbers of tokens.

In the table below, “character” means a Unicode code point. The byte-level column assumes that the starting alphabet contains all 256 byte values.

Characters or bytes as the starting alphabet

	BPE starting from characters	BPE starting from bytes
Starting alphabet	Selected code points; they can be taken from the training data or added separately.	The 256 byte values.
A character that is not in the starting alphabet	Needs an extra mechanism, such as an unknown token or the byte fallback below.	Represented by its UTF-8 bytes.
What does one token stand for?	One or more whole code points.	A sequence of bytes; sometimes only part of the UTF-8 form of a code point.
Cost of a code point that is not merged into a larger piece	One token, if it is in the alphabet.	One to four tokens, depending on its UTF-8 form and on the learned merges.
Coverage	Limited to the chosen alphabet and to the fallback mechanism, if there is one.	Covers every byte of UTF-8 encoded text.

A third design combines the two: byte fallback, which means spelling an unknown character out in bytes. The vocabulary is still built from characters and their merges, but one token for each of the 256 byte values is added to it. The tokenizer handles the characters it knows as usual. When it meets one it does not know, it writes that character’s UTF-8 bytes one by one with those byte tokens instead of writing <UNK>. The SentencePiece library offers this as an option (byte_fallback=True).

A small experiment shows it. Below, SentencePiece is trained twice on eight Turkish words that contain no ç: once with the option off and once with it on. Both models are then given the word çay:

import sentencepiece as spm

words = ["ev", "evde", "evden", "evler", "yol", "yolda", "yoldan", "yollar"]
with open("mini.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(words * 5) + "\n")            # no 'ç' anywhere in this text

for fallback in [False, True]:
    spm.SentencePieceTrainer.train(
        input="mini.txt", model_prefix="mini", model_type="bpe",
        vocab_size=20 + (256 if fallback else 0), byte_fallback=fallback,
        character_coverage=1.0, hard_vocab_limit=False, minloglevel=2,
        bos_id=-1, eos_id=-1, pad_id=-1)
    sp = spm.SentencePieceProcessor(model_file="mini.model")
    ids = sp.encode("çay")
    print(f"byte_fallback={fallback!s:<5}  vocabulary {sp.get_piece_size():>3}  "
          f"{sp.encode('çay', out_type=str)}  ->  {sp.decode(ids)!r}")

byte_fallback=False  vocabulary  20  ['▁', 'ç', 'a', 'y']  ->  ' ⁇ ay'
byte_fallback=True   vocabulary 276  ['▁', '<0xC3>', '<0xA7>', 'a', 'y']  ->  'çay'

In the first line ç is not in the vocabulary and is encoded with the unknown token’s id. The list of pieces still shows ç, because that list prints the unknown piece as it is spelled in the text. It does not mean the vocabulary has an entry for ç. When the ids are turned back into text, a ⁇ mark takes the letter’s place.

In the second line the vocabulary has grown from 20 to 276: in this experiment we raised the target vocabulary size by 256 when byte fallback was on, and the 256 byte pieces filled that room (with other training settings, check the actual vocabulary size). ç is represented as <0xC3> and <0xA7>. Those are the two bytes from the table above: hexadecimal C3 is 195 and A7 is 167. Turned back into text from the ids, çay comes out intact. The leading ▁ is the mark SentencePiece uses to make a space visible. Here it stands for the space that the default setting adds in front of the text; the next section comes back to this notation.

Which one is used today. The practical rule first: use the tokenizer the model was trained with, and count tokens with it; the same algorithm name does not mean the same vocabulary in two models. Many GPT-type models use byte-level BPE or a variant of it. Other model families also use WordPiece and Unigram. SentencePiece is a library that can train a tokenizer with either BPE or Unigram. GPT-2 introduced byte-level BPE with a 50,257-token vocabulary (Radford et al. 2019). OpenAI’s cl100k_base and o200k_base, measured in the next section, are byte-level. Llama 3 moved to a 128K-token vocabulary that combines 100K tokens from OpenAI’s tiktoken with 28K added for non-English languages (Grattafiori et al. 2024). BERT and its close relatives such as DistilBERT and ELECTRA still ship WordPiece over characters (Devlin et al. 2019) (RoBERTa, although built on BERT, switched to GPT-2’s byte-level BPE), and a later table (Section 8.4.5) shows which algorithm ships where.

Coverage and token count have to be kept apart here. If the starting alphabet contains all 256 byte values, no unknown token is needed to represent the bytes of a Turkish word encoded in UTF-8. That does not mean the normalization that runs before tokenization preserves the original text: letters it changed or spaces it deleted cannot be brought back by byte coverage.

How many tokens a text produces depends on the training data, the learned merges, and the size and content of the vocabulary. No separate vocabulary quota needs to have been set for each language. The next section measures this difference in token counts on Turkish examples.

This section in short. The coverage of BPE that starts from characters depends on the chosen starting alphabet and on extra mechanisms such as byte fallback, if there are any. BPE that starts from all 256 byte values can represent the bytes of any UTF-8 encoded text. In return, some code points can be split over more than one token. In both approaches the number of tokens depends on the learned vocabulary and on the content of the text.

Who pays for Turkish

A tokenizer can produce different numbers of tokens for the same content in different languages. The difference depends on the language mix and the content of the training corpus, on the vocabulary size, on the tokenization algorithm and on the preprocessing rules. Let us look at the difference between Turkish and English on a concrete example. Here is Article 1 of the Universal Declaration of Human Rights, in the official English and Turkish texts, run through OpenAI’s tokenizers:

import tiktoken

en = ("All human beings are born free and equal in dignity and rights. "
      "They are endowed with reason and conscience and should act towards "
      "one another in a spirit of brotherhood.")
tr = ("Bütün insanlar hür, haysiyet ve haklar bakımından eşit doğarlar. "
      "Akıl ve vicdana sahiptirler ve birbirlerine karşı kardeşlik "
      "zihniyeti ile hareket etmelidirler.")

for name in ["cl100k_base", "o200k_base"]:
    enc = tiktoken.get_encoding(name)
    a, b = len(enc.encode(en)), len(enc.encode(tr))
    print(f"{name}:")
    print(f"  english : {len(en):>4} code points -> {a:>3} tokens  "
          f"({len(en) / a:4.2f} code points/token)")
    print(f"  turkish : {len(tr):>4} code points -> {b:>3} tokens  "
          f"({len(tr) / b:4.2f} code points/token)")
    print(f"  Turkish/English token count ratio: {b / a:4.2f}")

cl100k_base:
  english :  170 code points ->  33 tokens  (5.15 code points/token)
  turkish :  160 code points ->  60 tokens  (2.67 code points/token)
  Turkish/English token count ratio: 1.82
o200k_base:
  english :  170 code points ->  33 tokens  (5.15 code points/token)
  turkish :  160 code points ->  46 tokens  (3.48 code points/token)
  Turkish/English token count ratio: 1.39

In these two-sentence texts of the same article, the Turkish token count is about 1.82 times the English one with cl100k_base, and 1.39 times with o200k_base. In this example the second encoding narrows the difference.

In a comparison where the same input price per token applies, more tokens mean a higher input cost. A fixed token limit can also correspond to different amounts of content in texts like these. The total cost, though, depends on the model chosen, on the lengths of input and output, and on the pricing terms.

Similar differences between languages have been shown in broader studies (Petrov et al. 2023). The two texts here do not give a universal ratio for Turkish. Nor is it enough to explain the difference by some Turkish letters taking two bytes in UTF-8; the learned vocabulary and the tokenization rules act together.

Where does the difference come from? Turkish encodes far more grammatical information inside a single word, with suffixes, than English does (Oflazer 1994), so one stem produces many surface forms. English expresses the same grammatical information with separate words in some constructions. If those words are frequent in the training corpus, they can be represented with short token sequences. In Turkish, how the many inflected forms are split depends on which shared sequences the tokenizer has learned.

Below you can see how the output of cl100k_base changes as suffixes are added to the stem ev (“house”). We encoded each word separately, with a single space in front of it:

ev             1 tokens  [' ev']
evler          2 tokens  [' ev', 'ler']
evlerimiz      4 tokens  [' ev', 'ler', 'im', 'iz']
evlerimizden   5 tokens  [' ev', 'ler', 'im', 'iz', 'den']

Morphemes and BPE pieces in one Turkish word

The accent-coloured box shows the stem ev. The other boxes hold suffixes or parts of suffixes: -imiz is split over two tokens, im and iz. BPE learns these units from the merge statistics of its training data, not from rules of grammar. The Turkish form in the last row is shown as five tokens and its English equivalent as three. If the same price per token applies, the difference carries over to the cost of processing those pieces of text. ␣ is the sign used here to make a real space visible.

This comparison shows that using fewer words does not mean producing fewer tokens. How far token boundaries coincide with morphemes is a separate question.

Three of the four internal boundaries in ev|ler|im|iz|den coincide with the morpheme analysis we use here, ev|ler|imiz|den (analyses that cut the suffixes differently are possible too). The extra boundary falls inside the first person plural possessive suffix -imiz (“our”). So this example shows that BPE pieces can partly coincide with morphemes without representing them one for one. From the output alone we cannot tell how frequent im and iz were in the training data.

o200k_base does not split -imiz in the same word and cuts it into four tokens: ev|ler|imiz|den. The difference shows that one word can be represented differently by different tokenizers; on its own it is not a comparison of how well they do on Turkish words in general.

Bostrom and Durrett (2020) found, in English and Japanese experiments run at equal vocabulary size, that the boundaries of Unigram LM, a relative of BPE (Section 8.4.5), coincided with morphological references better than BPE’s did. For Turkish the result would have to be tested separately.

One detail in that output is easy to miss: the first token is not 'ev' but ' ev', with a space in front. To see why, look at how a tokenizer handles a whole sentence.

Pre-tokens first, then tokens. GPT-2’s BPE tokenizer and BERT’s WordPiece tokenizer cut the text into smaller pieces before any subword operation. This stage is called pre-tokenization. GPT-2 separates runs of letters, digits, punctuation and spaces with a regular expression. BERT’s basic tokenization takes account of whitespace, punctuation and boundaries specific to some writing systems. BPE or WordPiece then works on those pieces. The resulting tokens cannot cross the boundaries that pre-tokenization set; but those boundaries are not word boundaries in every language.

How are spaces represented? To put the pieces back together you need to know how the boundaries between them were kept. Seeing only the sequences ev, ler, geld, ik is not enough to place the spaces: is it evler geldik, or ev ler geld ik? Tokenizers handle this information in different ways: some keep the spaces inside tokens, some show a space with a visible sign, and some mark the pieces that continue a word. These methods do not all recover the original text equally well. Here are the same two words, evlerimizden geldik (“we came from our houses”), in three tokenizer families:

Three ways of showing boundary information, on the same two words

Family (tokenizer that was run)	Tokens	How the boundary information is shown
GPT (cl100k_base)	ev ler im iz den ␣geld ik	In this example the space sits in the same token as geld; ␣ is used to make the real space visible.
SentencePiece (XLM-R)	▁ev lerimiz den ▁geldi k	▁ is the visible form of a space; it can also stand for the leading space the tokenizer adds at the start of the text.
WordPiece (multilingual BERT)	ev ##leri ##mi ##zde ##n geldi ##k	## says that the piece continues the same pre-token; it does not record how many spaces there were.

These notations carry different information about how the pieces are to be joined. The ' ev' in the cl100k_base output is a token that starts with a real space. WordPiece’s ## marks a continuation piece; it does not show the number or the kind of the original spaces. In the figure we used the sign ␣ to make the real space visible: ␣ev.

SentencePiece (Kudo and Richardson 2018) does not need the training text to be split into words beforehand; it can work from raw text. It represents spaces with ▁. The pieces it can form are still limited by its settings: the default split_by_whitespace=True, for example, limits merging the words on either side of a space into one piece. Not needing prior word segmentation makes it easier to use on text that is not separated by spaces, such as Japanese and Chinese.

Following tokenization in four stages. In the Hugging Face tokenizers library the process can be examined as normalization, pre-tokenization, subword encoding and post-processing. A tokenizer does not have to use all of these stages. Below we follow one Turkish sentence (“We drank tea, then we left our houses.”) through multilingual BERT’s tokenizer, step by step (with tokenizers 0.22):

from tokenizers import Tokenizer

tok = Tokenizer.from_pretrained("google-bert/bert-base-multilingual-uncased")
text = "Çay içtik, sonra evlerimizden çıktık."

norm = tok.normalizer.normalize_str(text)
print("1 normalize    :", norm)
print("2 pre-tokenize :", [w for w, _ in tok.pre_tokenizer.pre_tokenize_str(norm)])
print("3 subwords     :", tok.encode(text, add_special_tokens=False).tokens)
print("4 special      :", tok.encode(text).tokens)
print("  ids          :", tok.encode(text).ids)

1 normalize    : cay ictik, sonra evlerimizden cıktık.
2 pre-tokenize : ['cay', 'ictik', ',', 'sonra', 'evlerimizden', 'cıktık', '.']
3 subwords     : ['cay', 'ic', '##tik', ',', 'sonra', 'ev', '##leri', '##mi', '##zde', '##n', 'cıktı', '##k', '.']
4 special      : ['[CLS]', 'cay', 'ic', '##tik', ',', 'sonra', 'ev', '##leri', '##mi', '##zde', '##n', 'cıktı', '##k', '.', '[SEP]']
  ids          : [101, 21373, 11797, 13537, 117, 12624, 18110, 12737, 10555, 55567, 10115, 59161, 10167, 119, 102]

The first four lines show the processing stages, and the last line the token ids that result:

1. Normalization. This BERT tokenizer lowercases the text and, after Unicode decomposition, removes combining marks. So Çay → cay and içtik → ictik. Information deleted at this stage does not come back; the model can no longer tell çay (tea) from cay. ı is kept, because it does not decompose into an i and a mark. This is this normalizer’s behaviour, not Turkish lowercasing: the same normalizer turns ISPARTA into isparta and Iğdır into igdır (re-run), whereas in Turkish a lowercased I is ı. Do not generalize this output to another tokenizer.
2. Pre-tokenization. The text is cut into pre-tokens at whitespace and punctuation. The comma and the full stop are pieces of their own.
3. Subword encoding. WordPiece represents those pre-tokens with units from its vocabulary. In the example evlerimizden is five pieces; sonra is whole.
4. Post-processing. [CLS] is added at the start of this single-text BERT input and [SEP] at the end.

The integers on the last line are the token ids. A model can use auxiliary inputs beside them, such as an attention mask.

bert-base-multilingual-cased keeps the distinction between upper and lower case, but its normalization stage is not empty: text cleaning (dropping control characters, for instance) and adding spaces around Chinese characters are active. The cl100k_base and o200k_base encodings used here have no lowercasing or accent-stripping stage of the kind BERT has. SentencePiece can work without external word segmentation; it has normalization and piece-forming settings of its own.

That last stage shows up in the output immediately, so here are the names of the special tokens:

token	function
[UNK]	Takes the place of a piece that the encoding scheme in use cannot represent. A tokenizer with full byte coverage does not need it to represent the bytes of valid text.
[PAD]	Fills the sequences processed together (in one batch) up to a target length. Padded positions are excluded with the appropriate attention mask.
[CLS]	The special token at the start of a BERT input. The contextual representation at this position can be used in tasks such as sequence classification.
[SEP]	In BERT, marks the end of a text and the boundary between two texts, a question and a passage for instance.
BOS / EOS	The beginning-of-sequence and end-of-sequence markers a model uses. In generation EOS can serve as the stop signal.

The names and the jobs of special tokens differ by model. Markers such as <s>, </s> and <|endoftext|> are not interchangeable; follow the token ids and the placement rules the model expects. When you start counting your own tokens, remember that these take their share of the context window too.

One last practical detail for anyone working with text like Turkish. A letter with a mark, such as ö, can be stored in two different ways: as a single code point, or as the letter o followed by a separate “two dots” mark. The two can look the same, but they are different sequences of code points. Depending on the tokenizer’s normalization rules they can produce the same token sequence or different ones. When you use a pretrained model, follow the processing its tokenizer expects. If you decide to use the standard form called NFC in your own data pipeline, apply it consistently in training and in use:

import unicodedata

text = unicodedata.normalize("NFC", text)

In comparisons of token counts, which Unicode normalization was applied to the inputs should be stated as part of the measurement conditions.

TipIntuition check: why not just use characters and skip all this?

Working with a limited character set can give a small vocabulary; but the characters met in real text are not only those of the language’s alphabet. At the byte level, every value can be covered with 256 units. With a fixed character or byte vocabulary, cutting text into those units needs no learned merge rules of the kind BPE has. The language model still has to be trained. Since character-level and byte-level models exist too (Xue et al. 2022), let us look at what subwords gain.

We count three different units in the Turkish text of Article 1 above: the Unicode code points Python separates, cl100k_base tokens, and the pieces of text between spaces. In the last group punctuation stays attached to the word. The text form of the tokens is used only to make the output easier to read; decoding a single byte token on its own does not always give a valid piece of Unicode.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
units = {
    "code points": list(tr),
    "subword tokens": [enc.decode([t]) for t in enc.encode(tr)],
    "words": tr.split(),
}
for name, seq in units.items():
    n = len(seq)
    print(f"{name:<15} {n:>3} units  {n * n:>6,} pairs  first six: {seq[:6]}")

code points     160 units  25,600 pairs  first six: ['B', 'ü', 't', 'ü', 'n', ' ']
subword tokens   60 units   3,600 pairs  first six: ['B', 'üt', 'ün', ' insan', 'lar', ' h']
words            21 units     441 pairs  first six: ['Bütün', 'insanlar', 'hür,', 'haysiyet', 've', 'haklar']

The first reason is sequence length. In full self-attention, each attention head handles position relations of size \(n \times n\) (Chapter 7). The pairs column of the output is that theoretical number of position pairs. In this example 160 code points correspond to 25,600 position pairs and 60 subword tokens to 3,600. The sequence gets about 2.7 times shorter while that number falls about 7.1 times. That pair count on its own is not GPU time or an API fee.

At a fixed model width, the computation of full attention grows quadratically with sequence length and the per-position operations of the layer linearly (Vaswani et al. 2017). In this simplified model, lengthening the sequence five times raises the sum of the two components by between five and twenty-five times. That ratio is not a direct estimate of running time or of price; in architectures that use local or sparse attention the growth is different, too.

The second is the size of the unit. Read the first six units of each row. Code points such as ü and t contribute to the spelling of a word; but a larger pattern such as insan (“human”) is not presented as a single input unit. A character-level model has to learn such patterns from sequences of characters. In the subword row, insan and lar are separate units as they stand. How that affects learning and task performance has to be evaluated on the model and the data.

Splitting on spaces gives a shorter sequence still, and brings other problems. In this example splitting on spaces produces 21 units. But tr.split() does not separate punctuation, so hür, stays a different piece from hür. A more careful word tokenizer can fix that. What remains is that inflected forms are kept as separate entries in a fixed vocabulary; new forms that are not in the vocabulary need a solution of their own. The next example shows this.

In subword tokenization there is a trade-off between vocabulary size and sequence length. Taking larger pieces into the vocabulary can shorten the sequences; in return more vocabulary entries are needed. The designer sets the target vocabulary size and the starting units; under those conditions BPE learns which pieces to form from the pair frequencies in the training data. The approach resembles a compression problem, and it fits that BPE first appeared as a data-compression algorithm in 1994 (Gage 1994). The goal here, though, is not to make a file smaller but to choose the units the model will use.

Characters, subwords, whole words: one example

The table below compares the three choices on paper. First, watch all three on one small example, so that every row of the table is something you have seen happen.

The corpus is made of the bare forms of twelve stems and of the forms built with four endings (-ler, -de, -den, -lerde): 60 different word forms in all. The endings are spelled with vowel harmony (-ler/-lar, -de/-da). Counting the frequency the code gives each form, the corpus stands for 216 word occurrences in total.

The two test words, evlerden (“from the houses”) and okullardan (“from the schools”), are not in the corpus; but the characters that make them up and the endings involved occur in other forms. The word and character methods list the units in the corpus, while BPE learns merge rules from the frequencies. All three are then applied to the same two test words. The BPE line reuses pair_counts, merge and apply_merges from Section 8.3.1:

stems = {"ev": "e", "el": "e", "göl": "e", "köy": "e", "gün": "e", "yüz": "e",
         "okul": "a", "yol": "a", "kol": "a", "yıl": "a", "kız": "a", "masa": "a"}
corpus = {}
for stem, v in stems.items():          # v: the vowel the suffixes take
    for suffix, n in [("", 4), (f"l{v}r", 2), (f"d{v}", 4),
                      (f"d{v}n", 4), (f"l{v}rd{v}", 4)]:
        corpus[stem + suffix] = n      # ev, evler, evde, evden, evlerde, ...
test = ["evlerden", "okullardan"]      # neither word is in the corpus

chars = sorted({c for word in corpus for c in word})
words = {tuple(w) + ("</w>",): n for w, n in corpus.items()}
merges = []
while len(chars) + 1 + len(merges) < 60:   # stop at a 60-symbol vocabulary
    best, _ = pair_counts(words).most_common(1)[0]
    merges.append(best)
    words = merge(words, best)

tokenizers = {
    "whole words": (len(corpus), lambda w: [w if w in corpus else "<UNK>"]),
    "characters": (len(chars), lambda w: list(w)),
    "subwords (BPE)": (len(chars) + 1 + len(merges),
                       lambda w: list(apply_merges(w, merges))),
}
for name, (size, split) in tokenizers.items():
    pieces = [p for w in test for p in split(w)]
    print(f"{name:<15} vocabulary {size:>2}   {len(pieces):>2} units: {' '.join(pieces)}")

whole words     vocabulary 60    2 units: <UNK> <UNK>
characters      vocabulary 18   18 units: e v l e r d e n o k u l l a r d a n
subwords (BPE)  vocabulary 60    6 units: ev ler den</w> okul lar dan</w>

Now read that output against the rows of the table below:

• Vocabulary size. The word-level method has 60 different word forms. That number, as printed, does not include the <UNK> marker it also uses. The character method lists 18 different characters. The 60 shown for BPE covers the 18 characters, the </w> symbol and the units added by 41 merges. So the numbers should not be read as a comparison that counts special and structural symbols by one and the same rule. Real Turkish data holds many inflected forms beyond these four endings.
• Out-of-vocabulary words. The word-level method represents both test words with the same <UNK> marker. Their spellings, and the fact that they differ from each other, are lost. The character and BPE methods can represent these two inputs with existing units, because every character they need is in the starting alphabets. The functions used in this small example do not check separately for characters outside the alphabet.
• Sequence length. The three methods do not keep the same information, so the counts below are not an efficiency ranking. The two test words, processed separately, produce 18 character units, 6 BPE units or 2 <UNK> units in the output shown. The last is the shortest, but it loses the spelling of the words. The character count also has no separator between the words, while the BPE output uses an end-of-word marker.
• What the units correspond to grammatically. e, v and l are characters that make up the spelling of the word; in this example they do not stand for a stem or a suffix on their own. In the BPE output ev, ler and den coincide with the stem, the plural suffix and the ablative suffix respectively. </w> is a structural marker for the end of a word.
• Morphologically rich languages. The BPE units learned on this small corpus coincide with stem and suffix boundaries in the two test words. The algorithm does not label these pieces as stems or suffixes; on other words, or with another training corpus, the same match may not occur.
• Tokenizer training. Only the BPE line ran a learning loop. The other two listed the units they saw in the corpus.

In short, in this example the word level gives short sequences but loses the spelling of unseen forms; the character and byte levels give coverage but lengthen the sequence; subwords look for a balance between the two. The table below compares that split along coverage, length and morphology. Decisions about normalization, keeping boundaries, language coverage and fit with the model affect the result too.

A comparison of unit sizes in tokenization

Criterion	Character or byte	Subword	Whole word
Vocabulary size	For characters, depends on the chosen alphabet; a full byte alphabet is 256 values.	Depends on the target set in training and on the implementation.	Depends on the number of word forms chosen; the vocabulary in use is finite.
Out-of-vocabulary input	A character alphabet can fall short. A full byte alphabet can represent every sequence of bytes.	Alphabet coverage, the rules for the unknown token and byte fallback, if present, decide.	Words that are not in the fixed vocabulary need an extra method.
Sequence length	Usually longer; the ratio depends on the text and on the tokenizer compared with.	Usually between the character and word levels.	Known words can be represented with a single unit.
What the units correspond to grammatically	Represents the spelling with small units.	Pieces sometimes coincide with morphemes; that is not guaranteed.	Keeps the word form; does not fix the word’s meaning independently of context.
Rich morphology	Can represent the many word forms, as long as the alphabet covers them.	Can reuse shared pieces across different inflected forms.	The many inflected forms can make vocabulary coverage hard.
Tokenizer training	A fixed alphabet needs no merges to be learned.	Pieces or piece probabilities are usually learned from a corpus.	A word vocabulary is chosen or extracted from a corpus; no subword merging is learned.

Subwords can produce shorter sequences than the character level while reusing shared pieces across different words. That balance is why many language models prefer them. Character-level, byte-level and word-level approaches can be used too (Xue et al. 2022); the right choice depends on the task, the data, the model architecture and the compute budget.

NoteDeep dive: three ways to learn the same kind of unit — BPE, WordPiece, Unigram LM

You can skip this box on a first reading; the rest of the chapter does not depend on it.

This box compares three methods used in subword tokenization: BPE, WordPiece, which BERT uses, and Unigram LM, which T5 uses. What the methods are and which model ships which is told where we look at BERT’s tokenizer (Section 8.4.5). This box runs the experiment behind that account, for readers who want it: all three side by side on the corpus above. Their core ideas fit on one line each:

The three methods in one sentence each

Method	Core idea
BPE	merge the most frequent adjacent pair
WordPiece	score how much a merge helps, relative to its parts (the ratio below is a teaching approximation, not the exact recipe of BERT’s original trainer)
Unigram LM	start from a large set of pieces and prune it probabilistically

These methods differ both in how the vocabulary is built and in which pieces new text is split into. SentencePiece (Kudo and Richardson 2018) is not a fourth algorithm but a toolkit that can run BPE or Unigram LM.

One corpus, three methods. Let us compare the same two test words, aiming at 60 vocabulary entries in each implementation. The starting representations are not the same, though: BPE uses an end-of-word marker, WordPiece keeps word-initial and continuation pieces apart, and the small Unigram implementation uses neither. The experiment shows how the methods behave; it is not a comparison of performance that controls for every difference between them. The code is a small teaching version of WordPiece and of Unigram LM.

NoteDeep dive: minimal WordPiece and Unigram LM

The score below,

\[ \mathrm{score}(a,b)=\frac{\mathrm{freq}(ab)}{\mathrm{freq}(a)\,\mathrm{freq}(b)} \]

is a teaching approximation used to explain WordPiece training. It should not be taken as an exact reproduction of BERT’s original WordPiece trainer. The results in this box belong to this scoring rule and to the implementation given.

train_unigram is likewise a small teaching implementation inspired by the Unigram approach. Instead of taking every possible split of each word into account, it uses only the highest-scoring one. When it computes the cost of removing a piece it holds the other piece scores fixed; after pruning it builds new counts from the chosen splits. The starting count of 0.001 given to every piece left in the vocabulary prevents zero probabilities. So the loss below is not the same objective as the marginal likelihood of the full Unigram model (Kudo 2018).

import math

def train_wordpiece(corpus, size):
    words = {tuple([w[0]] + ["##" + c for c in w[1:]]): n for w, n in corpus.items()}
    vocab = {s for symbols in words for s in symbols}
    first = None
    while len(vocab) < size:
        pairs, single = Counter(), Counter()
        for symbols, n in words.items():
            for s in symbols:
                single[s] += n
            for a, b in zip(symbols, symbols[1:]):
                pairs[(a, b)] += n
        score = {p: c / (single[p[0]] * single[p[1]]) for p, c in pairs.items()}
        best = max(score, key=lambda p: (score[p], p))
        first = first or (pairs, score)
        glued = best[0] + best[1].removeprefix("##")
        new = {}
        for symbols, n in words.items():
            out, i = [], 0
            while i < len(symbols):
                if symbols[i:i + 2] == best:
                    out.append(glued)
                    i += 2
                else:
                    out.append(symbols[i])
                    i += 1
            new[tuple(out)] = n
        words, vocab = new, vocab | {glued}
    return vocab, first

def wordpiece_split(word, vocab):          # greedy: longest known piece first
    pieces, start = [], 0
    while start < len(word):
        for end in range(len(word), start, -1):
            piece = word[start:end] if start == 0 else "##" + word[start:end]
            if piece in vocab:
                break
        else:
            return ["[UNK]"]
        pieces.append(piece)
        start = end
    return pieces

def best_split(word, logp):                # Viterbi: the most probable split
    best = [(0.0, [])] + [(-math.inf, None)] * len(word)
    for end in range(1, len(word) + 1):
        for start in range(end):
            piece = word[start:end]
            if piece in logp and best[start][1] is not None:
                score = best[start][0] + logp[piece]
                if score > best[end][0]:
                    best[end] = (score, best[start][1] + [piece])
    return best[-1]

def train_unigram(corpus, size):
    counts = Counter()                     # start big: every substring
    for w, n in corpus.items():
        for i in range(len(w)):
            for j in range(i + 1, len(w) + 1):
                counts[w[i:j]] += n
    start_size = len(counts)
    logprobs = lambda c: {p: math.log(k / sum(c.values())) for p, k in c.items()}
    loss = lambda lp: -sum(n * best_split(w, lp)[0] for w, n in corpus.items())
    while len(counts) > size:
        lp = logprobs(counts)
        base = loss(lp)
        cost = {p: loss({q: v for q, v in lp.items() if q != p}) - base
                for p in counts if len(p) > 1}     # single letters always stay
        k = max(1, min(len(counts) - size, len(cost) // 5))
        for p in sorted(cost, key=lambda p: (cost[p], p))[:k]:
            del counts[p]                  # prune the pieces that cost least
        lp = logprobs(counts)              # re-estimate from the best splits
        counts = Counter({p: 0.001 for p in counts})
        for w, n in corpus.items():
            for piece in best_split(w, lp)[1]:
                counts[piece] += n
    return logprobs(counts), start_size

wp_vocab, (pairs, score) = train_wordpiece(corpus, 60)
uni_logp, start_size = train_unigram(corpus, 60)

print("first step, WordPiece initial pairs:")
print("  highest raw pair counts:  ",
      [(a + b.removeprefix("##"), c) for (a, b), c in pairs.most_common(3)])
print("  highest score (WordPiece):",
      [(a + b.removeprefix("##"), round(s, 3))
       for (a, b), s in sorted(score.items(), key=lambda x: -x[1])[:3]])
print(f"  Unigram LM starts from {start_size} substrings and prunes to {len(uni_logp)}")
print()
for w in test:
    print(f"{w:<11} BPE       : {' | '.join(apply_merges(w, merges))}")
    print(f"{'':<11} WordPiece : {' | '.join(wordpiece_split(w, wp_vocab))}")
    print(f"{'':<11} Unigram LM: {' | '.join(best_split(w, uni_logp)[1])}")

first step, WordPiece initial pairs:
  highest raw pair counts:   [('##de', 72), ('##da', 72), ('##rd', 48)]
  highest score (WordPiece): [('ok', 0.056), ('##ku', 0.056), ('ev', 0.028)]
  Unigram LM starts from 322 substrings and prunes to 60

evlerden    BPE       : ev | ler | den</w>
            WordPiece : ev | ##l | ##e | ##r | ##d | ##e | ##n
            Unigram LM: evlerde | n
okullardan  BPE       : okul | lar | dan</w>
            WordPiece : okullar | ##d | ##a | ##n
            Unigram LM: okullarda | n

The first two lines of output rank the pairs in WordPiece’s starting representation by two criteria: raw frequency and the ratio score. That shows how the criteria differ. BPE’s example has a different starting alphabet and an end-of-word marker, so the first line should not be read as its merge log.

BPE merges the most frequent adjacent pair of symbols at each step. In this corpus the repeated stem and suffix sequences lead to pieces that can be reused across words. In the result shown both test words are cut at stem and suffix boundaries: ev|ler|den and okul|lar|dan. That match is what was observed for this corpus and this vocabulary budget; it is not guaranteed for every new word.

WordPiece. This teaching implementation divides the frequency of a pair by the product of the separate frequencies of its two parts, \(\frac{c(ab)}{c(a)\,c(b)}\). At the start the pairs o + ##k and ##k + ##u get the same high score (0.056). The code picks o + ##k by the ordering of symbol pairs when scores are equal. In this small corpus those pieces occur together in okul and its inflected forms. The ## marks a piece that continues a word.

With the given vocabulary budget, the result contains long word-initial pieces such as okullar; the rest of okullardan is split into single letters in the output shown. That behaviour cannot be generalised to every WordPiece tokenizer. To process a new word, WordPiece does not replay a merge list; it moves along by taking the longest vocabulary piece that starts at the current position.

Unigram LM. The small Unigram implementation collects every contiguous substring of the words as a candidate; the printed output starts with 322 candidates. It then prunes the pieces that contribute least by its own approximate loss, down to 60. Real Unigram implementations can use different methods for building candidates, estimating probabilities and pruning.

The scores learned here put the split evlerde|n first for evlerden. That is the result of the small corpus and the approximate training procedure; it does not mean the best morphological analysis, or the best solution among all possible vocabularies. Unigram also scores the alternative splits of the same word. Here is every split of evlerden that this vocabulary allows:

def every_split(word, logp, prefix=()):
    if not word:
        yield sum(logp[p] for p in prefix), list(prefix)
    for end in range(1, len(word) + 1):
        if word[:end] in logp:
            yield from every_split(word[end:], logp, prefix + (word[:end],))

splits = sorted(every_split("evlerden", uni_logp), reverse=True)
print(f"{len(splits)} ways to split 'evlerden' with this vocabulary:")
for logp, pieces in splits:
    print(f"  score = {math.exp(logp):.1e}   {' | '.join(pieces)}")

4 ways to split 'evlerden' with this vocabulary:
  score = 2.1e-03   evlerde | n
  score = 1.2e-14   evler | d | e | n
  score = 1.3e-22   ev | l | e | r | d | e | n
  score = 1.1e-31   e | v | l | e | r | d | e | n

The value printed here is the product of the probabilities of the pieces in a split. These values are enough to rank the splits of one word. To turn the alternatives into a probability distribution, each value is divided by the sum of the values computed for all valid splits of the word.

Using fewer pieces does not by itself guarantee a higher score; the probabilities of the pieces compared change too. In this output the two-piece solution is about 11 orders of magnitude ahead of its nearest rival and about 28 ahead of the eight single characters (from the rounded output). That gap belongs to this small corpus and this teaching implementation’s piece probabilities; it should not be read as how real Unigram trainers behave in general.

Being able to score alternative splits is what makes it possible to sample among them probabilistically. The code above only lists the alternatives; it does not sample a split yet.

NoteDeep dive: checking the minimal Unigram LM against SentencePiece

The small version above is not the library. The code below trains a Unigram tokenizer with sentencepiece 0.2.2 on the same word forms and frequencies. The target vocabulary size is 60; with hard_vocab_limit=False, though, the actual vocabulary can be smaller. That budget includes the unknown token: one of the 60 entries of the model trained in this run is <unk>. SentencePiece’s representation of spaces and its other defaults differ from the small implementation too, so the comparison is not made under identical conditions.

import sentencepiece as spm

lines = [w for w, n in corpus.items() for _ in range(n)]
with open("corpus.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(lines) + "\n")

spm.SentencePieceTrainer.train(
    input="corpus.txt", model_prefix="uni", model_type="unigram", vocab_size=60,
    character_coverage=1.0, hard_vocab_limit=False, minloglevel=2,
    bos_id=-1, eos_id=-1, pad_id=-1)
uni = spm.SentencePieceProcessor(model_file="uni.model")
for w in test:
    print(f"{w:<11} {' | '.join(uni.encode(w, out_type=str))}")

evlerden    ▁evlerde | n
okullardan  ▁okul | l | a | r | d | a | n

The split shown for evlerden is again evlerde|n. The leading ▁ stands for the space SentencePiece adds at the start by default. For okullardan, ▁okul|l|a|r|d|a|n was chosen. On its own this output does not show that the piece okullarda is missing from the vocabulary; it shows only that the chosen split does not use it. In Unigram a piece can be in the vocabulary and still not be chosen, because another split scores higher. Looking at the vocabulary itself, in this run it contains ▁okul and ▁evlerde, and does not contain ▁okullarda.

SentencePiece’s Unigram training uses a different probability estimation and pruning process from the small implementation. Normalization, the starting candidates and the training settings can affect the result as well; other versions or settings can give different splits.

This artificial corpus of sixty different word forms was chosen to make the effect of the training rules visible. The WordPiece and Unigram implementations here contain teaching simplifications; their results should not be read as a general ranking of real tokenizers. The study cited at the end of this box compares BPE with Unigram LM on larger corpora; nothing about WordPiece can be concluded from that experiment.

Two distinctions matter in this example. First, the teaching WordPiece score uses not only the frequency of the pair but also the frequencies of its components. With the pair count held the same, more frequent components lower the score. That rule does not, however, guarantee in general that stems are learned before suffixes.

Second, Unigram can evaluate splits with probabilistic scores. In use, the highest-scoring split can be taken, or one can be sampled from the alternatives. The aim of subword regularization is to reduce a model’s dependence on a single split by letting it see the same text split in different ways (Kudo 2018). Sampling can happen whenever inputs are processed again during training; it is not limited to one choice per epoch. Its benefit has to be measured by experiment. Different splits can be produced in a similar way with BPE-Dropout (Provilkov et al. 2020).

SentencePiece (Kudo and Richardson 2018) is a library for training and applying methods such as BPE and Unigram on raw text. ▁ is the visible form of a space; depending on the settings, the original spaces can be normalized, or a space added at the start of the text.

Bostrom and Durrett (2020) compared Unigram LM with BPE at equal vocabulary size in English and Japanese experiments. The boundaries of Unigram LM coincided better with the morphological references used. Models pretrained with that tokenization also did as well as or better than their BPE counterparts on the downstream tasks evaluated. The findings show Unigram’s advantage under those experimental conditions; they do not establish a fixed ranking for all languages and models.

8.4 The tokenizer is a budget decision

One pretraining decision is usually made before the first gradient step and, for as long as the model is in use, sets how many tokens the same content costs and which embedding table the model learns. It is the tokenizer. Changing the tokenizer of a trained model is possible, but it takes extra adaptation and re-evaluation.

The previous section introduced subword tokenization as a way to avoid an unbounded vocabulary. That framing is correct and incomplete. At pretraining scale the tokenizer is not a preprocessing convenience — it is how the compute budget gets divided between languages, and the division is permanent for practical purposes. An ordinary fine-tune leaves the segmentation exactly where it was; replacing it is a separate project, with its own cost, described in Section 8.9.

Llama 3’s tokenizer is a good example of the decision being made explicitly. Its 128,000-token vocabulary is 100,000 tokens taken from OpenAI’s tiktoken plus 28,000 added specifically to support non-English languages (Grattafiori et al. 2024). The report notes that these 28K improved compression and downstream performance for other languages “with no impact on English tokenization”. That is a design choice, with a price tag, made once.

Fertility, and why the obvious metric misleads

The standard measure is fertility: tokens per word. Comparing two tokenizers on one language, the lower one is usually preferred; it does not by itself settle cost across languages.

fertility — how many tokens a tokenizer produces per word of input. Meaningful when comparing two tokenizers on one language; misleading when comparing two languages, as the next paragraph shows. ↩︎

Comparing Turkish with English, fertility misleads, and it can lead you to a conclusion that is exactly backwards. Turkish is agglutinative: it stacks suffixes onto a stem, so one Turkish word often carries what English needs several words to say. Evlerinizden is one word and means “from your houses”. Comparing tokens-per-word across the two languages compares units of different size.

The better measure across languages is tokens for the same content: take a sentence and its professional translation, tokenize both, and divide. That ratio gives the approximate token price of the same content — in context window, in latency, and on the API invoice. For your own product’s cost, measure the input and output distribution of real requests with the model’s own tokenizer as well; matched translation pairs do not represent every request.

The measurement

We ran both metrics over a real parallel corpus rather than a hand-picked paragraph, because a single passage can be a fluke. The source is the en-tr portion of OPUS-100, from which 3,000 sentence pairs were downloaded and filtered down to 1,157 usable ones: both sides at least four words, and a character-length ratio between 0.4 and 2.5, which drops some of the misalignments common in machine-built parallel corpora; it does not guarantee translation accuracy, and it changes the sample (it leaves out short sentences, for instance). That leaves 8,058 whitespace-separated Turkish units against 10,862 English ones of matched content. Note that disparity: the raw counts show the trace of agglutination, though attributing all of the gap to it would take further analysis.

Every number below comes from running the tokenizer that actually ships with the model — openai-community/gpt2, google-bert/bert-base-multilingual-cased, NousResearch/Meta-Llama-3-8B, Qwen/Qwen2.5-7B, dbmdz/bert-base-turkish-cased, and ytu-ce-cosmos/turkish-gpt2.

tokenizer	vocab	TR/word	EN/word	same content	median
GPT-2 (2019)	50,257	3.42	1.28	1.98×	1.92
BERT multilingual (2018)	119,547	2.38	1.43	1.23×	1.21
Llama 3 (2024)	128,000	2.12	1.28	1.23×	1.22
Qwen 2.5 (2024)	151,643	2.43	1.29	1.40×	1.39
BERTurk (Turkish-native)	32,000	1.59	2.03	0.58×	0.58
Cosmos GPT-2 (Turkish-native)	50,257	1.56	1.79	0.64×	0.64

The “same content” column is the aggregate ratio over the whole corpus (the sentences joined with spaces and tokenized in one go; the per-word columns come from the same count); the “median” is the median of the ratios over the first 800 pairs, each sentence tokenized on its own, computed separately so that one long outlier cannot drive the headline. We rewrote and ran the chapter’s code (re-run, October 2026): the same 1,157 pairs are selected and every value in the table comes out exactly. Taking the median over all 1,157 pairs moves the values by at most 0.01 (GPT-2 1.93; BERT multilingual 1.22; Llama 3 1.22; Qwen 2.5 1.38; BERTurk 0.58; Cosmos 0.64). For five of the six tokenizers the two summaries agree to within 0.02. That is a useful observation that the aggregate does not rest on one long sentence; it does not show that the sample represents all Turkish text. The vocabulary column is each tokenizer’s own reported size (vocab_size); Llama 3 adds 256 reserved special tokens on top, which is why its embedding table has 128,256 rows, the figure the \(6ND\) margin note uses below. GPT-2 is the exception, at 1.98 against 1.92. Resist reading a length effect into that gap. The aggregate is a length-weighted mean of the per-sentence ratios (each sentence weighted by its English token count), and a weighted mean can sit above a median without any relationship to length at all: three equal-length sentences with ratios 1, 1 and 2 give an aggregate of 1.33 against a median of 1. Nor does the gap come from the sample: over all 1,157 pairs the median is 1.93 (re-run). Testing whether GPT-2’s Turkish penalty grows with sentence length means plotting the per-sentence ratio against length on one common set of pairs, which this table does not do.

viewof fertMetric = Inputs.radio(
  [
    "Tokens for the same content (Turkish ÷ English)",
    "Tokens per word (the misleading view)"
  ],
  { value: "Tokens for the same content (Turkish ÷ English)", label: "" }
)

fertRows = [
  { name: "GPT-2 (2019)",              vocab:  50257, tr: 3.42, en: 1.28, same: 1.98 },
  { name: "Qwen 2.5 (2024)",           vocab: 151643, tr: 2.43, en: 1.29, same: 1.40 },
  { name: "BERT multiling. (2018)",    vocab: 119547, tr: 2.38, en: 1.43, same: 1.23 },
  { name: "Llama 3 (2024)",            vocab: 128000, tr: 2.12, en: 1.28, same: 1.23 },
  { name: "Cosmos GPT-2 (TR-native)",  vocab:  50257, tr: 1.56, en: 1.79, same: 0.64 },
  { name: "BERTurk (TR-native)",       vocab:  32000, tr: 1.59, en: 2.03, same: 0.58 }
]

fertOrder = fertRows.map(d => d.name)

fertPlot = {
  const sameMode = fertMetric.startsWith("Tokens for the same");

  if (sameMode) {
    return Plot.plot({
      width: 700,
      height: 320,
      marginLeft: 190,
      marginRight: 40,
      x: { label: "Turkish tokens per English token, same content →", domain: [0, 2.2], grid: true },
      y: { domain: fertOrder, label: null },
      marks: [
        Plot.barX(fertRows, {
          y: "name",
          x: "same",
          fill: d => d.same > 1 ? "#C1666B" : "#0E7C7B",
          rx: 3,
          title: d => `${d.name}: ${d.same.toFixed(2)}x`
        }),
        Plot.text(fertRows, {
          y: "name",
          x: "same",
          text: d => d.same.toFixed(2) + "x",
          dx: 16,
          fontWeight: "bold"
        }),
        Plot.ruleX([1], { stroke: "currentColor", strokeDasharray: "4 3" }),
        Plot.ruleX([0])
      ]
    });
  }

  const long = fertRows.flatMap(d => [
    { name: d.name, lang: "Turkish", v: d.tr },
    { name: d.name, lang: "English", v: d.en }
  ]);

  return Plot.plot({
    width: 700,
    height: 380,
    marginLeft: 190,
    marginRight: 40,
    x: { label: "tokens per word →", domain: [0, 3.7], grid: true },
    fy: { domain: fertOrder, label: null },
    y: { domain: ["Turkish", "English"], axis: null },
    color: {
      domain: ["Turkish", "English"],
      range: ["#C1666B", "#1F4E79"],
      legend: true
    },
    marks: [
      Plot.barX(long, {
        fy: "name",
        y: "lang",
        x: "v",
        fill: "lang",
        rx: 2,
        title: d => `${d.name} — ${d.lang}: ${d.v.toFixed(2)} tokens/word`
      }),
      Plot.ruleX([0])
    ]
  });
}

Switch between the two views and watch the two Turkish-native tokenizers. Under tokens-per-word they look like the clear winners on Turkish and it is not obvious what they lose. Under the same-content ratio the trade appears: they are below 1.0 on Turkish because they produce more tokens on the English side — 1.8 to 2.0 tokens per word against Llama 3’s 1.28. That is a tokenization trade-off; model quality and total product cost have to be measured separately.

Three things the table says

1. The gap narrowed a great deal, and it was a design outcome. GPT-2 charged Turkish a 98% surcharge — the same content cost almost exactly twice as many tokens. Llama 3 charges 23%. Turkish did not change between 2019 and 2024; the tokenizers did. The Llama 3 report says the 28K tokens added for non-English languages improved compression in some languages. But we cannot put the whole gap down to the language mix of the training data: the vocabulary grew from 50K to 128K, and the pre-tokenization rules and other design details changed too.

2. Vocabulary size alone is not the lever. This is the finding that surprises people, and it falls straight out of the table. Qwen 2.5 has a larger vocabulary than Llama 3 — 151,643 against 128,000 — and is worse on Turkish: 1.40× against 1.23×. A bigger vocabulary spent on other languages may do little for yours. The number of slots does not explain it alone; which texts and which preprocessing rules the vocabulary was learned with matter too.

3. A 32K Turkish-native vocabulary writes the same Turkish text in fewer tokens than a 128K multilingual one. On the same Turkish text BERTurk produces 1.59 tokens per word and Llama 3 2.12: about 25% fewer tokens with a vocabulary a quarter the size. Its ratio to English is 0.58× (Llama 3: 1.23×). None of this compares the models’ task performance. BERTurk also needs 2.03 tokens per English word, which is roughly GPT-2’s Turkish performance in reverse. There is no free lunch here and no “better tokenizer” in the abstract: a vocabulary is a fixed budget, and every slot given to Turkish morphology is a slot not given to English, code, or Chinese.

Look at what actually happens to a word

Ratios hide the mechanism. Here is one long but perfectly ordinary Turkish word — bilgisayarlaştırabildiklerimizden, “from the ones we were able to computerise” — cut three ways.

One Turkish word, segmented three ways

Segmentations produced by running each tokenizer, not by hand. Llama 3’s nine pieces cost 2.25 times what the Turkish-native tokenizer’s four do — every time this word is processed. Its effect on total request cost and on model performance is measured on task examples, not on one word.

Look at where Llama 3 cuts. It splits bilgisayar — an ordinary everyday noun, “computer” — into bil | gis | ayar. The word’s own parts are bilgi (“information”) and sayar (“counter”), and the cuts ignore that boundary: gis is not a Turkish unit at all, and ayar (“setting, adjustment”) is an unrelated real word. How that split affects how well the model handles the word has to be measured on task examples, not on one word.

Fewer tokens is not the same as better linguistics, and the difference matters. On evlerinizden (“from your houses”) Llama 3 produces ev | ler | iniz | den; those boundaries coincide with the morpheme analysis we chose here, ev|ler|iniz|den. The Turkish-native Cosmos tokenizer produces ev | lerinizden: two tokens, shorter, but it does not show the internal boundaries, since lerinizden spans three morphemes rather than being one (the splits were checked in a re-run).

So the two metrics genuinely can disagree. Compression is what you pay for and is easy to measure. That morphological alignment helps the model generalise to a suffix combination it has not seen is a plausible hypothesis; these two outputs do not prove it, and its contribution to task performance is hard to measure. Do not assume the cheaper segmentation is the more linguistically sensible one — check, as we just did.

BERT’s tokenizer: WordPiece and its relatives

The measurement table has six tokenizers in it; at the top level they fall into two algorithm families. Within a family the starting units, the pre-tokenization and the vocabulary can differ, so the same word need not be cut into the same pieces. GPT-2, Llama 3, Qwen 2.5 and Cosmos GPT-2 use BPE, the algorithm this chapter built step by step (Section 8.3.1). The two BERT models use WordPiece. A third algorithm that is not in the table, Unigram LM, is the one you will meet in models such as T5 and XLM-R.

All three do the same job: they learn from a corpus which subword pieces go into the vocabulary. They differ in what they count as a “good” piece. Knowing which one a model uses is worth it, because that is how you read its output and predict how it will cut a word.

Look at the output first. Run the same word, evlerinizden (“from your houses”), through the two BERT tokenizers and through XLM-R, which uses Unigram LM:

from transformers import AutoTokenizer

word = "evlerinizden"
for name in ["google-bert/bert-base-multilingual-cased",
             "dbmdz/bert-base-turkish-cased",
             "FacebookAI/xlm-roberta-base"]:
    tok = AutoTokenizer.from_pretrained(name)
    pieces = tok.convert_ids_to_tokens(tok.encode(word))
    print(f"{name.split('/')[-1]:<30} {' '.join(pieces)}")

bert-base-multilingual-cased   [CLS] ev ##lerini ##zde ##n [SEP]
bert-base-turkish-cased        [CLS] evlerini ##z ##den [SEP]
xlm-roberta-base               <s> ▁ev leriniz den </s>

BERT’s output has two things in it that the GPT family’s does not.

The special tokens at the start and the end. BERT puts [CLS] in front of every input and [SEP] after it. The contextual representation at the [CLS] position is used in tasks such as sentence classification (Chapter 12). [SEP] marks where a text ends, and when the input has two texts, a question and a passage for instance, one goes between them too. XLM-R uses <s> and </s> for similar roles; such markers are not interchangeable between models.

The ## mark. WordPiece puts ## in front of every piece that continues a word. A piece without ## starts a new pre-token. The space GPT keeps inside a token and the ▁ in XLM-R’s output also show how pieces are to be joined, but the three notations do not carry the same information (Section 8.3.3): ## does not record how many spaces there were. One consequence is that ev and ##ev are two separate entries in the vocabulary; the model learns one vector for ev at the start of a word and another for ev inside one. In multilingual BERT’s vocabulary of 119,547 entries, 43,606 start with ##; in BERTurk’s 32,000, 7,246 do.

How WordPiece splits a word. To split a new word, BPE replays the merges it learned, in order. WordPiece does not. Its rule is simpler, and it is greedy:

• Start at the beginning of the word. Take the longest piece that is in the vocabulary.
• Do the same for what is left, this time looking for pieces that start with ##.
• Continue until the word is used up.

Follow BERTurk through evlerinizden. The vocabulary has ev, evler, evleri and evlerini; it does not have evleriniz. So the longest match is evlerini. That leaves zden: ##zden and ##zde are not in the vocabulary, ##z is. That leaves den: ##den is there. Result: evlerini ##z ##den.

The greedy rule has a cost, and it shows here. The same vocabulary also has the piece ##lerinizden, so a two-piece split, ev ##lerinizden, was possible. Because the rule takes the longest piece at the first step, it never tries that split and produces three pieces.

WordPiece can emit [UNK]. If at some point no piece in the vocabulary matches, WordPiece turns not just that character but the whole word into [UNK]. Multilingual BERT’s vocabulary has no 😀:

çay 😀    ça ##y [UNK]
çay😀     [UNK]

In the second line the emoji was typed directly after the word, and çay (tea) was lost with it. A byte-level BPE tokenizer cannot do this (Section 8.3.2): cl100k_base spells the same emoji out in bytes, as two tokens. The behaviour depends on this tokenizer’s pre-tokenization and WordPiece settings; it is not required of every WordPiece model. So when you work with such a model, counting the [UNK]s in your input is a cheap and useful check (in a re-run, Çay😀 içtik also becomes [UNK] iç ##tik).

How the three algorithms learn the vocabulary. So far you have seen how a finished vocabulary is used. How it is learned is where the three algorithms part:

Three subword algorithms, and what each one is actually maximising

	What it optimises	Direction	Ships in
BPE (Sennrich et al. 2016)	merge the most frequent adjacent pair	bottom-up, the vocabulary grows	GPT-2/3/4, RoBERTa, Llama, Gemma
WordPiece (Schuster and Nakajima 2012)	in the usual account, merge the pair that most increases corpus likelihood, roughly maximising \(\frac{c(ab)}{c(a)\,c(b)}\) (BERT’s original procedure was not fully published)	bottom-up	BERT, DistilBERT, ELECTRA
Unigram LM (Kudo 2018)	start from a large candidate set and prune the pieces whose removal costs least likelihood, re-estimating piece probabilities by EM (expectation–maximization)	top-down, the vocabulary shrinks	T5, ALBERT, XLNet, XLM-R

Read the table row by row, with the intuition for each.

BPE merges the pair that occurs side by side most often. Because it rewards frequency, its vocabulary fills with pieces that many words share: ler, den, lar.

WordPiece does not stop at how often the pair occurs; it divides that count by how often each of the two parts occurs on its own. That penalises a merge of two parts that are already very common separately, and favours parts that are seen almost only together. One caveat about the ratio in the table: BERT’s original training procedure is not published in full. The ratio is the common approximate reconstruction used to explain its logic.

Unigram LM comes from the other direction. It starts with a very large set of candidate pieces and, in each round, drops the ones whose removal lowers the likelihood of the corpus least. It can evaluate the alternative splits of a word with probabilistic scores. In use, the highest-scoring split can be taken, or one can be sampled from the alternatives. Showing the same text split in different ways during training is called subword regularization; the aim is to reduce the model’s dependence on a single split (Kudo 2018). Its benefit has to be measured by experiment. Different splits can be produced in a similar way with BPE-Dropout (Provilkov et al. 2020).

One common confusion is worth clearing up here: SentencePiece (Kudo and Richardson 2018) is not a fourth algorithm but a toolkit that can run BPE or Unigram LM. Llama 2 and Gemma use it in BPE mode, T5 and ALBERT in Unigram mode.

Bostrom and Durrett (2020) compared Unigram LM with BPE at equal vocabulary size in English and Japanese experiments. The boundaries of Unigram LM coincided better with the morphological references used, and models pretrained with that tokenization did as well as or better than their BPE counterparts on the downstream tasks evaluated. The findings hold for those experimental conditions; they do not establish a fixed ranking for all languages and models.

If you want to see the three of them run. The collapsed box “Deep dive: three ways to learn the same kind of unit” at the end of the subword tokenization section (Section 8.3.4) runs a minimal version of each on a sixty-word Turkish corpus and has all three split the same two words.

Now tie this back to the chapter’s question. The algorithm explains a small part of the differences in the measurement table. Multilingual BERT and BERTurk use the same algorithm; one charges Turkish 1.23×, the other 0.58×. What makes the difference is not the algorithm but, largely, the corpus the vocabulary was trained on — even though BERTurk’s vocabulary is smaller.

What the surcharge actually costs

Take the measured corpus figure for Llama 3: Turkish costs 1.23× the tokens of English for identical content. That single number propagates into four separate bills. The rows below assume the same model, the same unit price and text like this corpus; total request cost and latency also depend on output length, caching and serving conditions.

where it lands	the effect at 1.23×
context window	a 128,000-token window holds about 104,000 English-equivalent tokens of Turkish
API cost (input)	one million English-equivalent tokens costs the price of 1.23 million
latency	generation is per token, so at the same tokens-per-second a Turkish answer takes about 23% longer
pretraining compute	at fixed FLOPs, a Turkish corpus buys about 19% less content than an English one

Watch the direction of that last number, because it is the single easiest mistake to make with a ratio. Turkish costs 1.23× more tokens, but a fixed token budget buys \(1/1.23 = 0.813\) of the content — 18.7% less, not 23% less. The two are not the same figure, and only the API-cost and latency rows multiply by 1.23; the context-window and compute rows divide by it. That is why 128,000 tokens holds about 104,000 English-equivalent tokens rather than 98,560.

The compute row is the one that compounds. From Section 7.8, compute scales with the token count, so a fixed budget spent on Turkish sees roughly a fifth less material than the same budget spent on English — about 18.7% less matched content for the same money. This measurement does not show that the number of facts or patterns learned falls by the same share; that effect is measured through the training mix and task performance. The tokenizer tax is not only charged at inference; it applies during pretraining too.

Measure this for your own language and your own text before you commit. It takes about fifteen lines. Take a few hundred sentence pairs of your domain text and its translation, run every tokenizer you are considering over both sides, and divide. Domain matters as much as language: legal Turkish, clinical Turkish, and Turkish social media do not tokenize alike, and a corpus average will not warn you.

Under 1.23× the pragmatic answer is usually “use the multilingual model anyway” — 23% is real but it does not overturn a decision. Under GPT-2’s 1.98× it does. That is precisely why you measure instead of assuming, and the full decision, including when a Turkish-native model wins outright, is Chapter 16’s subject.

8.5 What a run costs

You can approximately estimate the compute of a dense transformer pretraining run — how many FLOPs it consumed — from two numbers, the parameter count \(N\) and the token count \(D\). The estimate counts the weight matrix multiplies, which dominate the cost of a typical run. The main term it leaves out is attention; at the short contexts used for most of pretraining that adds a few to about fifteen percent, and this section comes back to when it matters more. An estimate this good from two numbers is unusual in systems work, and it is the tool that lets you sanity-check a press release.

One caveat first, because it decides which number you put in \(N\). The \(6ND\) rule counts operations per parameter used, and in a mixture-of-experts model (Section 7.7) a token only touches the experts its router picked (Jiang et al. 2024). So the figure you want is active parameters, not total:

\[C \approx 6\,N_{\text{active}}\,D\]

DeepSeek-V3 makes the gap concrete: 671B total parameters, about 37B active per token, 14.8T training tokens. Using the active count gives \(6 \times 37{\times}10^{9} \times 14.8{\times}10^{12} \approx 3.3\times10^{24}\) FLOPs; substituting the total would have over-counted the \(6ND\) term by 18×. That is the ratio between two \(6ND\) terms, not the difference in real training FLOPs or GPU-hours, since attention, the router and communication between experts cost something too. Whenever a release quotes two parameter counts, the smaller one is the one this first approximation wants.

FLOP — one floating-point operation: a single multiply or a single add on numbers stored in floating-point (non-integer) format. Training compute is counted in FLOPs; chips are rated in FLOP/s, operations per second. The two are easy to confuse and differ by a factor of however long you ran. ↩︎

Where the 6 comes from

\[ C \;\approx\; 6\,N\,D \quad \text{FLOPs} \]

The 6 is three twos from the dominant matrix multiplies, and each two has a cause.

Six floating-point operations per parameter per token

Two for the forward pass, four for the backward pass, once per parameter per token. Multiply by \(N\) parameters and \(D\) tokens and you have the run.

The forward two is the easy one. Every weight in a dense linear layer’s matrix multiply gets used exactly once per token: one multiplication and one addition into the accumulator. Two operations, per weight, per token.

The backward four is where people lose the thread, so name the two jobs separately. Backpropagation through a layer must produce two different gradients. It needs \(\partial L/\partial x\) — how the loss changes with the layer’s input — or the error cannot propagate to the layer below. And it needs \(\partial L/\partial W\) — how the loss changes with each weight — or the optimiser has nothing to update. Each of those is another matrix multiply of the same shape, so each costs another 2 per parameter per token. Two jobs, two each, four total. This count is for dense matrix multiplies; it does not cover embedding lookups, the unselected experts of an MoE, or other operations such as attention.

Does it agree with what the labs publish?

run	\(N\)	\(D\)	\(6ND\)	compute reported in the paper	difference between the two estimates
GPT-3	175B	300B	\(3.150\times10^{23}\)	\(3.14\times10^{23}\)	+0.32%
Llama 3 405B	405B	15.6T	\(3.791\times10^{25}\)	\(3.80\times10^{25}\)	−0.24%

Under half a percent, across a factor of 120 in compute and four years of architecture change. It is tempting to call this a validation of the rule. It is not, and the distinction matters more than the agreement does.

Those published figures are themselves \(6ND\). GPT-3’s Appendix D prints the arithmetic in columns — “Fwd-pass flops per active param per token: 2”, “Mult for bwd pass: 3”, “Flops per param per token: 6” — and states the assumption openly: “we ignore the attention operation, as it typically uses less than 10% of the total compute for the models we are analyzing” (Brown et al. 2020). Llama 3’s \(3.8\times10^{25}\) is likewise a budget: the number Meta fixed in advance and then chose a model size to fit (Grattafiori et al. 2024).

So the table shows that our bookkeeping matches theirs, which is worth confirming and is all it is. The reason to trust \(6ND\) is the derivation in the figure above, not this table. What the table does buy you is practical: since the labs count this way, you can read any announcement that gives parameters and tokens and reconstruct the number they would quote — and, as the next section shows, know exactly what that number leaves out.

Where it stops being true

\(6ND\) counts the matrix multiplies whose cost is proportional to parameters. Attention’s score matrix is not one of those: it depends on the sequence length, which appears nowhere in \(N\) or \(D\).

Section 7.6 gives attention’s forward cost for a whole sequence as \(4n^2d\) per layer. Three steps turn that into a per-token figure: divide by \(n\) to get \(4nd\) per token, halve it because causal masking means the average position attends to only half the sequence (in an implementation that really skips the masked pairs; kernels that apply the mask but still compute the full square do not get this halving), then multiply by 3 because the backward pass costs twice the forward. That leaves about \(6\,L\,n\,d\) per token for \(L\) layers, sequence length \(n\), and width \(d\).

How much does ignoring it matter? Exactly as much as your context is long.

model and context	\(6N\) per token	approximate attention per token	attention as share of \(6N\)	\((6N + \text{attention}) \div 6N\)
Llama 3 8B at 8K context	\(4.82\times10^{10}\)	\(6.44\times10^{9}\)	13%	1.13×
Llama 3 8B at 128K context	\(4.82\times10^{10}\)	\(1.03\times10^{11}\)	214%	3.14×
Llama 3 405B at 8K context	\(2.43\times10^{12}\)	\(1.02\times10^{11}\)	4%	1.04×

Read the last column as the approximate multiplier you get by adding this causal attention count to the \(6ND\) estimate; it is not measured GPU time or the true total operation count.

There is a smaller error pushing the other way. The input embedding is a lookup, not a matrix multiply, so it costs essentially no FLOPs — yet its parameters are counted in \(N\). Llama 3 8B’s untied embedding is \(128{,}256 \times 4096 = 525\)M parameters, 6.5% of its \(N\); in the wider models the embedding’s share shrinks to only 1.5% of the 70B’s and 0.5% of the 405B’s. The untied output layer, by contrast, is a real matrix multiply, and its parameters are counted correctly in \(6ND\). So on the input embedding, \(6ND\) over-counts small models slightly at the same time as it under-counts attention. For the 8B at 8K the two partly cancel.

Two lessons, both practical. First, \(6ND\) is a rule of thumb for short-context dense pretraining: it is accurate because pretraining mostly runs at short context, and the 405B is the most accurate case of all because a wide model buries the attention term. Second, at 128K context the 8B’s attention term, under this counting assumption, pushes the total past three times the \(6ND\) estimate (the multiplier is not the same for every model and implementation) — which is precisely why long-context training is done as a short separate stage on a fraction of the tokens rather than for the whole run.

One run, end to end

Take Llama 3.1 405B and follow it from parameters to an invoice. Every input is published: the paper gives \(N\) and \(D\), Meta’s model card gives the GPU-hours (Grattafiori et al. 2024).

quantity	value	where it comes from
parameters	405B	the paper
training tokens	15.6T	the paper
compute	\(3.79\times10^{25}\) FLOPs	\(6ND\)
H100-hours	30.84M	Meta’s model card
on 8,000 H100s	160.6 days	30.84M ÷ 8,000 ÷ 24; equal-allocation assumption
on 16,000 H100s	80.3 days	the largest fleet in the paper; same assumption, not the real run’s calendar
on 24,000 H100s	53.5 days	same assumption
rented at $2.00/GPU-hour	≈ $62M	hypothetical rental price; varies with date and provider
rented at $3.00/GPU-hour	≈ $93M	same sum at another unit price; not a real bill

Meta’s card puts the whole Llama 3.1 family at 39.3M H100-hours: 1.46M for the 8B, 7.0M for the 70B, and 30.84M for the 405B. The 405B alone is 78% of the family’s compute. The same card also gives these hours as carbon: on a location-based count, training the 405B produced 8,930 and the whole family 11,390 tons CO2eq; on a market-based count, 0 tons, because Meta says it matched its electricity use with renewable energy. These are emissions accounting under the card’s electricity and facility assumptions, not a life-cycle or hardware footprint. Energy and carbon are a decision axis separate from dollars; Section 8.6.2 comes back to how to read the figure.

Renting 16,000 H100s by the hour may not be possible at all. The dollar figures in the table are only a hypothetical external rental price; Meta’s real spending or opportunity cost cannot be read from this product. For your own project, recompute with a provider’s quote, capacity and service terms; renting is only one option beside buying and partnering.

The brochure and the datacentre

Now the number that separates people who have run a cluster from people who have read about one.

An H100 SXM is rated at 989 TFLOP/s for dense BF16. (You will also see 1,979, which is the same chip with 2:4 structured sparsity — a different operation that pretraining does not use. The PCIe card is about a quarter slower again, at 756 TFLOP/s dense. Check which number you are being quoted before comparing anything.) Divide the FLOPs a run needed by the FLOPs its GPUs could have produced in the hours it was allocated, and you get model FLOPs utilization: the fraction of the theoretical peak that actually became training.

model FLOPs utilization — the useful FLOPs counted for the model’s forward and backward passes per unit time, divided by the hardware’s rated peak FLOP/s; usually written MFU. Extra work such as recomputed activations does not enter the numerator, so it is not the same as total hardware utilization (HFU). When comparing, state the precision, the sparsity and how time is defined. Above 50% has so far been rare in large runs; the gap is communication, memory bandwidth, and downtime. ↩︎

The Llama 3 paper reports instantaneous BF16 MFU of 43% on 8,000 GPUs, 41% on 16,000, and 38% during the 128K-context stage (Grattafiori et al. 2024). Here is what falls out of the published GPU-hours:

model	GPU-hours	\(6ND\)	effective MFU
Llama 3.1 8B	1.46M	\(7.23\times10^{23}\)	13.9%
Llama 3.1 70B	7.00M	\(6.35\times10^{24}\)	25.5%
Llama 3.1 405B	30.84M	\(3.79\times10^{25}\)	34.5%

The 8B and 70B token counts come from the card’s prose, “~15 trillion tokens” — its spec table says “15T+” for the same runs. Only the 405B’s 15.6T is stated precisely, and only in the paper. A few percent of slack in \(D\) does not change the picture.

Look at the 405B row against the paper’s own 41%. The run delivered 34.5%. Where did the missing 6.5 points go? Most likely two places, and it is worth separating them, because one is a fact about datacentres and the other is a fact about our own arithmetic. But the split below is a reading, not an exact attribution.

Most of it is probably downtime. The paper says so in a sentence that is easy to skim past. During a 54-day snapshot of pretraining there were 466 job interruptions — 47 planned, 419 unexpected — and the run sustained “>90% effective training time” (Grattafiori et al. 2024). Take that bound as exactly 0.90 and apply one stage’s 41% to the whole run, and:

\[ 41\% \times 0.90 = 36.9\% \]

Both of those are choices, so 36.9% is an illustrative figure rather than a number the paper reports. On that reading it would account for 4.1 of the 6.5 points. Not bad kernels, not a weak framework, not incompetence. Hardware fails, a node drops out, the job restarts from the last checkpoint and redoes work. At 16,000 GPUs running for weeks, something breaking every three hours is not a crisis — it is the weather.

The rest looks like our own accounting, not the cluster’s. We estimated the numerator as \(6ND\), and \(6ND\) omits attention. Standard MFU accounting counts the model’s attention matrix multiplies as useful work (it leaves out only extra work such as recomputing activations to save memory), and the paper’s MFU presumably follows that convention; our numerator does not include attention at all. We are comparing two different measurements and calling the difference waste. Add attention back, using the \(6Lnd\)-per-token correction we derived above and the paper’s own schedule — 8K context for the bulk of the run, then “approximately 800B training tokens” ramping through six stages to 128K (Grattafiori et al. 2024):

model	\(6ND\) only	attention-corrected	41% \(\times\,0.90\), illustrative
Llama 3.1 8B	13.9%	15.8–17.2%	not published
Llama 3.1 70B	25.5%	27.4–29.0%	not published
Llama 3.1 405B	34.5%	36.0–37.1%	36.9%

The ranges bracket the long-context tail: the low end assumes it all ran at 8K, the high end at the full 128K, and the truth is somewhere between because the context grew in stages. Only the 405B has a published MFU to check against — the paper’s Table 4 covers “each stage of Llama 3 405B pre-training” and says nothing about the smaller runs. For that one model the corrected range brackets the figure predicted from the paper’s own numbers: 36.0–37.1% around a predicted 36.9%, which is as close as this kind of reconstruction gets. The brochure, the paper, and the invoice look consistent once you stop comparing apples to oranges; but the agreement does not prove how much of the gap comes from downtime and how much from FLOP counting.

The same Table 4 reports 430 TFLOP/s per GPU at the 43% stage. \(430/989 = 43.5\%\) — which confirms the paper is quoting against the dense 989 peak, not the sparsity number. Always worth checking.

Two lessons, and the second is the one that will save you: interruptions are a real and large tax that no amount of kernel tuning removes, and an MFU number means nothing until you know what went in the numerator. When a vendor quotes you utilization, ask.

Why the 8B still looks so much worse. The attention correction lifts the 8B only to 15.8–17.2%, still less than half the 405B’s. That is not the same phenomenon and should not be explained the same way. The possible factors: a small model is harder to keep an accelerator busy with (less arithmetic per unit of communication, less work to hide latency behind), the parallelism layout may have been tuned for another model, and the published hours may include downtime, evaluation or checkpointing — the card defines them only as “the total GPU time required for training each model”. With no published instantaneous MFU for the 8B we cannot apportion these factors. Do not read the 8B row as a measure of kernel quality or of small models’ efficiency in general; what we can say is that in these three runs the big run looks more efficient per GPU.

Estimate your own run

Everything above is four multiplications. Change the inputs and watch which one dominates.

viewof preN = Inputs.range([0.1, 500], { value: 8, step: 0.1, label: "Parameters (billions)" })

viewof preD = Inputs.range([1, 20000], { value: 300, step: 1, label: "Training tokens (billions)" })

viewof preMfu = Inputs.range([5, 60], { value: 35, step: 0.5, label: "Effective MFU (%)" })

viewof preGpu = Inputs.select(
  ["H100 BF16 dense — 989 TFLOP/s", "A100 BF16 — 312 TFLOP/s", "H200 BF16 dense — 989 TFLOP/s"],
  { value: "H100 BF16 dense — 989 TFLOP/s", label: "Accelerator" }
)

viewof preFleet = Inputs.range([8, 32000], { value: 512, step: 8, label: "GPUs in the fleet" })

viewof prePrice = Inputs.range([0.5, 8], { value: 2.5, step: 0.1, label: "Rental price per GPU-hour (USD)" })

preCalc = {
  const peak = preGpu.startsWith("A100") ? 312e12 : 989e12;
  const N = preN * 1e9;
  const D = preD * 1e9;
  const flops = 6 * N * D;
  const gpuSeconds = flops / (peak * (preMfu / 100));
  const gpuHours = gpuSeconds / 3600;
  const days = gpuHours / preFleet / 24;
  const dollars = gpuHours * prePrice;
  const perParam = D / N;
  return { flops, gpuHours, days, dollars, perParam, peak };
}

html`<table style="width:100%; border-collapse:collapse; font-size:0.95rem">
  <tr><td style="padding:6px 10px">compute</td>
      <td style="padding:6px 10px; text-align:right; font-weight:600">${preCalc.flops.toExponential(2)} FLOPs</td></tr>
  <tr><td style="padding:6px 10px">GPU-hours</td>
      <td style="padding:6px 10px; text-align:right; font-weight:600">${
        preCalc.gpuHours >= 1e6
          ? (preCalc.gpuHours / 1e6).toFixed(2) + "M"
          : Math.round(preCalc.gpuHours).toLocaleString("en-US")
      }</td></tr>
  <tr><td style="padding:6px 10px">wall clock on ${preFleet.toLocaleString("en-US")} GPUs</td>
      <td style="padding:6px 10px; text-align:right; font-weight:600">${
        preCalc.days < 1
          ? (preCalc.days * 24).toFixed(1) + " hours"
          : preCalc.days.toFixed(1) + " days"
      }</td></tr>
  <tr><td style="padding:6px 10px">rental cost</td>
      <td style="padding:6px 10px; text-align:right; font-weight:600">${
        preCalc.dollars >= 1e6
          ? "$" + (preCalc.dollars / 1e6).toFixed(1) + "M"
          : "$" + Math.round(preCalc.dollars).toLocaleString("en-US")
      }</td></tr>
  <tr><td style="padding:6px 10px">tokens per parameter</td>
      <td style="padding:6px 10px; text-align:right; font-weight:600">${preCalc.perParam.toFixed(1)}${
        preCalc.perParam < 20 ? " — under Chinchilla" : ""
      }</td></tr>
</table>`

Three settings worth typing in.

8B parameters, 15,000B tokens, 14% MFU reproduces Llama 3.1 8B. The table reads 1.44M GPU-hours against the 1.46M Meta published — the last 1.4% is the slider’s half-point step (13.9% would land on the nose) and the fact that the real model has 8.03B parameters, not 8.00B. Close enough that you can trust the tool.

405B and 15,600B at 35% reproduces the flagship: 30.42M GPU-hours against a published 30.84M. Nudge the MFU slider down one step to 34.5% — the effective figure we computed two sections ago — and it reads 30.86M, which is the published number. That is an internal consistency check: we derived the 34.5% from these very GPU-hours. Once you know a run’s MFU, four multiplications reproduce its GPU-hours; knowing a future run’s MFU in advance is another matter.

10B and 200B at 35% is a serious academic run — 20.0 tokens per parameter, an example of the Chinchilla study’s rough ratio (not a proven optimum for any particular corpus), for about 9,600 GPU-hours and $24,000 at $2.50/hour. Now drop the fleet from 512 to 64, which is closer to what a well-funded university lab actually owns, and watch the wall clock go from under a day to 6.3 days. Then raise the tokens until it stops being acceptable. That last step is the real lesson: at this scale the binding constraint is often not the FLOPs but the calendar; a real plan also has to account for interruptions and access to resources.

The H200 option is deliberately a no-op. Look at the code: it computes the same 989 TFLOP/s as the H100, so picking it changes nothing. That is not a bug. Per NVIDIA’s own datasheets the two chips have identical BF16 tensor-core throughput; what differs is memory — 80 GB at 3.35 TB/s on the H100 versus 141 GB of HBM3e at 4.8 TB/s on the H200. An H200 buys you bigger batches and longer contexts without sharding as hard, which shows up as a better achievable MFU, not as more peak FLOP/s. So if you want to model one, leave the accelerator alone; if you have measured, or have a reliable report of, an MFU gain on your workload, enter that on the slider, and otherwise do not raise it by default. Price it separately too.

Memory capacity and memory bandwidth are what you are usually short of, not arithmetic — and that is a lesson this calculator structurally cannot teach you, because \(6ND\) does not know memory exists. Keep it in mind every time a formula in this chapter gives you a clean answer.

Who actually followed Chinchilla?

Section 7.8 gave the compute-optimal ratio: about 20 tokens per parameter (Hoffmann et al. 2022). Apart from Chinchilla itself, no model in the table below follows it. But the ratio is a reference for Chinchilla’s experimental conditions, not a universal optimum: Meta’s own scaling law, fitted on its data mix, suggests a 402B-parameter model on 16.55T tokens for \(3.8\times10^{25}\) FLOPs — about 41 tokens per parameter. That is why Meta calls the 405B “an approximately compute-optimal size” for its budget (Grattafiori et al. 2024).

model	tokens per parameter
GPT-3 175B	1.7
Chinchilla 70B	20.0
Llama 3.1 405B	38.5
Llama 3.1 70B	212.5
Llama 3.1 8B	1,868.0

GPT-3 sits an order of magnitude below the line, which is exactly the finding Chinchilla was published to correct — it was badly undertrained. Every Llama sits above the Chinchilla line, and the small models sit far above it: the 8B saw almost 1,900 tokens per parameter, ninety-three times the Chinchilla reference. Meta says it did this on purpose: it trained the smaller models far past the compute-optimal point because such models “perform better than compute-optimal models at the same inference budget” (Grattafiori et al. 2024).

This is not a mistake and it is not fashion. It is a different optimisation with a different objective.

Train once, serve forever

A Chinchilla-style choice minimises the loss reached for a given training compute (equivalently: the training compute needed to reach a given loss). If you are going to serve the model on many requests, that objective is incomplete, because training is paid once and inference is paid on every request until the model is retired.

The arithmetic is short. Generating one token costs about \(2N\) FLOPs — the forward pass only, no backward. So for Llama 3.1 8B:

\[ \underbrace{7.23\times10^{23}}_{\text{training}} \;\Big/\; \underbrace{1.61\times10^{10}}_{\text{per generated token}} \;=\; 4.5\times10^{13} \]

Forty-five trillion generated tokens, and the serving FLOPs have equalled the training FLOPs. (generation FLOPs only, under the \(2N\) approximation). A hypothetical service producing 100 billion tokens a day passes that point in 450 days. Not every product reaches that traffic.

Say what that point is and is not. It is FLOP parity for one model: the moment its lifetime inference has cost as much arithmetic as its training. It is not the point at which extra pretraining pays for itself, because training the same model on more tokens does not lower its inference cost per token — the parameter count does that. The break-even that answers “should we have trained longer” is between two models of equal quality: \(A\), the smaller one trained longer, and \(B\), the larger one trained to the compute-optimal point. It is \(T_{\text{break}} = (\text{Train}_A - \text{Train}_B)/(\text{Serve}_B - \text{Serve}_A)\) generated tokens, where Serve is the cost per token. The formula gives a positive threshold only when \(A\) costs more to train and less to serve per token; the costs must be in the same unit, and input processing, idle capacity and operating costs have to be added separately. This is the calculation that makes a smaller, longer-trained model worth its extra training run. And the build-versus-buy question is a third thing again, comparing training plus deployment plus operations against an API invoice.

And 45 trillion is the most favourable number you will ever get. It compares FLOPs to FLOPs, which quietly assumes a generated token and a trained token cost the same to produce. They do not, for two reasons that both push the same way:

• Decode is memory-bound, not compute-bound. Generating one token at a time means reading the entire weight matrix out of memory to do a single small matrix-vector product. Training reads those same weights once and amortises them over thousands of tokens in a batch. Where pretraining sustains 35–40% MFU, decode typically runs at a few percent — the chip is waiting on memory, not computing.
• Prefill is free in this formula and is not free in life. Every input token also costs \(2N\), and most real workloads send far more input than they receive back. A long system prompt, a retrieved document, a chat history: all paid for, none of it counted above.

Redo the comparison in GPU-hours instead of FLOPs — the thing you actually rent — against the 8B’s published 1.46M H100-hours:

decode efficiency	break-even	at 100B tokens/day
13.9% MFU	45.0T tokens	15 months
10% MFU (large batches, well tuned)	32.4T tokens	11 months
5% MFU	16.2T tokens	5 months
2% MFU (typical interactive serving)	6.5T tokens	2 months

Look at the first row and at the 8B’s effective MFU back in the table above. They are the same 13.9%, and that is not a coincidence: dividing FLOPs by FLOPs silently assumes your decode server is exactly as efficient as Meta’s training cluster was. Written down like that, the assumption is obviously untenable — the whole point of decode is that it cannot batch the way training does.

So 45 trillion is the favourable end of this table; in GPU-hours the crossover is somewhere between about two months and just over a year of serious traffic, depending on decode efficiency. That crossover does not by itself say when overtraining pays for itself — the two-model break-even above does, with the training, serving and operating costs of two equal-quality models. What it does say is that under serious traffic the cost of inference can catch up with the cost of training sooner than the clean FLOP count suggests, which is why cutting the inference cost per token matters.

That is the justification for training an 8B model on 15 trillion tokens. You spend 93× the Chinchilla reference in tokens up front, to get a model that is cheaper per token to run than a larger model that would reach the same quality with less training compute. The larger equal-quality model might have been cheaper to train, but it would cost more on every token it serves.

The same arithmetic makes the first cut on your build-versus-buy question. Work out the total tokens you expect the model to generate over its lifetime, multiply by the model’s \(2N\), and compare to that model’s training compute. If you are three orders of magnitude below it, you are a rounding error in someone else’s inference fleet, and renting is most likely the more sensible choice than training. The full comparison that actually decides it — API bill against training plus deployment plus operations — is interview question 3 in Section 8.9.

8.6 The model landscape

Everything so far explains why there are perhaps a dozen organisations on earth doing stage 1 at frontier scale, and why the rest of us are choosing among their outputs. So: how do you tell those outputs apart?

Not by the word “open”. A release is not open or closed; it has to be read under at least four separate headings: weights, training data, training code and licence. They need not be equally open, each can be graded or conditional, and vendors mix them freely.

Four things a release can open, decided separately

The columns are decided separately. A release can publish weights and withhold data; it can use a permissive licence on a model nobody can reproduce. “Open” alone tells you nothing about which.

What each column buys you

Weights buy you deployment. Given them — this is what makes something an open-weights model — you can run it, as far as the licence allows, on your own hardware, in your own country. For requests really not to leave the organisation, the serving architecture, logs, telemetry and connected tools have to be checked too. For an institution weighing cross-border transfers of personal data under KVKK, Turkey’s data-protection law, this can be a strong argument; being subject to KVKK does not by itself require local hosting. You can also fine-tune, quantize, and inspect activations.

open-weights model — one whose trained parameters you can download and run yourself. It says nothing about whether the training data, the training code, or a permissive licence came with them; those are three separate questions. ↩︎

Training data buys you audit. Access to the data and its processing records lets you investigate directly whether your evaluation set leaked into training; without it that gets harder and only indirect tests remain (Oren et al. 2024; Magar and Schwartz 2022). Open data is not by itself a guarantee of a clean evaluation either. Leakage of this kind is the problem called benchmark contamination: if the test items were in the training data, a high score may measure memorisation rather than capability. Without the data you also cannot tell whether your language was represented, cannot answer a lawyer asking where the text came from (Longpre et al. 2024), and cannot investigate a memorised passage that turns up in production.

Training code buys you the ability to inspect the procedure and, more usefully, to modify it — being able to run the recipe on your own corpus rather than starting from a paper. Full reproduction also needs the data, the environment and the compute.

The licence sets the terms on which you may use and distribute, in a product, the components you can get; it gives no access to anything unpublished. This is the column that most often surprises people, because “open weights” and “open source” are routinely used as synonyms and are not.

It is also the column that varies within a row, which is why the diagram says “varies” rather than picking a symbol. All three models named in the middle row publish weights and withhold data and code — but Mistral 7B and most Qwen 2.5 sizes are released under plain Apache 2.0, the same licence the fully open models use, while Llama 3.1 carries custom terms. Two releases can be identical on three columns and differ completely on the fourth, so there is no shortcut here: you have to check the specific model. Sometimes you must check the specific size: Qwen 2.5’s 3B and 72B models ship under Qwen’s own licences rather than Apache 2.0.

Read the licence before the benchmark table. The Llama 3.1 Community Licence is permissive for nearly everyone and is not an open-source licence. Three concrete conditions, from the licence text itself:

• if, on the Llama 3.1 release date, the products or services made available by you or your affiliates had more than 700 million monthly active users in the preceding calendar month, you must request a separate licence from Meta, which Meta “may grant … in its sole discretion”;
• if you distribute or make available the Llama materials, or a product that contains them, you must display “Built with Llama” and provide a copy of the agreement;
• any model you train or improve using Llama or its outputs, and then distribute or make available, must have a name that begins with “Llama”.

None of those will stop a university project or most companies. All of them are conditions that an OSI-approved licence such as Apache 2.0 does not impose, which is exactly why the distinction has a name. The terms can change from version to version; read the original licence of the version you will use. OLMo, by contrast, releases code and weights under Apache 2.0 and its pretraining corpus Dolma under the ODC-BY data licence (Groeneveld et al. 2024; Soldaini et al. 2024) — and its successors have kept that commitment.

The names in the grid go out of date quickly; the grid does not. The examples above are 2024 releases. Since then (as of October 2026) the open-weights row has filled with large mixture-of-experts models under different licences — DeepSeek-R1 under MIT, Qwen3 under Apache 2.0, Llama 4 under another custom community licence — and OpenAI, the best-known API-only lab, released open-weights models of its own (gpt-oss, August 2025, Apache 2.0). So a row describes a release, not a company: one lab can sit in two rows at once. Check each release on all four columns, on the day you choose it.

The fully-open row exists for a reason that matters more to a university than to a startup: it is the row on which controlled replication and ablation are straightforward. If you want to ask “does adding Turkish data at 5% help or hurt English?”, you need the data and the code, not just the weights. The Pile (Gao et al. 2020), RefinedWeb (Penedo et al. 2023), Dolma (Soldaini et al. 2024), and FineWeb (Penedo et al. 2024) exist so that question can be asked at all.

Reading a model card critically

A model card is a short structured document describing what a model is, what it was trained on, and where it should not be used (Mitchell et al. 2019); the same idea applied to datasets is a datasheet (Gebru et al. 2021). They have become common, which means they can be marketing as well as documentation. Their level of detail and of independent verification varies from card to card; check the important claims separately. Read them the way you would read a drug label: the useful information is specific, and the interesting information is what has been left out.

model card — the document a lab publishes alongside a model, stating its intended use, training data, evaluation results, and limitations. Often voluntary and not independently verified, and still one of the most useful pages you will read about a model. ↩︎

Work through Llama 3.1’s card, which is better than most, and the technique becomes clear.

what the card says	how to read it
“15T+ tokens” in the table, “~15 trillion” in the prose	a floor and an approximation, on one page. The paper’s 15.6T is the real one
“cutoff of December 2023”	hard, checkable, and the single most useful line on the card
supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, Thai	Turkish is not on the list
“trained on a broader collection of languages than the 8 supported”	so the list is a support boundary, not a capability measurement
“11,390 tons CO2eq” location-based, “0 tons” market-based	the same training compute (all three models combined), two different Scope 2 accounting methods. A market-based zero does not mean the electricity was physically emission-free; the energy contracts and accounting boundary need checking
benchmark tables	check how they relate to training, tuning and example selection; the scores may be optimistic and do not measure your task distribution

The language row is the one to think hardest about. The card puts use “in languages beyond those explicitly referenced” out of scope — and yet our own measurement two sections ago found Llama 3’s tokenizer handling Turkish at 1.23× English, tied with multilingual BERT for the best of the multilingual tokenizers tested. These are different kinds of evidence: the token count shows that Turkish is encoded efficiently, but it does not measure how accurate or reliable the model is in Turkish. And the card is telling you what Meta will stand behind, not what the model can do.

That gap is the whole reason you measure. A card cannot tell you whether a model is good at your task in your language, because the vendor did not evaluate that. For Turkish, you will often have to do that evaluation yourself.

Four questions no card answers, which you should therefore ask yourself:

1. Was my evaluation set in the training data? You usually cannot know. Take the possibility of contamination seriously for public benchmarks older than the model, investigate it where you can, and keep a private held-out set (Kapoor and Narayanan 2023; Oren et al. 2024).
2. How much of my language was in there? Llama 3’s 8% multilingual slice is shared across dozens of languages. Assume small.
3. What does the model do at the boundary of its documented scope? Test it. That is what Chapter 22 is for.
4. What happens when this model is deprecated? For an API, the answer may be “your prompts stop working on a schedule you do not control”. With open weights the dependence on the provider’s schedule goes away — a real feature with a real price: the licence, the runtime environment, security and maintenance stay your responsibility.

Choosing one, in practice

The decision has four inputs, and they have to be weighed together. But starting from capability is often inefficient: writing down the hard requirements of the use first, eliminating candidates that fail them, and only then measuring the rest takes less work. This order is a good start:

1. Where is the data allowed to be processed? Whether an external API is acceptable depends on where it processes the data and on the contract terms; if personal data may not leave the country or the building and no API option meets that, the API row is gone and no benchmark score brings it back. Decide this first; it often eliminates more options than anything else.
2. What does the licence permit? Check the user-count clause, the naming clause, and the attribution clause against your actual product. Do this second, while it is still cheap to change your mind.
3. What can you afford to serve? From the arithmetic above: \(2N\) FLOPs per generated token. Multiply by your expected traffic. This usually chooses your parameter count for you, and it usually chooses something smaller than you assumed.
4. Only now, is it good enough? Evaluate the survivors on your data. The public leaderboard is a screening tool at best, and after contamination it is barely that.

A starting layout that is often worth trying. Start with the smallest open-weights model that clears your quality bar on your own evaluation set, and treat a frontier API as the thing you escalate to for the hard tail of requests. It is often cheaper and keeps the main workload under your control; but if the hard requests go to an external API, assess privacy for those requests separately. Test routing errors, total latency, the upkeep of two models and the cost on your own traffic distribution; do not assume that a cost advantage appears by itself as traffic grows. Chapter 31 takes this apart properly.

8.7 Lab: price a Turkish pretraining run

Everything in this lab is arithmetic and tokenizer calls; none of it needs a GPU. The aim is to make the assumptions visible before a large run starts: the decision this chapter is about gets made long before training begins.

1. Measure fertility on your own text. Take 2,000–5,000 sentences of Turkish from a domain you care about — support tickets, legal text, news, clinical notes, your own thesis. Tokenize with at least four tokenizers: one English-first (openai-community/gpt2), two multilingual (NousResearch/Meta-Llama-3-8B, Qwen/Qwen2.5-7B), and one Turkish-native (dbmdz/bert-base-turkish-cased). Those are the identifiers behind the table in Section 8.4, but do not expect the same numbers on your own domain text; try to explain the differences by domain, length and spelling. To reproduce the chapter’s table, use the same pairs: the first 3,000 pairs of the OPUS-100 en-tr training split, the same filter, the sentences joined with spaces before tokenizing. Report tokens-per-word and total tokens for each. Do not report tokens-per-word alone as a cross-language claim; you now know why.
2. Get the honest ratio. Translate — or find an existing parallel version of — 200 of those sentences into English. Recompute the ratio on matched content. State how far your domain’s number is from this chapter’s 1.23× for Llama 3, and give one concrete reason for the gap. Domain vocabulary is the usual culprit: proper nouns, code, numbers and acronyms tokenize almost identically in both languages and drag the ratio toward 1.
3. Turn the ratio into money. Pick a real API model and its current price list. Measure the input and output token counts of Turkish and English examples with that model’s own tokenizer, and compute what one million of your Turkish documents cost to process; then do the same for the English translations. The difference is the surcharge, in your currency, for your product. Use the other tokenizers for a compression comparison; do not carry their counts onto the bill of a model they do not belong to.
4. Find the context ceiling. Measure the token counts of representative Turkish and English documents with each model’s own tokenizer and chat template. Subtract the system prompt, the retrieved chunks and the budget you reserve for the answer from the 128K window; report how many Turkish words and how many English words fit in what is left. This is the number that decides whether a document fits in the context window in one piece or has to be split and served piece by piece through retrieval (Chapter 19).
5. Budget a run with \(C = 6ND\). Suppose you want a Turkish-first base model. Choose \(N\) and \(D\) yourself and justify both: \(N\) from what you intend to serve (Chapter 31), \(D\) from what Turkish text actually exists at the quality you need. Use the calculator in Section 8.5 to get FLOPs, GPU-hours, wall-clock on a fleet you could plausibly rent, and a dollar figure. Then state your assumed MFU and defend it — if you wrote down a number above 40%, say which paper you are matching.
6. Sanity-check against reality. Compare your budget to the three published Llama 3.1 runs. If your number is far smaller, find the assumption doing the work. It is usually MFU or the token count.
7. Now do the sensitivity analysis. Set up two separate scenarios: (A) the real token count of your representative Turkish corpus under your chosen Turkish-first tokenizer; (B) the real token count of the same corpus under your chosen multilingual tokenizer. Apply \(6ND\) and the MFU assumption to each separately; find the difference from the token counts you measured, not by adding a fixed 1.23 factor to your existing \(D\) (the 1.23 was measured on another corpus with other tokenizers, and applying it twice can double-count). Report the extra dollars. Then answer the question the number poses: at what point does it become cheaper to train a Turkish tokenizer and a smaller model than to keep paying the surcharge on a multilingual one?
8. Compute your own break-even, twice. Using \(2N\) FLOPs per generated token, find how many tokens your service must generate before serving costs exceed training. That is the FLOP-parity answer, and you now know it is the favourable end. So do it again in GPU-hours, assuming decode sustains 2–5% MFU against your training run’s 35%. Convert both to months at your projected traffic. If the honest answer is under a year, you have just discovered why inference-optimal and compute-optimal are different objectives. This calculation is illustrative; for the product decision, compare the training or adaptation, deployment, maintenance and API costs of the candidates that clear your quality bar over the same time horizon.

Deliverable: a one-page memo — the kind you would actually send — that recommends train, fine-tune, or use an API for a specific Turkish product, with the same-content ratio table, the compute budget, and the break-even month as its three pieces of evidence. State your data-access, licence and quality assumptions, and the assumption that, if wrong, would flip your recommendation.

8.8 Intuition checks

Answer with your own reasoning before expanding. In most of them the intuitive answer is wrong in a direction that costs money; watch which assumptions the numbers in the answers depend on.

Note1. With Llama 3’s tokenizer, Turkish needs 2.12 tokens per word and English needs 1.28. Does Turkish cost 66% more?

No; the comparison is invalid as posed. On this chapter’s parallel corpus Llama 3’s tokenizer produced about 23% more tokens for the same content in Turkish. That is an input-token difference, not a total fee before model price, output and traffic are counted.

Tokens-per-word compares units of different size. Turkish is agglutinative: evlerinizden is one word meaning “from your houses”. The honest measurement is tokens for the same content — tokenize a sentence and its professional translation and divide.

Do that on 1,157 aligned pairs and BERTurk costs 0.58× what English costs; among those tested, the Turkish-native tokenizers are the only ones where Turkish is cheaper. GPT-2 costs 1.98×, Llama 3 and multilingual BERT 1.23×.

The same corpus is 8,058 Turkish words against 10,862 English words of matched content — the agglutination is visible in the raw counts before any tokenizer touches it.

Note2. Your model tokenizes Turkish badly. Can you fine-tune your way out of it?

Not with an ordinary fine-tune, and that is why the tokenizer belongs in a pretraining chapter rather than a preprocessing one.

Changing or extending the tokenizer needs new token ids and embedding (and, where needed, output) rows for them. The old weights do not change by themselves; but because the rest of the network was learned on the old segmentation, using the new tokens reliably takes adaptation training, which updates those weights too. It is not impossible: you can extend the vocabulary, initialise the new embeddings from the pieces they replace, and continue pretraining until the model settles — a documented and occasionally worthwhile procedure, and vocabulary-adaptation methods exist that initialise the new rows from the old ones rather than from scratch. What it is not is free or safe. Its cost scales with how much of the vocabulary you change, what you start from and what quality you will accept, and it risks degrading things the model already does well; it is not automatically a second pretraining run, and it is not an ordinary fine-tune either. The tokenizer is how the compute budget gets divided between languages, and undoing that division is a project with its own budget. Absent that project, a user of a 1.98× tokenizer pays the surcharge in context window, in latency, and on every API invoice, for the life of the model.

Llama 3 shows the decision being made deliberately: 100,000 tokens from tiktoken plus 28,000 added specifically for non-English languages, reported as improving compression for those languages with no cost to English. That is a design choice with a price tag, made once, before the first gradient step.

Note3. A vendor quotes 40% MFU. Is that good?

Unanswerable until you ask what went in the denominator, and asking is the whole point of the question.

The same GPU has a dense peak and a sparsity peak twice as large, and the denominator decides the answer: dividing by the larger one halves the reported MFU. Llama 3’s Table 4 reports 430 TFLOP/s per GPU; against the H100’s dense BF16 peak that is \(430/989 = 43.5\%\), and against the 2:4-sparsity peak the same run would report \(430/1{,}979 = 21.7\%\). The 43.5% confirms they used the dense figure. Two MFUs are comparable only when they used the same peak, and vendors do not always say which they used; ask. Ask too which model operations the count includes and over what period the ratio was taken: Llama 3’s 41% for one stage and the ratio derived from the model card’s total GPU-hours are not the same kind of measurement.

The second trap is the numerator. Reconstructing Llama 3.1 405B from \(6ND\) alone gives 34.5%; adding the attention term gives 36.0–37.1%, which brackets the 36.9% our illustrative 41% × 0.90 estimate predicts. \(6ND\) omits attention, which is fine for a budget estimate and not fine for a utilization comparison.

Two numbers that look comparable and are not is the most common way engineering conversations about efficiency go wrong.

Note4. Where does the 6 in \(C = 6ND\) come from?

Three twos, each with a cause, and knowing them is what lets you adapt the formula instead of reciting it.

Two FLOPs per parameter in the forward pass, because each weight participates in one multiply and one add. Two more for the backward pass through the inputs, because the error has to keep travelling to earlier layers. Two more for the backward pass through the weights, because the optimiser needs to know what to change.

So the forward pass is \(2ND\) and is only a third of the bill — which immediately tells you that inference costs roughly \(2N\) FLOPs per token (hence \(2ND\) for \(D\) tokens), not \(6N\), and that is the number that decides whether serving or training dominates your budget. It also tells you why \(6ND\) is a pretraining rule and does not transfer unchanged to fine-tuning with frozen weights.

Note5. You have more compute. Should you train a bigger model or use more data?

Under Chinchilla’s experimental conditions, and if enough new data exists, both in roughly equal measure — and the interesting part is that this was the field’s mistake for several years. The split depends on how much data is available and on the target loss.

Chinchilla’s finding is that parameters and tokens should scale together; earlier models were badly undertrained for their size. But the compute-optimal answer is not the answer you want if you are going to serve the model: a smaller model trained longer costs more to train and less to run per token, which is why production models are routinely trained well past the compute-optimal point.

The lab’s break-even question is the practical form: if your model will serve many requests, inference-optimal and compute-optimal are different objectives. Which to aim for is decided not by service time alone but by the expected total request volume, the quality bar and the cost per token.

Note6. Can you just train on the same data twice instead of collecting more?

Up to a point; after that the marginal benefit shrinks.

Muennighoff et al. found that, in the regime they measured, repeated data “yields negligible changes to loss compared to having unique data” for up to about four epochs. Past that, each further epoch is worth less; the paper still finds gains up to around 16 epochs, after which returns fall off fast and the value of extra compute approaches zero.

The practical reading for a language like Turkish: in many Turkish pretraining runs the constraint may be the corpus more than the GPUs, and repetition buys you a small multiple rather than an escape. That is also why the data-filtering decisions in Section 8.2 get so much attention — when you cannot get more text, what you keep matters much more.

8.9 Interview questions

Pretraining questions are rarely about whether you would run one. They test whether you can reason quantitatively about a system you inherited — compute, data and tokenizer decisions, the conditions of model choice. Give numerical estimates together with the assumptions they rest on.

Almost nobody being interviewed has pretrained a model, and the interviewer knows it. What separates answers is whether you can do the arithmetic that decides build, fine-tune, or buy — and whether you know which pretraining decisions are expensive to change afterwards.

1. Estimate the compute for a 7B model on 2 trillion tokens. \(C \approx 6ND = 6 \times 7\times10^9 \times 2\times10^{12} \approx 8.4\times10^{22}\) FLOPs. Then convert to something a budget-holder understands: multiply the chosen accelerator’s dense peak by a measured or scenario MFU range for this job (say 25–40%) and by 3,600 seconds; dividing the budget by that gives a range of GPU-hours. Say out loud that \(6ND\) ignores attention, which for a 7B model with 32 layers and width 4096 adds roughly 4% at 2K context, 8% at 4K and 15% at 8K, and much more at long context, and that what blows up a pretraining schedule is often not the FLOPs but the calendar: interruptions, restarts and checkpoint reloads.

2. Which pretraining decisions are expensive to change later? The tokenizer leads the list. Instruction behaviour, safety, domain knowledge, even a good deal of factual content can be moved by post-training (Chapter 9). The tokenizer resists: changing or extending the vocabulary needs new embedding rows and an adaptation run in which the rest of the weights are updated too — possible, but a project rather than a fine-tune. The data mix, the architecture and the base capacity are also expensive to change later; rather than ranking “the one decision you cannot undo”, explain what each change would take. Quantify the stakes with the same-content ratio from the tokenizer table: a Turkish user on GPT-2’s tokenizer pays 1.98× the tokens of an English user for the same content, and on BERTurk 0.58×. Same language, same text, a 3.4× spread driven entirely by a decision made before training started.

3. How would you decide whether to pretrain, fine-tune, or use an API for a Turkish product? First the filters: quality on the target task, data handling conditions and the licence. Then three numbers, in this order. The same-content ratio (Turkish ÷ English tokens) measured on real Turkish requests, because it sets your per-content cost for as long as you use that tokenizer; but on its own it does not justify a Turkish model. The \(6ND\) budget for the smallest model that could work, which is often large enough to end the conversation. And the break-even month: the point where API spend at your projected traffic exceeds training plus serving. If that month is years away, buy. (This is the build-versus-buy break-even, not the inference-optimal one from Section 8.5; keep the two apart.) Compare the options over the same time horizon and show how the decision moves when an assumption changes. The answer often comes out as fine-tune or buy; but show it with arithmetic and your own quality measurement rather than assuming it.

4. Where does pretraining data actually come from, and what is thrown away? Common Crawl or a cleaned derivative such as FineWeb, and the honest headline is how much is discarded: filtering, deduplication and quality classification remove the large majority of raw crawl. Name the specific steps — boilerplate removal, language identification, deduplication across crawls, quality classifiers — and the counter-intuitive finding that FineWeb’s authors hit: aggressive deduplication across all crawls raised the share of low-quality text in what an old crawl kept, and did worse at an equal token budget. Deduplication is not monotonically good, which is why a filter’s strength needs testing with small, controlled runs.

5. A model claims a 128K context window. What should you be sceptical about? That the quality is uniform across it, and that you can afford it. On quality, training rarely happens at full length throughout — as in Llama 3, the context is often extended in stages near the end — so the last 100K tokens may have seen far less training than the first 8K. On cost, take Section 7.6’s arithmetic: for a model with 32 layers and width 4096 (Llama 3 8B’s dimensions), at 16 bits and one request, the KV cache at 100,000 tokens would be 52.4 GB per concurrent request with plain multi-head attention, against 14 GB of weights; with the grouped-query attention Llama 3 8B uses (8 KV heads instead of 32) the same cache is 13.1 GB. Other 7B–8B models can differ; measure long-context quality and per-request KV use on your own model. A 4× difference out of one line of config. The window being advertised and the window being usable at your budget are different numbers.

6. Why do small pretraining runs look so much less efficient than large ones? Because efficiency here is mostly about keeping an accelerator busy, and small models give it less to work with: less arithmetic per unit of communication, less work to hide latency behind. Reconstructed MFU for Llama 3.1 goes 15.8–17.2% at 8B, 27.4–29.0% at 70B, 36.0–37.1% at 405B. Then add the caveat that makes the answer honest: the card does not say how far the published GPU-hours include downtime, evaluation or checkpointing, and only the 405B has a published MFU to check against — so you cannot apportion the factors, and the 8B figure should not be read as a measure of kernel quality.

7. Your Turkish (or Arabic, or Finnish) product costs far more per request than the English one. Explain and fix. First compare how many tokens the same content produces under the tokenizer in use. Differences between languages can be related to the distribution of the training data, the vocabulary, the writing system and the tokenization rules (Petrov et al. 2023). In the cost of a request, also look separately at input and output lengths, model prices and any caching conditions. On the short parallel text in this chapter the Turkish/English token ratio was 1.82 for cl100k_base and 1.39 for o200k_base. Those figures illustrate a tokenizer-dependent difference; to estimate the cost ratio of your product, measure over a large number of representative requests. Fixes, in increasing order of effort: pick a model whose tokenizer produces fewer tokens in this language; measure cost in tokens per task rather than per request; given enough scale and training budget, consider training a domain- and language-specific tokenizer together with a compatible model, or adapting the existing model to it (Kudo and Richardson 2018). The tokenizer of a pretrained model cannot be swapped like an independent part without making the model compatible.

8. Explain BPE. What happens to a word the tokenizer has never seen? Describe the merge loop, then answer the real question: if the base alphabet covers the word’s characters, nothing special happens; the word decomposes into known subwords, in the worst case single characters. If the base alphabet covers the input, or the tokenizer can write an unknown character as its bytes (byte fallback), there is no need for <UNK> because of an unrepresentable character. That coverage does not mean no information is removed during preprocessing, though; losslessness has to be assessed for the tokenization pipeline as a whole. For unknown characters the base alphabet and byte-fallback support are decisive; the algorithm’s segmentation rules and the limits of the implementation matter too. The classic BERT WordPiece tokenizer can emit [UNK] for a character outside its vocabulary. Even with known characters, the word-length limit can give the same result: the mBERT tokenizer splits a word of 100 letters a into 50 pieces and turns one of 101 letters into a single [UNK]. Byte-level or byte-fallback designs write the unknown character as its bytes. Use the worked example from this chapter: the word lowest, absent from the BPE tokenizer’s training example, is split into low + est by the learned merges.

8.10 Where to go deeper

Two things are worth doing before reading any of the papers. Open FineWeb’s interactive report and read the ablation plots — a report that shows the failed variants alongside the shipped one, and lets you see controlled comparisons of data filters. Then open the tokenizer playground and paste in the longest Turkish word you know; comparing the splits shows the main idea of Section 8.4 in a few seconds.

Then watch someone do it: Karpathy’s Let’s reproduce GPT-2 (124M) is four hours of actual small-scale pretraining, end to end (it does not verify this chapter’s large-scale numbers, but it makes them concrete), and Stanford CS336 is a whole course built around doing it yourself.

Tokenization, hands on.

• Hugging Face LLM Course, Chapter 6. Builds BPE, WordPiece and Unigram tokenizers and explains which models use which. The BPE lesson is the direct sequel to our toy implementation.
• minbpe and the accompanying two-hour video, Andrej Karpathy (Karpathy 2024). A few hundred lines of clean Python that build a GPT-grade tokenizer from scratch. Watch the video, then read the code line by line; the two were designed together.
• The Tokenizer Playground. Paste text, switch tokenizers, watch the splits. Try your own name, your company’s part numbers, and a paragraph of Turkish.
• tiktoken. The library used for the cost measurements in this chapter.

The papers below are the ones this chapter’s numbers came from. Read them in roughly this order:

topic	source
a full pretraining run, documented	Grattafiori et al. (2024)
what the data pipeline does, with ablations	Penedo et al. (2024); Penedo et al. (2023)
the corpora, published in the open	Gao et al. (2020); Soldaini et al. (2024)
an end-to-end open model — weights, data, and code	Groeneveld et al. (2024)
the effect of duplication and deduplication; extracting memorised passages from training data	Lee et al. (2022); Carlini et al. (2021)
repeating epochs when data is limited	Muennighoff et al. (2023)
the compute-optimal frontier	Kaplan et al. (2020); Hoffmann et al. (2022)
subword tokenization, at the source	Sennrich et al. (2016); Kudo and Richardson (2018)
what tokenizers cost speakers of other languages	Petrov et al. (2023)
where the data actually came from, and its licences	Longpre et al. (2024); Dodge et al. (2021)
cards and datasheets	Mitchell et al. (2019); Gebru et al. (2021)
training on model output	Shumailov et al. (2024)
the argument that scale beats cleverness	Sutton (2019)

8.11 Exercises

1. Take one open model card and one closed model card. List three claims each makes that you could verify yourself, and three you cannot. Then state which of the six unverifiable claims would most change your decision if it turned out to be false.
2. The FineWeb team tested global deduplication with a pointed experiment. Inside the single old 2013-48 snapshot they took the ~31B tokens global dedup had kept and the ~460B tokens it had removed (re-deduplicated on their own, giving 171B), trained a model on each, and found the model trained on the removed data was better. Explain the mechanism — note that the experiment compares two slices of one snapshot, not two corpora, which is what makes it a clean test. Then name one filter elsewhere in the funnel that could fail in exactly the same way, and say what you would measure to catch it.
3. The published Llama 3 405B run ends with a 40M-token annealing stage: 0.00026% of the run. Separately, in an annealing experiment on a mix that upweighted high-quality code and maths, the paper reports a 24.0% improvement on the GSM8k validation set for the 8B model and essentially nothing on the 405B. The paper does not say whether 24.0% is relative or in percentage points; say how each reading would change how impressed you should be. Say which conclusions from that separate experiment cannot be carried over to the published 405B run. Then give an explanation consistent with both results. Your explanation should predict something — say what.
4. A vendor advertises 1,979 TFLOP/s for their accelerator. You measure a sustained 310 TFLOP/s on your own training job. Work through the gap in three named steps, using this chapter’s figures for each, and state what fraction of the shortfall you could plausibly recover with better engineering versus what is structural.
5. Compute \(6ND\) for a 3B model on 2T tokens. Then compute how many H100-hours it needs at 40% MFU, and how long that takes on 64 GPUs. Now recompute at 25% MFU — the figure you should assume for a first run on a cluster nobody has tuned. Report both wall-clock numbers and say which one belongs in a project plan.
6. Our corpus measurement found Qwen 2.5 (vocab 151,643) tokenizing Turkish at 1.40× English, while Llama 3 (vocab 128,000) managed 1.23×. Show that vocabulary size alone does not explain the ratio. Set up separate hypotheses for pre-tokenization, the language mix of the tokenizer’s training data and the vocabulary’s content; for each, say what you would need to compare, the smallest experiment that would confirm it, and what result would refute it.
7. For evlerinizden, Llama 3 produces ev|ler|iniz|den — four tokens that coincide with the four morphemes of the analysis we chose — while Cosmos GPT-2 produces ev|lerinizden: two tokens, half the cost, and a second token that is not a single morpheme. Argue for each as the better segmentation. Then state what you would measure to settle it, and be specific about the task, because the answer is not the same for every task.
8. GPT-3 used 1.7 tokens per parameter; Llama 3.1’s 8B used 1,868. Both were rational decisions. Reconstruct the reasoning behind each and explain the difference between compute-optimal training and low inference cost. Then show why the traffic break-even cannot be computed without the training and per-token serving costs of two equal-quality models; list the data you would need, and compute a threshold with values you state as assumptions.
9. You are given a hypothetical fixed budget of 10,000 H100-hours. First pick the dense BF16 peak and an achievable MFU range. Using this chapter’s arithmetic, and the Chinchilla study’s rough tokens-per-parameter ratio as one scenario, work out a few model sizes and token budgets you could pretrain from scratch. Then argue, with numbers, for spending the same allocation on continued pretraining of an existing open-weights model instead, taking target-task quality and data access into account. State which you would actually do and why.
10. Common Crawl’s WET extraction “retained too much boilerplate and menu text”, so FineWeb re-extracted from WARC with trafilatura. Boilerplate is highly repetitive. Explain why deduplication does not simply remove it, and what that tells you about the order of operations in a data pipeline.
11. Tokenizer economics. Take a paragraph of your own technical writing in Turkish and its English translation. Using the Tokenizer Playground or tiktoken, count tokens under two different tokenizers. First compute the ratio of token counts for the same content in the two languages. Then estimate the annual cost at 100,000 requests per month, using the unit prices of the model you choose and stating their date. Report input and output tokens, the shared prompt and any cached input separately, and state explicitly the assumption that one paragraph represents all requests.
12. Turkish morphology and token count. Repeat the evlerimizden example with terms from your own field: pick a technical term (basınç “pressure”, sıcaklık “temperature”, dayanım “strength”) and choose five different inflected forms of it. Measure the token count of each form with the same tokenizer, for example tiktoken. How does the number of tokens per word change? Then compare the token counts of forms that keep Turkish spelling with forms reduced to ASCII. Interpret the result as limited to these examples and this tokenizer: the conversion changes the spelling and distinguishing information, and producing fewer tokens does not mean a more accurate or more useful representation.
13. Measure the Turkish token cost over many examples. The Declaration measurement in Section 8.3.3 used one short pair of texts, and its 1.82 (cl100k_base) and 1.39 (o200k_base) ratios are not general; Section 8.4 gives a separate measurement over 1,157 pairs with other tokenizers. Collect 50 to 100 sentence pairs from a parallel text you have — articles of the Universal Declaration of Human Rights, a piece of software’s interface strings in two languages, your own bilingual reports. For each tokenizer make two separate measurements: the Turkish/English ratio of token counts for each aligned pair, and the number of tokens per 1,000 Unicode code points in each language. The first shows the token difference for the same content, the second the token density relative to the length of the writing. Report the median and the interquartile range; for an estimate of total cost, also use the total token counts over all examples and the expected distribution of requests. Where in that distribution did the single-pair measurement fall?

NoteExpected observations for exercises 11–13

Not solutions — but knowing what you are meant to see makes it easier to check yourself.

11. The Turkish paragraph can produce more tokens under some tokenizers. Measure the direction and size of the difference, and report the result together with the text pair and tokenizer version you chose.
12. The token count can rise as suffixes are added; but because of learned pieces the rise need not be monotonic. ASCII conversion does not reduce the token count in every example either. Illustrate the loss of spelling distinctions; if you want to claim a change in search quality, measure it separately.
13. Report the token density of each language and the token ratios of the aligned pairs separately. The direction and spread of the difference depend on the examples and on the tokenizer. A single example is one observation inside the distribution, not a summary of it.

Bostrom, Kaj, and Greg Durrett. 2020. “Byte Pair Encoding Is Suboptimal for Language Model Pretraining.” Findings of the Association for Computational Linguistics: EMNLP 2020, 4617–24. https://aclanthology.org/2020.findings-emnlp.414/.

Brown, Tom B. et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33.

Carlini, Nicholas, Florian Tramer, Eric Wallace, et al. 2021. “Extracting Training Data from Large Language Models.” 30th USENIX Security Symposium, 2633–50.

Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” Proceedings of NAACL-HLT.

Dodge, Jesse, Maarten Sap, Ana Marasović, et al. 2021. “Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus.” Proceedings of EMNLP.

Gage, Philip. 1994. “A New Algorithm for Data Compression.” The C Users Journal 12 (2): 23–38.

Gao, Leo, Stella Biderman, Sid Black, et al. 2020. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling.” arXiv Preprint arXiv:2101.00027. https://arxiv.org/abs/2101.00027.

Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, et al. 2021. “Datasheets for Datasets.” Communications of the ACM 64 (12): 86–92.

Grattafiori, Aaron, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. “The Llama 3 Herd of Models.” arXiv Preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783.

Groeneveld, Dirk, Iz Beltagy, Pete Walsh, et al. 2024. “OLMo: Accelerating the Science of Language Models.” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 15789–809. https://arxiv.org/abs/2402.00838.

Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, et al. 2022. “Training Compute-Optimal Large Language Models.” Advances in Neural Information Processing Systems 35. https://arxiv.org/abs/2203.15556.

Jiang, Albert Q., Alexandre Sablayrolles, Antoine Roux, et al. 2024. Mixtral of Experts. https://arxiv.org/abs/2401.04088.

Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, et al. 2020. “Scaling Laws for Neural Language Models.” arXiv Preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361.

Kapoor, Sayash, and Arvind Narayanan. 2023. “Leakage and the Reproducibility Crisis in Machine-Learning-Based Science.” Patterns 4 (9): 100804.

Karpathy, Andrej. 2024. Minbpe: Minimal, Clean Code for the Byte Pair Encoding Algorithm. https://github.com/karpathy/minbpe.

Kudo, Taku. 2018. “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates.” Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 66–75. https://aclanthology.org/P18-1007/.

Kudo, Taku, and John Richardson. 2018. “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing.” Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 66–71. https://aclanthology.org/D18-2012/.

Lee, Katherine, Daphne Ippolito, Andrew Nystrom, et al. 2022. “Deduplicating Training Data Makes Language Models Better.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 8424–45. https://arxiv.org/abs/2107.06499.

Longpre, Shayne, Robert Mahari, Anthony Chen, et al. 2024. “A Large-Scale Audit of Dataset Licensing and Attribution in AI.” Nature Machine Intelligence 6 (8): 975–87.

Magar, Inbal, and Roy Schwartz. 2022. “Data Contamination: From Memorization to Exploitation.” Proceedings of ACL.

Mitchell, Margaret, Simone Wu, Andrew Zaldivar, et al. 2019. “Model Cards for Model Reporting.” Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*).

Muennighoff, Niklas, Alexander M. Rush, Boaz Barak, et al. 2023. “Scaling Data-Constrained Language Models.” Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2305.16264.

Oflazer, Kemal. 1994. “Two-Level Description of Turkish Morphology.” Literary and Linguistic Computing 9 (2): 137–48. https://doi.org/10.1093/llc/9.2.137.

Oren, Yonatan, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. 2024. “Proving Test Set Contamination in Black-Box Language Models.” International Conference on Learning Representations (ICLR).

Ouyang, Long et al. 2022. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems 35.

Penedo, Guilherme, Hynek Kydlı́ček, Loubna Ben allal, et al. 2024. “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.” Advances in Neural Information Processing Systems 37 (NeurIPS 2024) Datasets and Benchmarks Track. https://arxiv.org/abs/2406.17557.

Penedo, Guilherme, Quentin Malartic, Daniel Hesslow, et al. 2023. “The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.” Advances in Neural Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track. https://arxiv.org/abs/2306.01116.

Petrov, Aleksandar, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. “Language Model Tokenizers Introduce Unfairness Between Languages.” Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2305.15425.

Provilkov, Ivan, Dmitrii Emelianenko, and Elena Voita. 2020. “BPE-Dropout: Simple and Effective Subword Regularization.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1882–92. https://doi.org/10.18653/v1/2020.acl-main.170.

Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models Are Unsupervised Multitask Learners. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

Schuster, Mike, and Kaisuke Nakajima. 2012. “Japanese and Korean Voice Search.” 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5149–52. https://doi.org/10.1109/ICASSP.2012.6289079.

Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. “Neural Machine Translation of Rare Words with Subword Units.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1715–25. https://aclanthology.org/P16-1162/.

Shumailov, Ilia, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. “AI Models Collapse When Trained on Recursively Generated Data.” Nature 631: 755–59.

Soldaini, Luca, Rodney Kinney, Akshita Bhagia, et al. 2024. “Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research.” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 15725–88. https://arxiv.org/abs/2402.00159.

Sutton, Richard S. 2019. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30.

Xue, Linting, Aditya Barua, Noah Constant, et al. 2022. “ByT5: Towards a Token-Free Future with Pre-Trained Byte-to-Byte Models.” Transactions of the Association for Computational Linguistics 10: 291–306. https://doi.org/10.1162/tacl_a_00461.