9 Post-Training: Adaptation and Alignment
Chapter 8 ended with a model that had read the web and could not be spoken to. This chapter is about the second and third stages: the ones that are comparatively cheap, that you can actually run, and that shape much of the behaviour a user notices. What the model knows and can do still comes largely from pretraining and the architecture.
Learning objectives. After this chapter you can:
- explain why a base model answers a question with another question, and what supervised fine-tuning changes so that it stops;
- write a training example in chat-template form and say which tokens the loss is computed on and why the rest are masked;
- work out, from a model’s
config.json, how many parameters a LoRA adapter on given target matrices adds, and estimate the run’s GPU memory once you state your precision, batch, sequence-length and training-setup assumptions; - distinguish SFT, RLHF, DPO, and GRPO by the data, the reward signal and the training compute each one needs;
- choose between prompting, retrieval, fine-tuning, and distillation with an explicit decision rule;
- look for catastrophic forgetting by evaluating the model before and after on general tasks and held-out benchmarks chosen before training.
9.1 The model that answers a different question
Here is the problem in one representative exchange (the actual output varies with the model, the prompt and the sampling settings). You have a base model — the raw output of stage 1, nothing else done to it. You type:
What is the capital of Turkey?and it may continue:
What is the capital of France?
What is the capital of Germany?
What is the largest city in Turkey?Nothing has gone wrong. The model did exactly what it was trained to do, and that is the problem. It was trained on one objective — given this text, what text usually comes next — and reading the prompt as the start of a list of questions is a reading that objective allows. It may be completing a worksheet; with another prompt or sampling setting it may just say “Ankara”. Supervised fine-tuning aims to make the question-and-answer behaviour reliable.
Try this yourself: on the Hub, models named Llama-3.1-8B and Llama-3.1-8B-Instruct are the same pretraining run. The first is the base model. The second has been through the two stages in this chapter. The difference in behaviour is large; the post-training compute for these two models is not published. For scale, a separate example: the SFT (4.9) and PPO-ptx (60) figures reported for InstructGPT’s 175B version total 64.9 petaflop/s-days against 3,640 for GPT-3’s pretraining — about 1.8%, or a factor of 56, leaving out the reward model and data collection.
Give the same base model a fact it very likely knows, phrased as a sentence to be finished — The capital of Turkey is — and it may well produce “Ankara”. That shows it can use the fact in some contexts, not that it will answer every question correctly and consistently. What is missing here is less information than a convention: that a question addressed to you is a request for an answer from you.
This is worth being precise about, because it sets up everything that follows.
Four things you might be missing, and what to try first for each.
| what is missing | symptom | what to try first |
|---|---|---|
| knowledge | confidently wrong about your documents, your prices, last week | retrieve the source and verify the answer (Chapter 20) |
| format | right content, wrong shape: prose where you needed JSON | a clear prompt, constrained decoding or an output schema |
| behaviour | answers the wrong kind of question; wrong register; inconsistent refusals | evaluate with examples; fine-tune if needed (this chapter) |
| cost | accurate enough, but too slow or too expensive per call | a smaller model, quantization or distillation |
These are not strict mappings (retrieval fails if it fetches the wrong document, and knowledge can also be added by continued pretraining), but they are a good start. Fine-tuning a model whose knowledge is out of date on behaviour examples alone may not fix the problem if the problem is a wrong source.
We will come back to that table at the end and turn it into a decision tree. First, the machinery.
What we will look at. First the training example and loss mask of supervised fine-tuning, then LoRA and preference optimization; after that, feedback that is not human, distillation and continued pretraining. Finally, when not to fine-tune, and how to measure performance regressions after you adapt a model.
9.2 What the stages actually do
Chapter 8 drew the three stages by scale. Here is the same pipeline drawn by mechanism: what each stage puts in, what signal it optimizes, and which weights it is allowed to touch.
The same three stages, drawn by what they optimize
The column that matters is the third one. Pretraining scores what came next in the corpus; SFT scores what should come next in the answer (usually on the assistant’s tokens only); the preference-tuning example shown scores which of two answers is better. That last signal cannot be written down as a single target string — which is why it needs different algorithms. Other post-training methods can use a verifiable reward or a chosen response text instead; which fits depends on the data you have.
Notice what stages 2 and 3 have in common: neither one is where the model’s knowledge of the world mostly comes from. That is not a claim that supervised fine-tuning cannot teach facts — it demonstrably can, and continued pretraining on domain text is a standard way to add some. The claim is about scale and reliability: a stage-2 run touching a few million tokens should not be expected to revise, broadly and dependably, a world model built from millions of times more text (15.6 trillion tokens against a few million). Chosen examples can change some factual answers, but whether the change lasts and generalises to other questions has to be tested separately. When people reach for fine-tuning as their knowledge-update mechanism, it often disappoints. For information that changes often or needs a citable source, retrieval is usually the better place to start; Chapter 20 covers it.
9.3 Supervised fine-tuning
One example, in full
A supervised fine-tuning example is usually one or more user messages and the assistant response you wish the model had given; below we use a single-turn one. The idea is simple. Most of the difficulty is in the details of how it is written down, and two of those details can break runs silently.
supervised fine-tuning — training on example messages and target responses with the ordinary next-token loss. In this chapter’s setup the loss is computed on the assistant response alone; other setups target other positions too. The same objective as pretraining; a different choice of which text it is applied to. ↩︎
Here is a single Turkish example as it exists in a JSONL file (in JSONL each example is one line):
{"messages": [{"role": "user", "content": "Bu yorum olumlu mu olumsuz mu? 'Kargo çok hızlı geldi, teşekkürler.'"}, {"role": "assistant", "content": "Olumlu."}]}Before the model sees it, that structure is flattened into a single token sequence by the model’s chat template — a string template that in most distributions ships with the tokenizer configuration and differs between model families; training and serving should use the template that matches the model version:
chat template — the exact string format, with roles, message boundaries and special marker tokens, that turns a conversation into model input. In most distributions it ships with the tokenizer configuration, and using the wrong one usually degrades quality quietly rather than raising an error. ↩︎
<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Bu yorum olumlu mu olumsuz mu? 'Kargo çok hızlı geldi, teşekkürler.'<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Olumlu.<|eot_id|>Those <|...|> markers are special tokens with their own embeddings in this model family’s tokenizer, and they mark the boundary between the user’s message and the assistant’s turn. The behaviour of answering rather than continuing is learned from training on examples formatted with them: mechanically, the model learning what should follow the assistant header.
Do not hand-write the template; check its output. Call tokenizer.apply_chat_template(messages, tokenize=False) on a training example and print the result; check that the role markers and the response boundary are where you expect. Model families use different markers, and a template mismatch between training and serving usually degrades quality with no error message anywhere — quietly enough that people can spend weeks blaming the data. It is one of the common ways an otherwise correct-looking fine-tune underperforms.
At inference, if the template expects a separate assistant header, call it with add_generation_prompt=True: that appends the empty assistant header, which is the model’s cue that it is now its turn to answer (training examples do not get it, since the answer is already there). Forget it and even a well-tuned chat model may simply continue the user’s message; check the sequence it produces.
Loss masking: the detail that silently breaks runs
Now the second detail. With Llama 3’s tokenizer the sequence above is 31 tokens long (re-run), and four of them are the answer: Ol, umlu, . and <|eot_id|>. On which tokens do you compute the loss?
If you say all of them — which is what you get if you naively reuse the pretraining objective — then most of the loss terms, 27 positions, teach the model to generate the prompt and the template. Those terms do not have equal-sized gradients, but the target is no longer just the answer: you are pushing the model to model the user’s text as well. For answer-focused SFT, masking the non-assistant target positions is the more direct choice.
Loss masking: which tokens the gradient is allowed to come from
Masking is not about hiding the prompt from the model. The user’s message stays in the input and the model still attends to every prompt token; that is what the response being conditioned on it means. The mask only decides which next-token predictions enter the loss: the target labels of the user message and template markers are -100, those of the assistant response are token ids.
In practice you build a labels tensor that is a copy of input_ids with every non-response position set to -100, the default ignore_index of PyTorch’s cross-entropy loss, which then ignores those positions. Modern SFT libraries do this for you if you tell them where the response starts; the failure mode is reaching for a plain causal-LM training loop and forgetting. Two related details. If your model or trainer shifts the loss internally, do not shift the labels yourself: Hugging Face causal-LM models already compare the prediction at position \(t\) with the label at \(t+1\), and shifting twice trains the model to predict two tokens ahead. Check the alignment on a small example. And in a multi-turn conversation the common choice is to keep the loss on every assistant turn and mask every user turn, not just the last exchange; choose which turns to target for your task.
There is one legitimate reason to train on prompt tokens: if your prompts are themselves in a domain the base model has never seen, the extra signal helps the model learn the language of that domain. It is a deliberate choice, and it should be justified in your run notes rather than inherited from a copied script.
How many examples do you actually need?
The honest answer is that the field’s own numbers span three orders of magnitude. They are figures reported by different projects with different aims, data types and quality bars — not a common scale of need — but the spread itself is informative.
| project | reported SFT input | what they were for |
|---|---|---|
| LIMA (Zhou et al. 2023) | 1,000 | showing that a little is enough |
| InstructGPT (Ouyang et al. 2022) | 12,725 prompts | a general-purpose assistant |
| Self-Instruct (Wang et al. 2023) | ~52,000 instructions | synthetic data; only 175 human-written seed tasks |
| Tülu 3 (Lambert et al. 2025) | 939,344 prompts | a competitive open recipe |
That last row hides the real work. Tülu 3’s team assembled 23,327,961 candidate prompts and kept 939,344 of them for supervised fine-tuning — under one in twenty-four — with a further 425,145 routed to preference tuning (Lambert et al. 2025). The published artefact is a filtering decision far more than it is a collection effort.
LIMA is the interesting entry. Its authors fine-tuned a 65B LLaMa on exactly 1,000 curated prompt-response pairs — 750 harvested from community Q&A sites (Stack Exchange, wikiHow, Reddit), 250 written or adapted by the authors — with no reinforcement learning and no preference modelling at all. Against GPT-4, human raters judged LIMA’s answers equal or better 43% of the time; against Bard 58%, against the RLHF-trained DaVinci003 65% (Zhou et al. 2023).
There is a second reason a small set can go so far, shown in 2021 by the FLAN work, before LIMA: a model fine-tuned on a collection of instruction-phrased tasks improves on tasks it never saw during that fine-tune (Wei et al. 2022). What SFT installs is not a list of answers but the habit of treating an instruction as one.
To explain their own experiments, the LIMA authors propose what they call the Superficial Alignment Hypothesis: essentially all of a model’s knowledge and capability is acquired during pretraining, and alignment mainly teaches it which sub-distribution of formats to use when interacting with users. If that is right, a thousand good examples can be enough to specify a format. The hypothesis does not mean that fine-tuning can teach no facts, or that a thousand examples suffice for every task; you have to measure on yours.
The practical reading. Start with a small, clean set that represents your target inputs and run a baseline; then measure the gain on a separate validation set as you add examples. As a rough intuition: in small sets added examples still buy format, as the set grows they buy coverage of the variety of inputs, and larger still they buy robustness on the long tail. When the gain flattens, choose between collecting more, fixing labels, covering rare cases and widening the evaluation; the marginal hour of quality control is often worth more than the marginal example.
LIMA and Tülu 3 are examples, at two very different scales, of data selection shaping the result: in both, curation mattered as much as collection.
9.4 Parameter-efficient tuning
Everything so far assumed you can run the training step at all. As models grow, a full training step soon stops fitting on the single GPU you are likely to have. The limit is set by the optimizer state, the activations, the sequence and batch length and the GPU memory as much as by model size: in the example setup below, 1B parameters need about 16 GB for parameter-related state alone, before activations. Most of that is not the weights themselves but the extra state that training them brings.
Where the memory actually goes
Count the bytes one parameter costs during mixed-precision AdamW training in a common example setup — bf16 compute weights and gradients, an fp32 master copy and two fp32 Adam moments. Check which copies your implementation actually keeps, and the activations and temporary buffers, separately. Other configurations exist and cost differently: 8-bit optimizer states, fused fp32-free schemes and sharded optimizers all change this table. The 16 is a default, not a law.
| what is stored | bytes per parameter | why it exists |
|---|---|---|
| bf16 weight | 2 | the model itself |
| bf16 gradient | 2 | one per trainable parameter |
| fp32 master weight | 4 | bf16 has too few mantissa bits to accumulate tiny updates |
| fp32 Adam \(m\) | 4 | running mean of the gradient |
| fp32 Adam \(v\) | 4 | running mean of the gradient squared |
| total, trainable | 16 | |
| total, frozen | 2 | a weight you never update only has to be read |
The fp32 master copy is the least obvious row and the one people try to delete first. Without it, an update of size \(10^{-5}\) added to a weight of size \(1\) rounds straight back to \(1\) in bf16 — while fp32 keeps it, as 1.00001 — and the training silently stalls. Pick the example carefully: \(10^{-8}\) would vanish in fp32 as well, since single precision has about seven decimal digits. In this setup it is the fp32 master copy that accumulates the small updates. That does not make an fp32 copy mandatory for every low-precision run: other optimizer and update schemes solve the problem differently, and then the memory arithmetic changes too.
Read the two totals against each other. In this example setup the model itself is one eighth of the parameter bill. The fp32 rows — the master copy and Adam’s two moments — are three quarters of it, and the gradient is the remaining eighth. That ratio is where parameter-efficient fine-tuning (PEFT) gets its saving on parameter state: freeze a parameter and seven of its eight instalments disappear, leaving only the bf16 weight you still have to read. Activations and compute remain; do not carry this saving over to total GPU memory as a factor of 8 (the lab in Section 9.11 measured 1.40×).
parameter-efficient fine-tuning (PEFT) — the family LoRA belongs to: train a small fraction of the weights (or a few added ones) and leave the rest frozen. Usually written PEFT. ↩︎
The arithmetic is checkable against a published figure. QLoRA states that 16-bit fine-tuning of a 65B LLaMA “requires more than 780 GB of GPU memory” (Dettmers et al. 2023). One example calculation that explains its size:
\[ 65 \times 10^9 \;\text{params} \times 12 \;\text{bytes} = 780 \;\text{GB} \]
Twelve bytes per parameter — and the paper does not itemise its memory, so this product is a possible decomposition, not the paper’s own. One reading against the table above is the three fp32 rows, master weight, \(m\) and \(v\); in the 16-byte setup the bf16 weight and gradient bring parameter-related state to \(65 \times 16 = 1{,}040\) GB, before activations and buffers, consistent with “more than”. A second reading fits the same twelve exactly: bf16 weight, bf16 gradient, fp32 \(m\), fp32 \(v\), no master copy, with “more than” absorbing activations instead. In either reading most of the headline number is state that exists only because the parameter is being trained.
Reconstructing a published memory figure is worth doing precisely because it usually has more than one solution. Papers report totals, not itemisations; you infer the itemisation from the total, and honesty requires saying which one you assumed.
The LoRA paper’s figure moves in the other direction: on GPT-3 175B, training VRAM falls “from 1.2TB to 350GB” (Hu et al. 2022). The endpoint is clean —
\[ 175 \times 10^9 \times 2\;\text{bytes} = 350\;\text{GB} \]
is the raw size of the frozen bf16 weights. The starting point is less clean. 1.2 TB across 175B parameters is about 6.9 bytes each, well under the 16 the table predicts, so that figure was measured under a setup the paper does not spell out. Either cheaper per-parameter state (8-bit or 16-bit optimizer moments, no fp32 master copy) or keeping part of the state off the GPU (CPU offloading, for instance) could explain it; the paper does not say which. Sharding across many cards and activation checkpointing do not explain it on their own: sharding spreads the same bytes over more GPUs, and checkpointing trims activations, not the per-parameter state. The 350 GB, too, matches the raw size of the frozen bf16 weights; the adapter and activations also take room, so the equality does not account for every item in the paper. Quote both figures as their report: you can recompute what the 350 GB corresponds to, but you cannot recover the 1.2 TB’s itemisation from the two totals.
The low-rank hypothesis
So we want to freeze the weights. But a frozen model does not learn. The question is what you can add on the side that is small enough to be cheap and expressive enough to be useful.
LoRA’s answer starts from an assumption about what fine-tuning does. A pretrained weight matrix \(W_0\) is a general-purpose feature map, built from a very wide range of text. The change you need for one task can plausibly be much narrower. The hypothesis (Hu et al. 2022) is that this adjustment \(\Delta W\), while it is a full \(d \times k\) matrix in shape, can be approximated well enough by a low-rank product — it has low intrinsic rank. That is not a guarantee that the full fine-tuning update is low-rank in every task and every layer; it is a constraint LoRA finds useful.
If that is true, you can write it as a product of two thin matrices:
\[ W_0 + \Delta W = W_0 + BA, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k}, \qquad r \ll \min(d,k) \]
LoRA — low-rank adaptation: freezing a pretrained weight matrix and learning a small pair of matrices whose product is added to it. The pair has \(r(d+k)\) parameters instead of \(dk\). In a compatible serving setup the update can be folded back into the original weight, removing the extra matrix multiply; measure the latency of an unmerged or quantized model separately. ↩︎
Count what that saves. For one of Llama 3.1 8B’s \(4096 \times 4096\) query projections, \(\Delta W\) in full has \(16{,}777{,}216\) parameters. At \(r = 8\), \(BA\) has \(8 \times (4096 + 4096) = 65{,}536\) — 256 times fewer, for the same output shape.
What \(A\) and \(B\) actually hold
The two matrices are not symmetric in role, and the asymmetry is the part worth understanding.
The frozen path and the trainable detour
Follow the detour from \(x\) to \(h\): \(A\) projects the input down to an \(r\)-dimensional space, and \(B\) projects those \(r\) numbers back up to the output size. On an intuitive reading \(A\) picks which few directions of the input are used and \(B\) what to write into the output from them. But that is a reading of the computation; we cannot give the learned \(A\) and \(B\) a unique meaning each, because different factor pairs give the same product: for any invertible \(R\), \((BR^{-1})(RA) = BA\).
The initialisation, too, makes sense once you ask what happens otherwise. For \(h = W_0x + BAx\), \(\partial\mathcal{L}/\partial B\) is proportional to \(Ax\) and \(\partial\mathcal{L}/\partial A\) to \(B\) (given a non-zero upstream loss gradient):
| \(A\) init | \(B\) init | what happens |
|---|---|---|
| zero | zero | \(BA = 0\) and both gradients are zero: gradient descent cannot move the pair off zero |
| random | random | \(BA\) is generally \(\neq 0\) at step 0, so training starts by changing the pretrained model |
| random | zero | \(BA = 0\) at step 0, \(\partial \mathcal{L} / \partial B\) generally \(\neq 0\): the run starts exactly at the pretrained model and \(B\) moves first |
| zero | random | also \(BA = 0\) at step 0, \(\partial \mathcal{L} / \partial A\) generally \(\neq 0\): also escapes zero, with \(A\) moving first |
Two of the four work, not one. Both of the bottom rows start the adapted model exactly at the pretrained model and have somewhere to go, because the gradient with respect to each matrix is proportional to the other one — so you only need one of the pair to be non-zero. For the starting output, what matters is the product, not which factor carries the randomness. The training trajectory need not be the same for the two choices: gradient and optimizer dynamics can differ.
LoRA takes the third row: “a random Gaussian initialization for \(A\) and zero for \(B\)” (Hu et al. 2022). The paper states the convention and does not argue for it over the fourth row, and neither will this book. What the \(B = 0\) initialisation buys is that the adapter leaves the pretrained model untouched at step 0: you start from a model that already works, where random–random initialisation would start the run away from the pretrained model.
Rank and alpha, and why there are two knobs
The update is not added raw. It is scaled by \(\alpha / r\), where \(\alpha\) is a constant you choose (Hu et al. 2022):
\[ h = W_0 x + \frac{\alpha}{r} B A x \]
The stated purpose of the division is practical rather than theoretical. The authors write that the scaling “helps to reduce the need to retune hyperparameters when we vary \(r\)” (Hu et al. 2022) — the aim is to stop a rank sweep from silently becoming a learning-rate sweep at the same time. That does not mean gradient size or the best learning rate stays the same across ranks (Kalajdzievski (2023) shows that under \(1/r\) scaling gradients shrink at high rank); do the rank sweep on validation data.
\(\alpha\) is a fixed multiplier on the whole adapter path, and the paper says how it sees it: “tuning \(\alpha\) is roughly the same as tuning the learning rate” (Hu et al. 2022). Roughly: \(\alpha\) scales the adapter’s contribution in the forward pass and interacts with the learning rate without being identical to it; its best value has to be judged together with the learning rate, the rank and the target modules. The widespread convention \(\alpha = 2r\) means, in the forward pass, “scale the adapter by 2”.
That convention is the community’s, not the authors’. They report setting \(\alpha\) “to the first \(r\) we try” and then leaving it alone. In other words, they did not tune it. Note also that this is not \(\alpha = r\) at every rank: if you fix \(\alpha\) at 8 because you tried \(r = 8\) first, then \(r = 16\) runs at \(\alpha/r = 0.5\). Treat \(\alpha = 2r\) as a default someone else picked for you, not as a result.
Two knobs, two questions; tune both by measuring.
- \(r\) sets the adapter’s parameter count and the rank the update may have: how much capacity does this adaptation need? When the task is far from pretraining (a new language, a new notation), a higher rank is worth trying.
- \(\alpha/r\) scales the adapter’s contribution: how hard should the adapter push?
Underfitting or confident nonsense can come not only from rank or alpha but from the data, the learning rate, the number of epochs and the target modules. Try a small rank/scale grid at fixed data and learning rate, and judge the results both on task performance and on regression against the starting model. They are commonly conflated because \(\alpha = 2r\) makes them move together; break the convention when you sweep.
The arithmetic of an adapter
A LoRA adapter on a \(d_{\text{in}} \times d_{\text{out}}\) matrix adds exactly \(r(d_{\text{in}} + d_{\text{out}})\) adapter parameters. Everything else is bookkeeping: which matrices you attach to, and how many layers there are. The total trainable count can also include biases, a classification head or other modules left trainable; the table below counts adapters only (the lab table adds the head).
The three configurations below were computed from each model’s own config.json (we recomputed every row):
| model | targets | \(r\) | trainable | share of model |
|---|---|---|---|---|
| BERTurk base (110.6M, 12 layers) | query, value | 8 | 294,912 | 0.267% |
| + key, output | 8 | 589,824 | 0.533% | |
| + the MLP | 8 | 1,327,104 | 1.200% | |
| Llama 3.1 8B (32 layers, \(d\)=4096) | q_proj, v_proj | 8 | 3,407,872 | 0.042% |
| all attention | 8 | 6,815,744 | 0.085% | |
| attention + MLP | 8 | 20,971,520 | 0.261% | |
| Qwen 2.5 7B (28 layers, \(d\)=3584) | q_proj, v_proj | 8 | 2,523,136 | 0.033% |
| attention + MLP | 8 | 20,185,088 | 0.265% |
LoRA is not the only parameter-efficient method; it is one of the most widely used today. Prompt tuning (Lester et al. 2021) trains the representations of a handful of virtual tokens prepended to the input and leaves the base model’s weights untouched; in its authors’ experiments it becomes competitive with full fine-tuning as the model grows. The memory, quality and latency trade-offs of these methods vary with model size and task.
Qwen’s q_proj, v_proj adapters come to 2.52M against Llama’s 3.41M, and it is tempting to credit grouped-query attention (Section 7.6) — Qwen 2.5 7B uses 4 key-value heads to Llama’s 8, making v_proj a \(3584 \times 512\) matrix rather than \(4096 \times 1024\). Give Qwen 8 KV heads and the count rises to 2.64M, so GQA is only about 13% of the 884,736-parameter gap. The rest is \(d = 3584\) against 4096, over 28 layers against 32.
Two things fall out. First, adding the MLP matrices to an attention adapter is the largest jump on the table: 2.25× on BERTurk, 3.08× on Llama (from q_proj, v_proj alone, 4.5× on BERTurk, 6.2× on Llama, 8.0× on Qwen). The jump comes from the number and size of the target matrices; separately, the MLP does hold much of these models’ weights (by our count ~70% of Llama 3.1 8B, ~75% of Qwen 2.5 7B, ~51% of BERTurk). Second, even the most generous setting here is under 0.3% of an 8B model. Raising \(r\) from 1 to 128 multiplies the adapter count on the same targets by 128 — no one should expect quality to rise in proportion — yet even the top of that range stays well under one percent of the model:
| \(r\) | 1 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|
| trainable (q,v only) | 425,984 | 1.70M | 3.41M | 6.82M | 13.6M | 27.3M | 54.5M |
| share of Llama 3.1 8B | 0.005% | 0.021% | 0.042% | 0.085% | 0.170% | 0.340% | 0.679% |
Even at \(r = 128\) you are training two thirds of one percent of the model.
Work out your own run
The arithmetic above is for one configuration. Yours will differ, and the numbers move faster than intuition does — so rather than trusting a rule of thumb, put your own model size and rank in. The widget gives an approximate subtotal of weights, gradients and optimizer state for the parameter count and storage scheme you choose; the GPU memory a run needs also depends on activations, sequence and batch length, temporary buffers and the implementation. Do not use the result as a guarantee that a run fits.
The defaults are Llama 3.1 8B (8.03B parameters) with an \(r{=}16\) adapter on attention and MLP — 41,943,040 parameters, or 0.52% of the model. The widget’s static-state-only sums: full fine-tune 119.7 GiB, past every common 24–80 GB single-GPU configuration; LoRA on a bf16 base 15.5 GiB; QLoRA on a raw 4-bit base 4.3 GiB. Without activations, temporary buffers and QLoRA’s scaling metadata you cannot say which card they fit; the caution below shows why that matters.
What this widget leaves out is the part that will surprise you. It counts weights, gradients and optimizer state, because those are what LoRA removes. It does not count activations, and once the optimizer is gone they are what is left. QLoRA’s authors measured it: for a 7B LLaMA at batch size 1, with an adapter at “commonly used 0.2% of the original model weights”, the LoRA input gradients occupy 567 MB while the LoRA parameters occupy 26 MB; with gradient checkpointing the first figure drops to an average of 18 MB per sequence (Dettmers et al. 2023). In that example the LoRA input gradients take about twenty-two times the room of the adapter you are training. Those values belong to that model, batch size and sequence length; they do not carry over to another setup.
This is why gradient checkpointing appears in common LoRA recipes, and why the widget’s numbers are a subtotal rather than an estimate. Before choosing a card, run a short pilot on the target setup and measure peak memory. It is also why the lab at the end of this chapter measures a much less dramatic ratio than the arithmetic here predicts.
QLoRA: quantize the part you are not training
That last step needs one more idea. In the LoRA row above, the 15.5 GiB static subtotal is now almost entirely the frozen bf16 base — the adapter’s optimizer state is a rounding error. So squeeze the base.
QLoRA (Dettmers et al. 2023) does exactly that, storing frozen weights in a 4-bit format called NF4 and dequantizing each block back to bf16 only for the moment it is needed in the forward and backward pass. Three pieces make it work:
- 4-bit NormalFloat, a data type whose 16 levels are placed at the quantiles of a normal distribution — chosen because trained network weights are close to normally distributed, so the levels land where the weights actually are;
- double quantization, which quantizes the per-block scaling constants themselves, because at small block sizes those constants become a non-trivial share of the total;
- paged optimizers, which, on supported unified-memory setups, let optimizer state spill to CPU memory during memory spikes and so manage some of them; the gain and the speed cost have to be measured on the implementation.
The headline result: fine-tuning a 65B model goes from “>780 GB of GPU memory to <48 GB without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline” (Dettmers et al. 2023) — a result reported for the paper’s own task comparisons. Their Guanaco 65B, trained this way in 24 hours on a single GPU, reached 99.3% of ChatGPT’s score on the 80-question Vicuna benchmark, as scored by GPT-4 acting as judge rather than by humans; that ratio is not a general accuracy or human-preference measure (on the limits of LLM judges see Section 22.4).
“Without degrading” is measured against full fine-tuning, not plain LoRA. The paper’s claim covers runtime too, but its comparison is a 16-bit full fine-tune. Against plain LoRA, dequantizing blocks on every forward and backward pass costs arithmetic; on the other hand, lower memory use can ease other bottlenecks in some setups. QLoRA buys you the ability to run at all on a small card; on a card where plain LoRA already fits, measure the step time of both on the same hardware and batch.
The payoff nobody mentions until they are on call
LoRA has an operational property that matters more, in a company, than any of the training savings: an adapter can be merged back into the base weights. Since \(h = W_0x + \frac{\alpha}{r}BAx = (W_0 + \frac{\alpha}{r}BA)x\), you can compute that sum once, in a suitable linear layer, and ship an ordinary model; the separate adapter multiply goes away — the paper’s “no additional inference latency” (Hu et al. 2022). Real latency still depends on the serving stack, precision and quantization; measure it. It is a significant advantage over bottleneck adapters (Houlsby et al. 2019), which insert extra layers.
Or you can not merge, keep the base shared in memory, and swap adapters per request:
| tenants | one shared base + adapters | a separate 8B copy each | ratio |
|---|---|---|---|
| 10 | 15.7 GiB | 149.6 GiB | 9.5× |
| 100 | 22.8 GiB | 1,495.7 GiB | 65.7× |
| 1,000 | 93.1 GiB | 14,957.0 GiB | 160.7× |
The adapter size those rows assume is an \(r{=}16\) adapter on attention and MLP, saved in bf16: \(41{,}943{,}040 \times 2\) bytes, exactly 80 MiB. Change either choice and the figure moves a long way — q_proj, v_proj only at the same rank is 6.82M parameters, 13 MiB, and peft will happily save the same adapter in fp32 and double whatever you expected. Always state targets and dtype when you quote an adapter size; on its own the rank tells you almost nothing.
Even at the generous end, the weights of a thousand customers’ private fine-tunes — one bf16 base and a thousand adapters — come to 93.1 GiB. That is a weight-storage sum; it does not show that one node can serve a thousand tenants at the concurrency and latency you need: KV caches, adapter loading, workspace and traffic have to be budgeted separately. Still, it is one reason “fine-tuning” can be a product at all, rather than a research technique — and it is worth noticing that the paper made the same point about GPT-3 in 2021: 100 adapted models cost 350 GB + 35 MB × 100 ≈ 354 GB, against 35 TB for 100 separate copies (Hu et al. 2022).
9.5 Preference and reward optimization
The thing supervised fine-tuning cannot do
Take a question with no single right answer:
Kredi kartı borcumu nasıl kapatabilirim?
(How can I pay off my credit card debt?)There is no single required answer. There are a thousand acceptable answers and they differ along axes you can recognise but find hard to write down as rules: too curt, too long, too confident about your finances, gives advice it should have refused to give, correct but reads like a pamphlet. A chosen, safe and helpful answer can still be written as an SFT target. The real distinction is the kind of supervision you have: if you can write acceptable example answers, SFT fits; if you can judge the relative quality of two candidates more reliably, preference pairs add a signal.
But you can still do something: show two candidate answers to a person and ask which is better. The result is a preference pair. For some tasks producing one is an easier judgement than writing the ideal answer, and one you can collect from people who could not have written either answer themselves; but labellers need clear criteria, expertise where the task needs it, and a check on disagreement.
preference pair — one prompt, two candidate answers, and a label saying which was preferred. On its own a binary preference does not measure either answer’s absolute adequacy; both may be poor. ↩︎
Preference optimization turns the relative order of two candidates directly into a training signal (Christiano et al. 2017). The chosen answers can be used for SFT too; the difference is how the comparison, and the rejected answer’s information, enter the loss.
RLHF: the four-model dance
The original recipe, reinforcement learning from human feedback (RLHF) (Stiennon et al. 2020; Ouyang et al. 2022), turns those comparisons into a training signal in two steps: fit a reward model to the preferences, then optimize the language model against it.
Two words from reinforcement learning appear from here on. The policy is simply the model being trained, seen as defining a probability distribution over possible answers (token by token) given a prompt. The KL divergence \(\mathbb{D}_{\text{KL}}[\pi \,\|\, \pi_{\text{ref}}]\) measures how far one probability distribution has moved from another; it is zero when the two are identical and grows as they separate. Here it compares the policy’s answer distribution with the frozen reference’s, averaged over prompts in the training objective.
reward model — a model trained on human comparisons that reads a prompt and an answer and returns one number. It exists to turn A is better than B into something an optimizer can maximise. ↩︎
RLHF, and the components you have to budget for
The reward model is the interesting object: it converts an ordering — A beat B — into a scalar, which is what a reinforcement-learning algorithm needs. The diagram shows the four functions of InstructGPT-style PPO-based RLHF: the policy being updated, a frozen reference, the reward model and a value component for advantage estimation. They need not be separate networks, nor held on one GPU at once (the value estimator can share the trunk, components can sit on different devices); the memory cost depends on the implementation. The reinforcement-learning algorithm in the middle is PPO (Schulman et al. 2017), one of the methods used to update the policy.
The result was not marginal. On the prompts studied, human labellers preferred the outputs of InstructGPT’s 1.3B model, put through this pipeline, to those of the 175B GPT-3 — “despite having 100x fewer parameters” (Ouyang et al. 2022). That is a preference for particular answers, not a general capability advantage. The cost figure is separate and belongs to the study’s 175B version: training the 175B SFT model took 4.9 petaflop/s-days and the 175B PPO-ptx model 60, “compared to 3,640 petaflops/s-days for GPT-3” (Ouyang et al. 2022). The reported SFT and PPO-ptx runs together cost about 1.8% of what pretraining cost, and that total leaves out the reward model.
That comparison is one of the strongest arguments in this chapter for taking post-training seriously. It is also a good argument against believing that parameter count alone tells you what a model can do.
DPO: deriving the shortcut away
Look again at the diagram and ask which box is doing indispensable work. The reward model exists only to score samples during PPO. If the policy’s probabilities, read as a log-ratio against the reference, could serve as a reward, the box would not need training separately.
In the KL-regularized preference objective, they can. This is the direct preference optimization result (Rafailov et al. 2023): it re-expresses the reward through the log-ratio of policy and reference, so training on preference pairs needs no separate reward network and no PPO sampling loop. The preference data and the reward-modelling assumption do not go away, and the derivation is worth stating because it is short and the conclusion is otherwise mystifying.
direct preference optimization — training the policy on preference pairs with a loss derived from reparameterising the KL-regularized reward objective. It needs no separate reward network and no sampling during training; how well it works depends on its assumptions and on how well the data fit. Usually written DPO. ↩︎
Before the shortcut, the two objectives it is a shortcut around, because the derivation below leans on both. First, the reward model is fitted to the comparisons with the Bradley–Terry likelihood — the assumption that a person prefers \(y_w\) over \(y_l\) with probability \(\sigma(r(x,y_w) - r(x,y_l))\). Here \(y_w\) is the preferred (winning) answer, \(y_l\) the rejected (losing) one, and \(\sigma\) the logistic function. If you know chess ratings, this is the same model as Elo: only the difference between two scores decides who is expected to win.
\[ \mathcal{L}_{\text{RM}} = -\,\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\!\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right)\right] \]
Only differences of \(r_\phi\) within a prompt are constrained, so rewards are identified only up to an additive function of \(x\): add the same \(f(x)\) to every response and nothing changes. Do not read that as “the scale is arbitrary”. Doubling every reward does change the model — \(\sigma(1) = 0.731\) against \(\sigma(2) = 0.881\) — unless you rescale the link temperature to match. The additive freedom is what makes the \(Z(x)\) term cancel a few lines from now, because the offset is the same for both responses to one prompt.
Second, the policy is optimised against that frozen reward, leashed to the reference:
\[ \max_{\pi_\theta}\; \mathbb{E}_{x \sim \mathcal{D}}\Big[\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x, y)\big] \;-\; \beta\, \mathbb{D}_{\text{KL}}\!\left[\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\right]\Big] \]
The number \(\beta\) is the strength of the leash. A large \(\beta\) keeps the policy close to the reference and gives up some reward; a small \(\beta\) lets it chase the reward further — and, with a learned reward model, closer to reward hacking.
PPO is one way of taking a step on that objective: it maximises a clipped surrogate so that a single batch cannot move the policy far (Schulman et al. 2017). That clip range is one of the hyperparameters the table below charges to PPO’s account.
That objective — maximise reward, penalised by KL divergence from the reference — has a known closed-form optimum:
\[ \pi^{*}(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \, \exp\!\left( \frac{1}{\beta} r(x, y) \right) \]
Rearrange for \(r\):
\[ r(x, y) = \beta \log \frac{\pi^{*}(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x) \]
In this derivation the reward is represented, up to a term that depends only on the prompt, as the log-ratio of policy and reference. That does not make learned reward models pointless; it means that, under these assumptions, the policy can be updated on preference pairs without training a separate reward model. And the awkward \(Z(x)\) — a sum over every possible response, which is why nobody computes it — depends only on the prompt, so it cancels the moment you compare two responses to the same prompt. Which is precisely what preference data gives you.
Substituting into the Bradley–Terry model of pairwise preference leaves an ordinary supervised loss over preference pairs \((x, y_w, y_l)\):
\[ \mathcal{L}_{\text{DPO}} = -\,\mathbb{E}\left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] \]
Read it in words and it stops being intimidating: make the preferred answer’s log-probability ratio against the reference larger than the rejected answer’s, so you are rewarded for moving in the right direction rather than for agreeing with the reference. The loss does not fix the direction of either probability on its own.
One precision on “raise” and “lower”. The loss only asks for the gap between the two log-ratios to widen. In practice both probabilities often fall during DPO, the rejected one faster, so do not be surprised if the preferred answer’s own likelihood goes down while the loss improves.
Notice what \(\pi_{\text{ref}}\) is doing here, because it is the reason for the “avoid DPO on a base model” advice you will meet later.
DPO’s setup assumes the preference pairs were sampled from the reference. So the paper sets \(\pi_{\text{ref}} = \pi_{\text{SFT}}\) whenever an SFT model exists — and when one does not, it recommends fine-tuning on the preferred completions first and using that, explicitly “to mitigate the distribution shift between the true reference distribution which is unavailable, and \(\pi_{\text{ref}}\) used by DPO” (Rafailov et al. 2023).
Which locates one likely failure. The loss never samples anything; if the preference pairs come from an instruction-following model, a base model is a weak reference, and the problem is the mismatch between reference and data, not an inability to “rank”. DPO on a base model is mathematically well defined; whether it works depends on data coverage, the choice of reference and the task evaluation.
What DPO did not remove. You still need reliable preference labels and an evaluation of their coverage; labelling can be done by people, by a model or by a verifier, and its cost depends on the method. DPO makes the algorithm cheap, not the data. It also inherits a subtler problem: if the pairs were collected once, from some other model’s outputs, they can represent your policy less well as it moves away from whatever generated them. In the Tülu 3 study, on-policy preference data — generated by the model being trained — improved downstream results over reused off-policy pairs (Lambert et al. 2025); do not assume the same gain for every corpus.
GRPO: delete the value model instead
DPO deletes the reward model. GRPO (Shao et al. 2024) takes the other route: keep reinforcement learning, but delete the value model.
The value model exists to supply a baseline. In policy-gradient methods you do not want the raw reward — you want to know whether an action was better than expected, because a reward of 8 means something different when your usual score is 2 than when it is 9. Training an estimator of that expectation is the standard answer, and it costs something: a separate model or a head sharing the policy’s trunk, with a memory cost that depends on the training setup.
GRPO’s observation is that if you are going to sample from the policy anyway, you can get a baseline without training a separate value estimator — though generating and scoring several answers per prompt has its own compute cost. Sample \(G\) answers to the same prompt, score them all, and use the group’s own mean as the expectation:
\[ A_i = \frac{r_i - \operatorname{mean}(r_1, \ldots, r_G)}{\operatorname{std}(r_1, \ldots, r_G)} \]
\(A_i\) is called the advantage of answer \(i\): positive means “better than this group’s average”, and the update makes that answer more likely; negative makes it less likely. Dividing by the standard deviation puts every prompt on the same scale, so easy and hard prompts pull with comparable force. One consequence to remember: if all \(G\) answers get the same reward — all right or all wrong — every \(A_i\) is zero (implementations add a tiny constant to avoid dividing by zero, or skip the group), and that prompt gives no signal from the reward comparison; other terms such as the KL can still act.
The question stops being how good is this answer in absolute terms and becomes was this attempt better than my other attempts at the same question — a relative signal that needs no separate value model. How stable it is depends on \(G\), the reward distribution and the normalisation; a small \(G\) can bring high variance. GRPO still keeps the frozen reference and the KL leash, and it still needs some reward — a reward model or, as the next section shows, a checker. In the paper’s words (critic is another name for the value model), GRPO “foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources” (Shao et al. 2024). On DeepSeekMath-Instruct 7B it moved GSM8K from 82.9% to 88.2% and MATH from 46.8% to 51.7%, using only instruction-tuning data already in hand — a result of that setup, not something to credit to the group mean alone.
Where the reward comes from, and why that is the real question
There is one more move, and it is among the ones that have shaped the field most recently. Most of the methods above assumed the reward is learned — from human comparisons, so it inherits every flaw of a learned model, including being gameable.
For some tasks it does not have to be. If the answer is a number, check the number. If it is code, run the tests. DeepSeek-R1-Zero — the variant trained by reinforcement learning with no supervised warm-up — uses “rule-based rewards” of two kinds: accuracy rewards, which evaluate whether the final answer is correct on tasks such as maths and code (the answer is asked for in a specified format so it can be checked), and format rewards, which check that the reasoning was placed where it was asked to be (Guo et al. 2025). Tülu 3 builds the same idea into an open recipe under the name RLVR, reinforcement learning with verifiable rewards (Lambert et al. 2025).
This is also the sharpest limitation. Verifiable rewards need a verifier, and most things people want from a language model — was this summary faithful, was this tone appropriate, was this refusal correct — have no verifier. Maths and code are unusually lucky domains. Chapter 10 follows what happens when you push this idea as far as it goes.
The trade, roughly: a learned reward model can approximately judge open-ended qualities but is limited by the coverage of its labels and can be gamed; a rule-based verifier gives a strong signal on the narrow set of tasks whose correctness a criterion can check, and is usually harder to game — but only as far as its specification and implementation go. A test suite with gaps rewards a program that hard-codes the visible cases; a loose answer parser rewards whatever matches its regex (Chapter 10 has the version of this that bites in practice). Because the two kinds of reward fail in different ways, many recipes use both, on different subsets of the data; which to use is decided by the verifiable part of the task and the behaviour you want.
Choosing, by what data you have
Strip the acronyms and every method above is a claim about what you can collect.
| you have | worth trying | what it costs you, what else it needs |
|---|---|---|
| prompts with ideal answers written out | SFT | writing the answers; a task evaluation |
| pairs of answers with a “which is better” label (from people, a model or a verifier) | DPO | annotation; a matching reference and label coverage |
| a preference/reward signal and a budget for online RL | RLHF/PPO | a generation loop, reward and value computation; hard to tune |
| prompts whose answers can be scored | GRPO (often with verifiable rewards) | a reward function, and a lot of sampling |
| a stronger model whose outputs you may use | distillation (Section 9.7) | inference bills, licence and quality measurement |
The order to try things in, for a team of one or two. First set up a baseline with prompting, an output schema and, if needed, retrieval (Section 9.9). If you have examples of the behaviour, try SFT. Compare DPO when you have a specific complaint about the SFT model’s behaviour that you can express as pairs — “it should have refused this”, “it should have been shorter”. Consider an RL loop only if it buys a measurable gain on the target task in exchange for the extra generation and reward-evaluation compute; the reward can come from a verifier or a vetted reward model, but with the latter you sign up to maintain a reward model as well as a policy.
9.6 Constitutional AI and feedback that isn’t human
Where rule-based rewards do not apply, the bottleneck is often annotation. Constitutional AI (Bai et al. 2022) studies how a model can produce harmlessness feedback with less dependence on human labels.
The method has two halves. In the supervised half, the model answers a red-team prompt, then critiques its own answer against a written principle, then revises it; the revision becomes the training target. In the RL half, the same principles are used to label preference pairs for harmlessness, replacing the human annotator there — reinforcement learning from AI feedback, RLAIF. (In the paper, the helpfulness comparisons were still labelled by people.)
Anthropic’s team wrote 16 principles for the harmlessness experiments and sampled one at random at each critique-and-revision step (Bai et al. 2022). The randomisation was not for fairness: they found that ensembling over principles produced notably more robust preference-model behaviour than using one principle for every label.
Two things about this are worth separating, because they are usually praised together and only one of them is unambiguous.
What is clearly good: the principles are an artefact. They are a text file. You can read them, argue with them, version them, and hand them to a regulator. Human preference data can also have guidelines and quality control behind it, but when those are unpublished the values can only be recovered by probing the finished model. Moving the specification into a document helps accountability, independent of whether the technique works better; a written principle still does not prove on its own that behaviour follows it.
What is not settled: AI feedback can carry the evaluator model’s errors and biases into the training signal. The evaluator is often the same model as the one being trained, or a close relative, and then shared blind spots need particular scrutiny. Where it cannot tell that an answer is subtly wrong, the training signal is confidently wrong, and scale alone does not fix a bias shared by both sides of the loop. A reasonable expectation is that AI feedback is more reliable on properties a model can check (is this rude, does this contain the requested section, does this refuse) and less reliable where checking needs domain expertise; but measure that per criterion against human- and expert-checked examples.
9.7 Distillation into something you can afford
Everything so far has made a model better. Knowledge distillation (Hinton et al. 2015) aims to transfer a large teacher’s behaviour to a smaller student, which can be cheaper to serve. Measure the overall gain together with the cost of generating the teacher data and training the student.
knowledge distillation — training a small student on a teacher’s probability distribution or on selected outputs. What the student learns depends on the data, its capacity and the evaluation; inheriting the teacher’s mistakes is a risk, which filtering the outputs can reduce. ↩︎
The setup: a large teacher produces outputs, a small student is trained to reproduce them. In the classical form the student matches the teacher’s full output distribution — its “soft targets” — which can carry more information per example than a hard label does. Knowing that the teacher put 70% on positive, 25% on neutral, and 5% on negative tells the student that this example is borderline; a one-hot label of positive does not. How useful that is depends on the teacher’s calibration and the task.
For generative models the modern form is blunter and works well: have the teacher answer a large set of prompts, filter the answers, and fine-tune the student on them as ordinary SFT data. DeepSeek-R1’s authors did exactly this, using 800,000 curated samples from their large model to train small dense models (Guo et al. 2025).
Check the terms before you use a teacher’s outputs. Hosted services’ terms of use and open-weight models’ licences can set different conditions on using outputs to train another model: some restrict it, some permit it, some require attribution in the derived model’s name (the Llama 3.1 licence is one, Section 8.6.2). This is a contract question, not a technical one: read the terms of the specific version of the model, service and dataset you will use, and get your organisation’s legal review if needed.
9.8 Continued pretraining, and the Turkish case
There is a situation where the “post-training is not a reliable way to add knowledge” principle bends: when what is missing is not a fact but a distribution. A model that has seen little Turkish is not missing one fact about Turkish; it is missing the patterns of the language. But if a model does poorly in Turkish, first measure why on the target tasks: the data mix, the tokenizer, model size and instruction tuning each have their own effect.
The tool for that is continued pretraining: take the base model and keep running stage 1 on a domain corpus, with the same next-token objective and no instruction format at all (Gururangan et al. 2020). It sits between the stages — pretraining’s objective, with a data and compute need that depends on the corpus and the goal rather than being fixed in advance.
continued pretraining — running the pretraining objective again on a narrower corpus, starting from an existing base model. Used when the gap is a whole domain or language rather than a task or a format. ↩︎
For Turkish you have three options, and Section 8.4 already measured the token side of one of them for you:
| route | possible benefit | risk and cost to measure |
|---|---|---|
| adapt a multilingual model to the task | fastest path; builds on the ready model’s existing abilities | Turkish task quality; Turkish token counts under that model’s tokenizer; adaptation compute |
| continued pretraining on a Turkish corpus, then task adaptation | may strengthen language and domain patterns | corpus and training cost (a real, if small, pretraining run); regression in general abilities |
| start from a Turkish-native model | fewer tokens on target text (lower fertility) and possibly better-suited representations | task quality, licence, size and serving cost; available Turkish-native models are mostly small |
The token side is measurable, and we measured it in Chapter 8: on the same Turkish text Llama 3’s tokenizer produces 2.12 tokens per word against 1.59 for BERTurk’s; a token-count difference of about 33%. They belong to different models; you cannot put BERTurk’s tokenizer on Llama 3. So the difference is not an automatic 33% bill or latency increase for one model but a token gap between two candidates; decide with each candidate’s own tokenizer, task quality and real price. (This is a different comparison from the 23% in Section 8.4: there we compared the same content in Turkish and English, here the same Turkish text under two tokenizers.) For a concrete map of Turkish-first models and for deciding between multilingual and monolingual with a measurement, see Section 16.6 and Section 16.7.
Compare by task. For classification or extraction on short texts, compare Turkish-first and multilingual candidates on the same held-out data for quality, latency and cost; Section 9.11 shows a 110M-parameter Turkish-first model doing well on one task, but only that task. If the task needs broad knowledge or reasoning, add tests for it; the Turkish-first models you can afford may lag multilingual ones there. Consider continued pretraining with a pilot run if you have enough Turkish text and a measured quality gap, and measure regression in existing abilities too.
9.9 When not to fine-tune
Most of this chapter has been about how. This section is about whether, and it is the part to remember if you remember one part.
There is a common failure mode. A team has a model that gets things wrong about their business. Fine-tuning is the technique that sounds like “teach the model about our business”. They collect examples, run a LoRA job, and the model can come back more fluent and still wrong — because the examples taught it the style of their documentation without giving it the current contents in a form it can reliably recall. First test where the wrong information comes from and how the right source can be put in front of the model.
Which technique, and why — walk it top to bottom
The tree starts from the options with low setup cost and separates kinds of problem, and every exit before question 4 is a way of not running a training job; the branches are not a universal price ranking. A fine-tune needs maintenance together with its training data and a regression evaluation: when the base model updates it has to be re-evaluated and often retrained. Prompts, retrieval and schemas also need re-testing when the model version changes, but that upkeep is usually lighter.
Branch 2 is not a formality. “Have you tried a better prompt” is a question with a large literature behind it — one survey catalogues 58 distinct text-only prompting techniques (Schulhoff et al. 2025), and Chapter 17 works through the ones that matter. Teams who reach for fine-tuning have often tried only a few.
The two rules the tree encodes.
- Retrieval first for knowledge, fine-tuning if needed for behaviour. If the error comes from missing current or organisation-specific information, first supply and verify the source through retrieval; for behaviour, try prompting and, if needed, fine-tuning. They are not alternatives; they can compose: a common and strong architecture is a model that has learned your format and tone, answering over retrieved documents that supply your facts. This is a starting intuition, not a strict split: knowledge can also be added by continued pretraining, and behaviour can change with a prompt.
- Do not DPO straight from a base model. DPO’s loss compares the policy against a frozen reference on two fixed completions, and it works best when those completions look like things the reference would produce. A base model is a poor reference for preference data written by an instruction-following model, and the objective spends itself on that mismatch instead of on the preference. Nothing in the loss forbids it — this is a statement about distribution match, not a definedness condition — but SFT first is the recipe that works.
9.10 Catastrophic forgetting
There is one more cost, and it does not appear on any invoice.
A fine-tune moves weights, and weights carry shared representations: the parameters that help represent how to classify a Turkish review can also contribute to representing Turkish. A model can improve on the new task while regressing on tasks it used to do well — the phenomenon called catastrophic forgetting (Kirkpatrick et al. 2017). It has also been observed in large language models during continual fine-tuning (Luo et al. 2025); its size depends on the training and the evaluation.
catastrophic forgetting — when training on a new task degrades performance on tasks the model previously handled, because the two relied on shared representations. It is invisible unless you evaluate the old task on purpose. ↩︎
The reason it is dangerous is not that it happens. It is that your evaluation will not show it. You built a test set for your task. Your fine-tune improves on your test set. Every number you look at goes up. If something went down, it is something you were not measuring, and you may find out from a user. Choose the abilities you want to keep before adapting, and measure them on the same held-out examples before and after.
Measuring it on our own model
We can do better than assert this. Take the same BERTurk model and the same Turkish reviews task from Chapter 4, fine-tune it two ways, and then ask a question that has nothing to do with sentiment: how well does the encoder still model Turkish?
The probe is the original pretraining objective. Mask 15% of the tokens in held-out Turkish text and measure the mean cross-entropy of the model’s own masked-language head at those positions, before and after. This is a masked-language-modelling probe; it is not directly comparable with the autoregressive perplexity in Chapter 6.
One methodological point decides whether the numbers mean anything. In the LoRA row the adapter is active: here it was probed merged into the base weights. (Merging is not required; in the re-run the merged model and the model with the adapter active differed by at most \(4.8\times10^{-6}\) in any logit, in fp32.) Deleting the adapter would recover the starting model and the measurement would be a tautology. What is being asked is: when you deploy the LoRA-adapted model, has that model’s grip on Turkish loosened?
One limit of the probe, stated before the numbers rather than after. The masked-language head is frozen at its pretrained weights while the encoder underneath moves, so a rising loss measures the encoder drifting away from that head — which mixes genuine loss of linguistic information with a representation the old head can no longer read. Those are not the same thing, and this design cannot separate them. What would separate them is a freshly fitted probe on top of each encoder, or a downstream Turkish task the encoder has not been tuned on. Read the table as a change in fit seen through the existing frozen head, not as a measurement of how much Turkish knowledge was lost.
Two probe sets, because they answer different questions. General Turkish is 1,826 sentences of ordinary prose from a parallel corpus (Zhang et al. 2020) — nothing to do with reviews, nothing to do with sentiment. Held-out reviews is the test split from the lab below (Section 9.11): same domain as the fine-tune, included so that nobody can dismiss the result as a domain-shift artefact. The masked tensors are generated once and reused for all three encoders, which removes sampling differences; the evaluation mode, the MLM head and the numerical precision should be held equal too.
| encoder | general TR loss | top-1 | reviews loss | top-1 |
|---|---|---|---|---|
| pretrained, untouched | 2.523 | 50.7% | 2.660 | 48.3% |
| after a full fine-tune | 2.837 (+12.5%) | 46.4% | 4.564 (+71.6%) | 28.9% |
| after LoRA \(r=8\), merged | 2.592 (+2.7%) | 49.6% | 3.276 (+23.2%) | 42.0% |
Loss is mean cross-entropy in nats over masked positions — 5,406 of them for general Turkish, 9,893 for the reviews — and top-1 is how often the model’s single best guess is the token that was actually there. Lower loss and higher top-1 are both better, and the base row is the model as the Hub ships it.
The probe’s code had not been saved; we rewrote it from the text and ran it (re-run, October 2026). For general Turkish we used the Turkish side of the OPUS-100 en-tr test split (1,976 deduplicated sentences, 2,860 masked positions), for reviews the first 2,000 reviews of the test split (9,753 positions); the full fine-tune is our 2e-5 run (85.34 macro-F1), and LoRA \(r=8\) is the run that reproduces the book’s lab row exactly. The pretrained model scored 2.897 loss and 47.0% top-1 on general Turkish, 2.602 and 49.1% on reviews. After the full fine-tune the loss rose +9.7% on general Turkish (top-1 44.1%) and +55.5% on reviews (35.0%); with merged LoRA the rises were +1.7% (46.2%) and +25.9% (42.2%). Because the text and the full fine-tune’s settings differ, the magnitudes differ from the book’s table; the ordering and the in-domain pattern are the same.
Three things in that table are worth more than the paragraph that introduced them.
Both adaptations look worse under the old head, and the full one by about 4.6× more. Say what that ratio is: 12.5% against 2.7% is a ratio of relative loss increases measured through a frozen head, not a measurement that one model forgot 4.6 times as much Turkish. On text that has nothing to do with the task, a full fine-tune raises the masked-language loss measured through the frozen head by 12.5% and costs four points of top-1 accuracy. That is not catastrophic in the dramatic sense, but it is a meaningful regression under this probe; it does not show that the model’s general Turkish ability fell by the same share. Test the product abilities you want to keep separately.
In this experiment LoRA’s loss increase is smaller than the full fine-tune’s, but not zero. +2.7% is small, but it is not zero, and the merged model reads measurably worse through the head it started with. Anyone who tells you parameter-efficient tuning cannot forget is quoting the reversibility argument and calling it the forgetting argument.
The regression is larger on the in-domain probe. This is the row that surprises people: the full fine-tune’s loss rises 12.5% on general Turkish and 71.6% on held-out reviews — nearly six times as much on the domain it was trained on (9.7% against 55.5% in the re-run). One possible reading: the model reorganises, for the task, the representations your data touched most — review text shifts from “everything a sentence can mean” towards “which of two buttons to press” — and the masked-language head can no longer read it as well. How much of that is representational change and how much a mismatch with the old head, this design cannot separate. The practical point stands: in this experiment the probe most sensitive to regression was the in-domain one, which is exactly the kind of probe that looks redundant next to the task test set and so does not get built.
Both probes deteriorated while the number you were watching improved. Loss rose and top-1 fell on every row and every probe set; the sentiment score went the other way. The full fine-tune is the best sentiment model in the lab — 83.90 macro-F1, the top of that table — and it is the weakest of the three under this probe, on both sets. Neither fact is visible from the other’s evaluation. This is the trap in its exact shape: not a model that got worse, but a model that got better at the one thing you measured.
What to do about it
The field’s standard answer is the one InstructGPT used. Having found “performance regressions on public NLP datasets” — the alignment tax — the authors mixed pretraining gradients back into the RLHF update, producing the variant they call PPO-ptx, and reported that this minimises the regressions on certain public NLP tasks (Ouyang et al. 2022). It is an example of mixing old data back in helping; which data share protects which ability has to be measured again on the target tasks.
In practice, three habits help:
| habit | what it costs | what it gives you |
|---|---|---|
| evaluate several old tasks, chosen before training, before and after | extra evaluation runs | makes regression on the chosen abilities visible |
| mix general data into the adaptation data in controlled shares | extra data and tuning runs | its effect on the abilities you want to keep can be measured |
| compare full fine-tuning and LoRA under the same quality and compute conditions | extra runs | shows the quality–regression trade-off on this task; for us LoRA, at a cost of 1.2 macro-F1 points (1.7 with tuned rates in the re-run), cut the general-Turkish loss increase measured through the frozen head from 12.5% to 2.7% (9.7% to 1.7% in the re-run) |
“Before training” is doing real work in row one. A benchmark chosen after you see the fine-tuned model’s failures is a benchmark chosen to make the model look good on the failures you happened to notice.
9.11 Lab: LoRA on a Turkish task, measured
This section looks at the same Turkish product-review task from Chapter 4, run seven ways on one Apple M-series GPU, with the numbers reported as they came out. They are observations of this setup; do not carry them directly to other hardware or another split.
The setup is deliberately modest, because the point is that this is reachable. BERTurk base (dbmdz/bert-base-turkish-cased, 110.6M parameters), the Chapter 4 split — 40,000 training reviews, 10,000 held out, seed 42 — two epochs, sequence length 128, batch size 32, on an Apple M-series GPU. LoRA adapters on query and value only, \(\alpha = 2r\), dropout 0.05. The original runs’ code, software versions and learning rate were not recorded; the re-run below says how we filled the gaps.
The limit of the comparison. All rows were run at one shared learning rate, so the gap between full fine-tuning and LoRA does not show each method’s separately tuned best. The rank rows are comparable at the shared rate, though each rank’s best rate may differ too. For a reliable method comparison, search each method’s rate on a validation slice held out of the training split; do not use the test set for the choice. The re-run measured how much this limit matters (below).
| what we trained | trainable | share | macro-F1 | F1(neg) | accuracy | wall clock | peak GiB |
|---|---|---|---|---|---|---|---|
| head only (a linear probe) | 1,538 | 0.001% | 48.37 | 0.00 | 93.68 | 3m 35s | 0.41 |
| full fine-tune | 110,618,882 | 100% | 83.90 | 70.16 | 95.62 | 11m 59s | 6.07 |
| LoRA \(r=1\) | 38,402 | 0.035% | 80.47 | 63.01 | 96.09 | 10m 01s | 4.32 |
| LoRA \(r=4\) | 148,994 | 0.135% | 80.50 | 62.97 | 96.26 | 9m 46s | 4.32 |
| LoRA \(r=8\) | 296,450 | 0.267% | 82.69 | 67.23 | 96.50 | 10m 04s | 4.32 |
| LoRA \(r=16\) | 591,362 | 0.532% | 82.59 | 66.99 | 96.58 | 10m 03s | 4.33 |
| LoRA \(r=64\) | 2,360,834 | 2.090% | 82.39 | 66.60 | 96.55 | 10m 30s | 4.37 |
For reference, Chapter 4’s best non-neural model on this same split — tf-idf over word 1-2 grams, no GPU — scored 76.21 macro-F1 and 56.47 F1(neg).
“Peak GiB” is the largest live MPS allocation measured in this study, in a fresh process per row. It is not the operating system’s total GPU or shared memory use: the MPS driver’s own high-water mark runs about 0.7 GiB higher across the rows. The re-run showed that when the column is sampled matters too (below). For real device planning, use the same measurement method and batch/sequence conditions.
Reading the table
Row 1 is the control, and it fails. Freezing the encoder and training only the classification head — 1,538 parameters — produces a model whose macro-F1 is 48.37 and whose accuracy is 93.68. Those are not new numbers. They are exactly Chapter 4’s majority-class baseline. A negative-class F1 of zero is consistent with a model that has fallen back on the majority class, and in the re-run the prediction counts confirmed it: it answered positive for all 10,000 test reviews. On a corpus that is 93.68% positive that is a locally sensible thing to do. A frozen encoder with a trainable head is a linear probe rather than an adapter-based method, and on an imbalanced task it can collapse — whether you file linear probes under “parameter-efficient fine-tuning” is a naming question; what matters is that they change no representation. The F1 on the negative class is 0.00. Discount this row by the learning-rate caveat above: it was run at the shared rate. In the re-run, class weights in the loss and a higher rate (1e-3 and 1e-2) lifted the probe to 63.85 and 64.90 macro-F1: better than 48.37, but more than 15 points behind the LoRA rows. Exercise 1 is that experiment.
That row is in the table because the difference between it and the LoRA rows is the entire argument. The linear probe trains only the final classification head; LoRA here adds small adapters to the query and value projections of every encoder block, so the gradient passes through the encoder. LoRA does not train every weight or every module; it spreads its change inside the encoder, across all blocks.
In this run rank stops mattering quickly. Going from \(r=1\) to \(r=8\) buys 2.2 macro-F1 points; from \(r=8\) to \(r=64\) — eight times the parameters — the score goes from 82.69 to 82.39. That small difference comes from one run; it does not show that the higher rank hurt. The re-run did not repeat that last step either: over three seeds \(r=1\) scored 80.55 ± 0.65 and \(r=64\) 84.05 ± 0.26 (\(r=8\) 82.66 on one seed), so in our runs rank kept giving small gains at 2e-4. On the other hand, \(r=8\) at its own validation-chosen rate (1e-3) reached 83.68 ± 0.40 — tuning the rate bought as much as an eightfold rank.
One confound in that row is worth naming, because it is the thing the literature actually argues about. With \(\alpha = 2r\) the effective scale \(\alpha/r\) is a constant 2 at every rank, so this sweep is not the disguised scale sweep it would have been under a fixed \(\alpha\). But holding \(\alpha/r\) constant is itself a choice: Kalajdzievski (2023) shows that a \(\Theta(1/r)\) scaling makes gradients shrink as rank grows, and that stable learning at high rank wants \(\Theta(1/\sqrt{r})\) instead — with “higher rank does not help” as the predicted symptom. But this single-seed curve does not show that the \(r=64\) result comes from that scaling alone; the learning rate and the seed are confounders too. How much rank pays under the common \(\alpha = 2r\) rule, and what changes under rank-stabilised scaling, needs a separate rate-and-scale sweep over several seeds; this chapter did not run it.
In the book’s run the curve rises once and flattens; in the re-run it flattens more slowly. What both share is that after \(r=1\) the gain from rank is the same size as, or smaller than, the effect of the learning rate. The low-rank hypothesis in Section 9.4 leads you to expect good results at small ranks, but it does not by itself prove this task’s curve; choose the rank over a few seeds on validation score, memory and cost.
The full fine-tune wins, and it wins on one class. 83.90 against \(r=8\)’s 82.69 is a 1.21-point difference in this run, and the minority class is where it comes from: F1(neg) 70.16 against 67.23, worth +2.93 points on that class. It is not all of the story, though — back out the other half and the full fine-tune is slightly behind on the majority class, F1(pos) 97.64 against 98.15. A 2.93-point gain on one class and a 0.51-point loss on the other average to the 1.21 you see. But look at the accuracy column — in this table every single LoRA run is more accurate overall than the full fine-tune (96.09–96.58 against 95.62). On imbalanced data the two metrics reward different trade-offs; how the decision boundary moved, only per-class errors and the probability distribution can show. In the re-run this split did not repeat: full fine-tuning (2e-5) also led on accuracy — 96.83% against 96.60% for tuned LoRA \(r=8\). The confusion matrices (seed 42) show full fine-tuning catching more of the negative class (415 against 382 of the 632 negatives) and making fewer errors overall (317 against 331). So the accuracy-versus-macro-F1 split in this table may be a property of this setup rather than of LoRA in general. Which metric you want is still a product decision, not a training one.
Rank 1 is the result worth sitting with. 38,402 trainable parameters — 36,864 in the adapters plus the 1,538-parameter head, 0.035% of the model, a file you could email — take a frozen encoder from 48.37 to 80.47 macro-F1, clearing Chapter 4’s tf-idf baseline by more than four points (80.55 ± 0.65 in the re-run). A small adaptation capacity spread across the layers can be enough for this task; that does not mean capacity plays no role.
The wall-clock column. On this device and settings \(r=8\) took 10m 04s and the full fine-tune 11m 59s: about 1.19×. The 373× difference in trainable parameters did not carry over to time. In the re-run (M5 Max) the ratio was 1.42× (6m 54s against 9m 47s), so the ratio depends on hardware and software. A profile would show which operations set the time.
The head-only row shows something by contrast. It finishes in 3m 35s (3m 14s in the re-run) — 3.3× faster than the full fine-tune — because with nothing trainable inside the encoder, backpropagation never enters it. LoRA’s adapters sit inside the blocks; their gradients follow the computation path through the encoder. Gradients for the frozen main weights are not kept, but the backward pass does not go away. In this setup LoRA bought memory and portability but did not markedly shorten the backward pass; measure the speed-up on your own model and implementation.
And the memory column is where the chapter’s arithmetic meets the floor. The 16-bytes-versus-2 table compared only selected parameter state; it did not predict an eightfold cut in total peak memory. Measured, LoRA cuts it 1.40× — 6.07 GiB to 4.32 GiB. Subtract the resident parameters — 0.41 GiB, measured before the first forward pass — from \(r=8\)’s 4.32 GiB peak and 3.91 GiB is left over. Most of that is activations, but not all of it: gradients, workspace and temporary tensors are in there too, and without a detailed memory profile we cannot split it. The choice of rank barely changes it. QLoRA’s authors measuring 567 MB of input gradients against 26 MB of adapter parameters at 7B points the same way. The re-run also showed that when the column is sampled matters: LoRA’s 4.32 matches exactly a value taken at the first forward pass, before any optimizer state exists; for full fine-tuning the first forward pass held 3.82 GiB and later steps, with Adam state, 6.96 GiB (a ratio of 1.61×).
Notice also that peak memory is essentially constant across ranks — 4.32 GiB at \(r=1\) and 4.37 GiB at \(r=64\) (4.40 in the re-run) — for the same reason. And notice the head-only row at 0.41 GiB: no backward pass through the encoder means no retained activations, an order of magnitude less memory, and a model that does not work.
What the numbers do not show
Three honest caveats, because a lab that only produces good news is a demonstration, not an experiment.
- These ratios do not carry over to other scales. The 1.40× memory ratio and the 1.19× speed-up (1.61× and 1.42× in the re-run) are laptop numbers at 110M parameters with a batch of 32, where activations dominate. Parameter state varies with model size; activations and temporary memory with the number of layers, batch, sequence length and checkpointing; as models widen the balance often tips towards parameters. The 119.7 and 15.5 GiB given for 8B are static-state estimates, not measured peaks. Do not quote this table’s ratios as general claims about LoRA.
- Peak memory is one number standing in for a distribution. It is the largest live allocation seen across 30 steps of a fixed-shape batch. Real training has variable sequence lengths and therefore a memory profile with a tail, and the tail is what actually kills a run at 3 a.m. Treat the column as a comparison between rows, not as a provisioning figure.
- One seed, one dataset, one language. The table’s runs are two-epoch runs on a single split, one seed per row. The 0.30 separating \(r=8\) from \(r=64\), and the 0.03 separating \(r=1\) from \(r=4\), are not enough to read a ranking into. The re-run’s repeated seeds give the scale: the standard deviation across seeds within a row was 0.26–0.65 points, and on MPS full fine-tuning moved 0.36 points even with the same seed. At that scale the gap between row 1 and everything else, and the 1.7 points between tuned full fine-tuning and LoRA, are above the noise; the accuracy-versus-macro-F1 split that the table shows for every LoRA row did not repeat in the re-run. (The parameter counts are the configuration’s arithmetic, not measurements, so no seed can move them.)
Your turn
- Reproduce row 1’s collapse, then fix it. Run the head-only baseline and confirm from the predicted class counts that you get the majority-class result. Now fix it without unfreezing anything: tune the learning rate and class weights in the loss (or resampling) on a validation slice. Report how close you get to the LoRA rows on the same test set, and discuss whether the gap is consistent with a change in representation; the training setups differ, so do not read an exact split off one experiment.
- Find the rank that stops paying. Sweep \(r \in \{1, 2, 4, 8, 16, 32\}\) with three seeds each. Plot macro-F1 against \(\log_2 r\) with error bars. State the rank at which the curve flattens, and — this is the part that matters — state whether your error bars can distinguish it from the rank below it.
- Break the \(\alpha = 2r\) convention on purpose. Fix \(r = 8\) and sweep \(\alpha \in \{2, 8, 16, 64\}\). Watch how the \(\alpha/r\) scale from Section 9.4 affects the loss, the gradients and the task score, and compare the effect with a learning-rate sweep; do not assume the two are exactly equivalent.
- Change what you attach to. Keep \(r = 8\) and move the adapters from
query, valueto the MLP (intermediate.dense,output.dense). Work out the new parameter count from Section 9.4’s formula before you run it — the adapter-only count is 2.5× thequery, valuefigure, and you should be able to say why the factor is 2.5 and not 2 (adding the head to the total trainable count changes the ratio). Compare scores under the same validation setup: does the score follow the parameters? - Show the undo. Keep the base checkpoint of the same revision. Fine-tune with LoRA, save the adapter, then load the base model and compare its weights to the original checkpoint tensor by tensor. Then count how many tensors differ in the task checkpoint after a full fine-tune, and show that reloading the saved base checkpoint undoes the full fine-tune too. Explain the operational difference from Section 9.10: it is not reversibility as such but managing the task update as a separate, small file.
- Measure your own forgetting. Take your fine-tuned encoder and run the masked-language probe from Section 9.10 on Turkish text from a domain neither you nor the fine-tune touched. Compare full fine-tuning against LoRA. Then mix 10% general Turkish into your fine-tuning set and measure again.
- Go where the baseline cannot follow. Chapter 4’s tf-idf model reached 76.21 macro-F1 with no GPU at all. Find an input where the fine-tuned model is right and tf-idf is wrong, and one where the reverse holds. The second is harder to find and more informative when you do.
Deliverable: one table with the same columns as ours, your own numbers in it, and a paragraph that writes down your minimum macro-F1, negative-class F1, latency and maintenance requirements and justifies the lowest total-cost configuration that clears them. Keeping the tf-idf baseline is an option too; if nothing clears the bar, say so and defer deployment.
9.12 Intuition checks
Answer before expanding. Every one of these is a place where a plausible mental model of fine-tuning produces the wrong engineering decision.
Note1. LoRA trains 0.035% of the weights. Does it use 0.035% of the memory, and run ~3,000× faster?
Neither, and the gap between those expectations and the measurements is the most useful thing in this chapter.
On memory: in this chapter’s chosen 16-byte setup a frozen parameter loses its gradient, master copy and Adam moments; the weight itself and the intermediate tensors needed to backpropagate to the adapters remain. In this lab peak memory fell from 6.07 to 4.32 GiB, only 1.40× (1.61× in the re-run once Adam state exists).
On speed: \(r = 8\) trains 373× fewer parameters and in this lab ran 1.19× faster (1.42× in the re-run, on another device). Measure the time and memory ratios separately for other models and hardware; LoRA’s advantage on memory and deployment is clearer than on the wall-clock of a small fine-tune.
Note2. Freezing the encoder and training just the classifier head — is that parameter-efficient fine-tuning?
It is a linear probe rather than an adapter-based method — some people file probes under the PEFT umbrella and some do not, so say which you mean. The substantive difference is that a probe changes no representation, and on this task, at the shared learning rate this lab used, it collapses completely.
Training 1,538 parameters on a frozen encoder produced macro-F1 48.37 and accuracy 93.68 — exactly Chapter 4’s majority-class baseline, with F1 on the negative class of 0.00 (in the re-run all 10,000 predictions were positive; with class weights the probe reached 64.90).
In this experiment LoRA trained small adaptations in the query and value projections of every encoder block, while the probe left the encoder unchanged. Rank 1 — 38,402 trainable parameters, 0.035% of the model — takes the same frozen encoder from 48.37 to 80.47, clearing the tf-idf baseline by four points. (The saved adapter file’s size depends on dtype and format; measure it.)
Note3. Higher LoRA rank means more capacity. Should you use the highest rank you can afford?
Not by default; measure. In this lab’s single run \(r=1 \to r=8\) bought 2.2 macro-F1 points and \(r=8 \to r=64\) showed a 0.30-point drop. The second gap is not reliable: in the re-run, over three seeds, \(r=64\) stayed 3.5 points ahead of \(r=1\) (84.05 ± 0.26 against 80.55 ± 0.65), and \(r=8\) at its own tuned rate scored 83.68 ± 0.40. So rank’s payoff is small, and the same size as the learning rate’s. Choose the rank over a few seeds on validation score, memory and cost; the low-rank hypothesis in Section 9.4 does not by itself prove this task’s curve.
Note4. Your full fine-tune beats LoRA on macro-F1 but loses on accuracy. Which is broken?
Most likely neither; the two metrics reward different things.
Full fine-tuning scored 83.90 macro-F1 against \(r=8\)’s 82.69, and the minority class is where the difference lives: F1(neg) 70.16 against 67.23, +2.93 points, against a 0.51-point loss on the majority class (F1(pos) 97.64 against 98.15). But every single LoRA run is more accurate overall — 96.09–96.58 against 95.62. (The re-run did not repeat this split; full fine-tuning led on accuracy too, 96.83% against 96.60%.)
On imbalanced data macro-F1 rewards success on the minority class and plain accuracy the total number of errors; how the decision boundary moved, per-class errors and the probability distribution show. Which you want depends on what a missed complaint costs relative to a false alarm, which is Section 12.7’ question, and no training table can answer it for you.
Note5. A 65B model needs “more than 780 GB” to fine-tune. Where does that number come from?
The QLoRA paper reports more than 780 GB of GPU memory for full fine-tuning at 65B. \(65 \times 10^9\) parameters × 12 bytes = 780 GB is a hypothetical parameter calculation that explains the size, not the paper’s itemisation. The instructive part is that the paper does not say which twelve, and two readings both fit.
Reading one: the three fp32 rows — master weight, Adam’s \(m\), Adam’s \(v\); adding the bf16 weight and gradient, this chapter’s 16-byte setup gives \(65 \times 16 = 1{,}040\) GB of static parameter state, consistent with “more than”. Reading two: bf16 weight, bf16 gradient, fp32 \(m\), fp32 \(v\), no master copy, with “more than” absorbing activations instead.
Activations and the distribution layout come on top; do not merge two different calculations into one real setup. Still, both decompositions point the same way: most of the headline is state that exists only because the parameter is being trained. Freeze it and that state evaporates — which is the one-sentence case for PEFT.
Note6. Your fine-tuned model got worse at things it used to do. What happened, and is it avoidable?
First test whether the drop on the old tasks is evaluation noise or a real regression. If it is real, one likely explanation is catastrophic forgetting: gradient descent on a narrow objective can move weights that other capabilities depended on. It is not an inevitable consequence of every fine-tune, but it is a common risk. Compare the learning rate, the number of epochs, the data mix (mixing some general data back in) and the scope of the adaptation; LoRA with an active adapter can also lower old-task performance (as Section 9.10 measured).
The reason this matters more than it sounds: it is measured against whatever you thought to evaluate. If your eval suite only covers the new task, the model can lose a capability you never tested and you will ship it. Build the regression suite before the fine-tune, not after.
9.13 Interview questions
The questions below test when to choose a technique as much as how it works; every answer has a measurement or a piece of arithmetic behind it.
A good signal in this area is a candidate who volunteers when not to fine-tune. As in the decision tree (Section 9.9), first pin down the kind of error and a measurable quality bar; then compare prompting, retrieval and adaptation.
What does fine-tuning actually fix, and what does it not? It fixes behaviour — format, style, task framing, refusal patterns, domain register. It can teach chosen facts too, but for keeping current organisational knowledge reliable and citable, consider retrieval separately; a fact added by fine-tuning is expensive to update and hard to attribute. The framing that makes this land: a base model is not broken, it is answering a different question — given this text, what comes next. Fine-tuning teaches it which question you are asking; it does not reliably teach it new answers, and when a team uses fine-tuning as its knowledge-update mechanism, it often disappoints. Then say the practical half: first test whether the error comes from knowledge, the prompt, the output schema or the model’s ability, and choose the method accordingly.
Explain LoRA to someone who knows linear algebra. Freeze \(W\) and learn \(\Delta W = BA\) where \(B\) is \(d \times r\) and \(A\) is \(r \times k\) with \(r \ll \min(d,k)\). Forward becomes \(Wx + \frac{\alpha}{r}BAx\), and at serving time you can fold \(\frac{\alpha}{r}BA\) into \(W\), removing the separate adapter multiply. Two details that show you have used it: \(A\) is randomly initialised and \(B\) is zeroed, so the adapter starts as an exact no-op and training cannot destabilise the pretrained model at step 0; and \(\alpha\) behaves roughly as a learning-rate multiplier on the adapter path, which is why \(\alpha = 2r\) is a convention rather than a law.
How much memory does fine-tuning need? Decompose it rather than guessing. Per trainable parameter in a chosen mixed-precision Adam setup: bf16 weight 2, bf16 gradient 2, fp32 master weight 4, fp32 \(m\) and \(v\) 8 together — roughly 16 bytes of static parameter state. Then name the term that decomposition omits and that often dominates at small scale: activations and temporary buffers, which vary with batch, sequence length and checkpointing and which freezing parameters does not much reduce (in the lab LoRA cut peak memory only 1.40×). That last sentence is what separates someone who has read about LoRA from someone who has watched it fail to fit.
SFT or DPO, and why did preference tuning become a separate stage? SFT teaches “here is a good answer” from one target per prompt. It cannot express this answer is better than that one, and for qualities like helpfulness or tone the judgement is comparative — which is exactly why preference optimization exists as a separate stage. DPO’s contribution is showing you can optimise the KL-regularized preference objective without training a separate reward network: in that objective the reward can be expressed as the policy/reference log-ratio (Rafailov et al. 2023). Practical note worth adding: an SFT start gives data–reference match in most recipes; DPO on a base model is mathematically defined, but it often disappoints when the preference data come from an instruction-following model.
When would you not fine-tune? When a simpler prompt, schema or reliable retrieval solves the measured problem; when you lack enough clean training and evaluation data; when the target behaviour will change often; when maintenance and regression costs outweigh the gain; or when you cannot build a regression suite for what you might break. Add the deployment argument: serving adapters cuts storage and the number of copies — but measure routing, versioning and concurrency costs too.
You fine-tuned and the score got worse. How do you debug it? First check the data split, the label distribution and that the evaluation code is correct. Then check the chat template — training with the wrong one teaches a format the server will never send; check whether the model collapsed to the majority class, which on an imbalanced task looks like a plausible accuracy (ours read 93.68% while scoring 0.00 F1 on the minority class); check the learning rate, since too high destroys pretrained structure and too low does nothing; and check whether your eval moved at all, because a frozen number usually means the trainable parameters were not the ones you thought. These are possible causes; test the learning rate and class balance in separate validation experiments rather than pinning one symptom on one cause.
9.14 Where to go deeper
Read one thing before any paper: open a chat model’s tokenizer and print tokenizer.apply_chat_template(messages, tokenize=False) for a two-turn conversation. Checking the role and end-of-turn markers helps catch a training–serving template mismatch early, and it takes a few seconds.
| resource | what it is |
|---|---|
| Deep Dive into LLMs like ChatGPT — Andrej Karpathy | three hours covering pretraining through RLHF; the clearest available account of where each training stage’s behaviour comes from |
trl |
SFT, DPO and PPO in one library, with readable trainers — the fastest way to see what each objective actually optimises |
peft |
LoRA and its relatives; the source is short enough to read and settles what rank actually changes |
| Interconnects — Nathan Lambert | ongoing, well-sourced writing on post-training from someone who publishes open recipes |
| Lil’Log on reward hacking | the failure mode that every preference-optimisation method inherits, surveyed properly |
| topic | source |
|---|---|
| the technique this chapter is mostly about | Hu et al. (2022) |
| fitting it on one card | Dettmers et al. (2023) |
| what LoRA replaced, and why latency mattered | Houlsby et al. (2019); Lester et al. (2021) |
| the recipe that made instruction-following standard | Ouyang et al. (2022) |
| preferences, before the language models | Christiano et al. (2017); Stiennon et al. (2020) |
| the policy update in PPO-based RLHF | Schulman et al. (2017) |
| preference tuning without a separate reward network | Rafailov et al. (2023) |
| a group reward instead of a value estimator: GRPO | Shao et al. (2024) |
| reasoning training with verifiable rewards | Guo et al. (2025) |
| an open recipe with every stage documented | Lambert et al. (2025) |
| how little SFT data can be enough | Zhou et al. (2023) |
| generating the data instead of writing it | Wang et al. (2023) |
| instruction tuning generalising to unseen tasks | Wei et al. (2022) |
| principles instead of annotators | Bai et al. (2022) |
| making a model smaller | Hinton et al. (2015) |
| adapting to a domain by pretraining more | Gururangan et al. (2020) |
| what fine-tuning costs you elsewhere | Kirkpatrick et al. (2017); Luo et al. (2025) |
| the prompting you should try first | Schulhoff et al. (2025) |
9.15 Exercises
- Print
apply_chat_templatefor a two-turn conversation on two different model families. List every token that differs. Then explain, in terms of what SFT actually optimizes, why serving with family A’s template a model trained with family B’s degrades quality without raising an error. - Reproduce the example from Section 9.3 with Llama 3’s tokenizer: 31 tokens, 4 of them the answer. Write out the
labelstensor for both the masked and unmasked choice, and compute, for each, what fraction of the loss terms comes from prompt and template positions; do not mistake that share for gradient magnitude. Then run a short pilot with each mask and compare the answering behaviour on the same prompts. - Section 9.4 gives two possible decompositions of QLoRA’s “>780 GB” that both come to 12 bytes per parameter. Separate the memory items the paper reports from those it does not, and say what evidence you would need to choose between them. Then compute a conditional total for 8-bit Adam (\(m\) and \(v\) in one byte each), listing the weight, gradient, master copy, moments and scaling metadata you assume; state the new ratio between a trained and a frozen parameter and one thing the reduced precision could plausibly cost you. Do not present the result as real peak memory.
- Open the
config.jsonof a model not in Section 9.4’s table and check the input/output sizes of the actual targetLinearmodules (head_dim, gated MLPs and module names vary by model). Compute the adapter-only parameter count for LoRA \(r=16\) on (a)q_proj, v_projand (b) attention + MLP. Then load the model withpeftand compare with the total trainable count; verify the source of any difference (biases, a classifier head or other modules left trainable) in the model. - LoRA initialises \(A\) random and \(B\) zero. Work out what happens for \(A\) zero and \(B\) random: write down \(\partial\mathcal{L}/\partial A\) and \(\partial\mathcal{L}/\partial B\) at step 0 (state the non-zero input and upstream-gradient assumption) and say whether the run escapes. Two of the four combinations start exactly at the pretrained model and generally have a non-zero gradient — identify them and say what the gradients have in common. Then propose an experiment that tests whether the same starting output guarantees the same training trajectory.
- A team says their model “gives wrong prices for our products” and asks for a fine-tune. Walk them down Section 9.9, out loud, in five sentences. Then construct the variant of their complaint for which fine-tuning is the right answer, and say what data it would need.
- You have 4,000 prompts, one written answer each, and a complaint that the model “is too verbose and too eager to agree”. Decide between more SFT and DPO. Say exactly what you would collect for each, how many items, and how you would know within a week whether it worked.
- A colleague explains a disappointing DPO run on a base model with “the model does not answer yet, so there is nothing to rank”. Look at \(\mathcal{L}_{\text{DPO}}\): does the loss generate new answers during training, and what does it hold fixed? Explain the role of the reference log-probabilities. Then design an experiment that compares a base model and an SFT model on the same preference pairs, measuring data coverage and evaluation results; do not assume the failure or its cause before measuring.
- GRPO replaces the value model with the mean reward of \(G\) samples from the same prompt. Under independent samples the group mean has variance \(\operatorname{Var}(r)/G\). Taking a learned value estimator’s error variance and bias into account too, discuss which estimator would have the lower mean squared error; if the samples are not independent, say how correlation changes this. Then describe a task where the group mean would be the worse baseline.
- Design the forgetting evaluation you would actually run for a Turkish customer-support fine-tune. Name the benchmark, say why it is the right one, and state the threshold at which you would refuse to ship. Then explain why the threshold has to be written down before you see the fine-tuned model’s results.