Appendix B — Math Refresher
The book assumes less mathematics than its subject matter suggests. What it does assume is here, and each entry is written to be enough to read the chapter, not enough to derive the result.
B.1 Linear algebra
A vector is a list of numbers, and here it is a point in space. A word, a sentence or a document becomes a vector so that similar can mean close. The whole of Chapter 3 is that one move.
The dot product measures alignment. For two vectors \(\mathbf{a}\) and \(\mathbf{b}\), \(\mathbf{a} \cdot \mathbf{b} = \sum_i a_i b_i\). It is large when the two point the same way, zero when they are perpendicular, negative when they oppose. Divide by both lengths and you get cosine similarity, which is the dot product with magnitude removed — used everywhere from Chapter 3 to Chapter 19, and the reason a long document does not automatically look similar to everything.
Matrix multiplication is many dot products at once. Multiplying a matrix by a vector takes the dot product of the vector with each row. That is the bulk of what a neural network layer does — add a bias vector and pass the result through a non-linearity and you have the whole of one — and it is why GPUs matter: the dot products are independent, so they run in parallel. Chapter 31’s prefill-versus-decode split is a direct consequence — reading a prompt is many dot products at once, writing a token is one after another.
Dimensionality reduction finds the directions that carry the most signal. Singular value decomposition factors a matrix into directions ordered by how much of its squared magnitude each accounts for, and truncating after the first few gives the best low-rank approximation there is, in the Frobenius and spectral norms — that is the Eckart–Young theorem, and it is why “keep the top components” is a theorem rather than a heuristic. It is optimal for those norms, not for every measure of “close”. One qualifier that matters in practice: those directions describe variance only if you centred the columns first, which is what PCA does. On raw counts the leading direction is usually close to the mean, and a first component that correlates with document length is the result. LSA uses exactly that uncentred truncated SVD on purpose, so it is a choice to make knowingly rather than automatically a mistake. Chapter 3 uses it; Chapter 11 relies on the intuition.
If one fact from this section survives: similarity is geometry. Nearly every retrieval, clustering and embedding decision in this book is a claim about which points are close, and the arguments are about whether the geometry matches the meaning.
B.2 Probability
Conditional probability is probability after you know something. \(P(A \mid B)\) is the chance of \(A\) given that \(B\) happened. A language model is exactly this: \(P(\text{next token} \mid \text{everything so far})\).
The chain rule turns a sentence into a product. The probability of a sequence is the product of each token’s probability given the ones before it — \(P(w_1 \ldots w_n) = \prod_i P(w_i \mid w_{<i})\). Chapter 6 builds the whole subject on this line.
Expectation is the average you would get in the long run, weighting each outcome by its probability. It is why the same metric measured on 30 items and on 1,000 are different objects — estimates of one expectation, with very different uncertainty — which Chapter 22 turns into sample-size arithmetic.
Cross-entropy measures how surprised your model is by the true answer, averaged over the data: low when the model put high probability on what actually happened. Perplexity is its exponential, and reads as “how many options was the model effectively choosing between”. A perplexity of 20 means roughly a twenty-way guess per token.
Softmax turns arbitrary scores into probabilities by exponentiating and normalising: \(\text{softmax}(z)_i = e^{z_i} / \sum_j e^{z_j}\). Two properties matter downstream. The exponential exaggerates differences, but how much depends on their size in score units rather than on their being small: scores of 0 and 0.1 come out as 0.475 and 0.525, while 0 and 3 come out as 0.047 and 0.953. The lead has to be large on the logit scale before it is large in probability. And dividing the scores by a temperature before the exponential controls how much — high temperature flattens the distribution, temperature approaching zero makes it pick the maximum. That is the single parameter behind Chapter 18, and behind Chapter 15’s measurement that the same setting is right for chat and wrong for translation.
B.3 Inference on measurements
This is the mathematics the book leans on most, because almost every chapter ends in a proportion measured on a small sample. Four tools cover nearly all of it.
A proportion needs an interval, and Wilson’s is the one to use. The textbook \(\hat{p} \pm z\sqrt{\hat{p}(1-\hat{p})/n}\) breaks exactly where this book lives: at \(\hat{p} = 1\) it gives zero width, and near 0 it gives negative bounds. The Wilson interval solves the same problem the other way round and behaves at the edges:
\[\frac{\hat{p} + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \;\pm\; \frac{z}{1 + \frac{z^2}{n}}\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}\]
Thirty out of thirty is not 100% ± 0; it is [88.6, 100]. Chapter 22 makes the case, and most chapters after it report one.
Paired binary outcomes can be compared with McNemar’s test. When two systems run on the same items, what carries the information is the disagreements: \(b\) items the first got right and the second wrong, \(c\) the other way. The items they both got right or both got wrong tell you nothing about which is better. The exact test is a binomial sign test on those discordant pairs, \(\min\bigl(1,\ 2\,P(X \le \min(b,c))\bigr)\) for \(X \sim \text{Binomial}(b+c, 0.5)\). The cap matters: at \(b = c\) the doubled tail exceeds one, and with no discordant pairs at all there is no evidence either way (\(p = 1\)). It routinely finds differences that two overlapping marginal intervals would call inconclusive. Chapter 17 and Chapter 22 both turn on this. For paired scores that are not binary — a ROUGE per item, a rating — the paired analogue works on the per-item differences.
A bootstrap is often the practical choice for a mean score. Resample the items with replacement a few thousand times, recompute the statistic each time, and take the 2.5th and 97.5th percentiles. It avoids a parametric model of the scores, though not every assumption — the resampling unit has to match how the data were sampled, and the percentile interval behaves poorly for some statistics — which is why it is a common default for a ROUGE mean or an average rating. Resample at the level the data is clustered at: two judgments of the same document are not two independent facts.
Power decides how big the sample has to be, and it is a question you can answer before you collect anything. For two proportions at significance \(\alpha\) and power \(1-\beta\), the minimum detectable difference \(\Delta\) needs
\[n \;\approx\; \frac{\left(z_{1-\alpha/2}\sqrt{2\bar{p}\bar{q}} + z_{1-\beta}\sqrt{p_1 q_1 + p_2 q_2}\right)^2}{\Delta^2}\]
where \(n\) is the number of items in each of two independent groups, \(p_1\) and \(p_2\) are the two proportions, \(q_i = 1 - p_i\), \(\bar{p} = (p_1 + p_2)/2\) and \(\bar{q} = 1 - \bar{p}\). \(z_{1-\alpha/2}\) is the two-sided critical value and \(z_{1-\beta}\) the one-sided value for the target power, both standard-normal quantiles. A paired design on the same items needs a different calculation, built on the expected disagreement rate. The formula above is the arithmetic behind Chapter 22’s finding that a hundred-item eval set cannot see a five-point improvement. The useful direction is backwards: start from the difference that would change a decision, and solve for \(n\).
The failure mode all four are guarding against is the same one: a number reported without an interval, compared against another number reported without an interval, on a sample too small to separate them. It is the most common mistake in this field’s published results, and it is arithmetic rather than statistics.
B.4 Optimization
A loss is a number you want to be small, measuring how wrong the model currently is. Training is the search for parameters that make it smaller.
Gradient descent is walking downhill. The gradient points in the direction of steepest increase, so step the other way, by a distance set by the learning rate. Too large and you overshoot; too small and convergence becomes impractically slow. Everything else in training — momentum, Adam, schedules — is a refinement of that sentence.
Backpropagation is the chain rule applied to get the gradient, computed efficiently by working backwards through the network. You do not need to derive it to use it; you need to know what it is and is not. It computes the gradient — it does not do the optimising, and it does not stop training being a search through parameter space. What it makes cheap is knowing which way to step.
Regularisation is anything that trades training fit for generalisation. The form with the cleanest statistical reading is a penalty on large weights, which says simpler explanations are more likely to be right and corresponds to a prior on the parameters. It is not the only form: early stopping, dropout, data augmentation and — the one this book keeps returning to — simply having more data all do the same job by different means.
Overfitting is fitting the training sample more closely than the population it came from. The model gets better on data it has seen and worse on data it has not — and outright memorisation is only the extreme case; learning a real pattern that happens to hold in your sample and not in general is the common one. It is why every number in this book is reported on held-out data, and why Chapter 13’s finding that 13.06% of a benchmark’s test set sat verbatim in its training split matters so much: the held-out set was not held out.
B.5 Notation
The notation this book uses, and roughly where each symbol first appears.
| symbol | meaning | first used |
|---|---|---|
| \(\mathbf{v}\), \(\mathbf{x}\) | a vector (bold) | Chapter 2 |
| \(v_i\) | the \(i\)-th component of \(\mathbf{v}\) (plain) | Chapter 2 |
| \(\mathbf{A}\), \(\mathbf{W}\) | a matrix (bold capital) | Chapter 7 |
| \(\mathbf{a} \cdot \mathbf{b}\) | dot product | Chapter 2 |
| \(\lVert \mathbf{v} \rVert\) | length (norm) of a vector | Chapter 2 |
| \(\cos(\mathbf{a}, \mathbf{b})\) | cosine similarity | Chapter 2 |
| \(\lvert V \rvert\) | vocabulary size | Chapter 2 |
| \(d\) | embedding dimension | Chapter 3 |
| \(n\) | number of items, or n-gram order | Chapter 22 |
| \(\theta\) | model parameters, collectively | Chapter 6 |
| \(P(A \mid B)\) | probability of \(A\) given \(B\) | Chapter 6 |
| \(w_i\), \(w_{<i}\) | token \(i\); all tokens before it | Chapter 6 |
| \(\mathbb{E}[\cdot]\) | expectation | Chapter 22 |
| \(\text{softmax}(\cdot)\) | scores to probabilities | Chapter 7 |
| \(T\) | temperature | Chapter 18 |
| \(\sum_i\), \(\prod_i\) | sum and product over \(i\) | Chapter 6 |
| \(\mathcal{L}\) | a loss function | Chapter 6 |
| \(Q\), \(K\), \(V\) | query, key and value matrices | Chapter 7 |
| \(k\) | a count of things kept — beams, neighbours, documents | Chapter 19 |
Two conventions worth stating because they are inconsistent in the wider literature, and one collision worth warning about.
Bold means a vector or matrix; plain means a scalar — \(\mathbf{v}\) is a vector, \(v_i\) is one number in it. One exception, inherited from the literature rather than chosen: attention’s \(Q\), \(K\) and \(V\) are set plain even though they are matrices, because every paper on the subject sets them that way and bolding them would make this book the odd one out.
\(k\) is usually a count of retained things — beam width (Chapter 15), neighbours (Chapter 19), documents (Chapter 20). It is an index in two places, both of them sums over classes or samples rather than over a vector’s components: Chapter 4‘s macro-\(F_1\) averages over \(K\) classes, and Chapter 3’ negative-sampling loss sums over \(K\) negatives. The distinction to hold is that \(i\) indexes components and \(k\) indexes a set whose size is itself a parameter.
\(V\) carries two meanings and they are a chapter apart. \(\lvert V \rvert\) is the vocabulary size from Chapter 2 onwards; \(V\) unadorned is the value matrix in Chapter 7’ attention. Nothing disambiguates them but context, which is the convention the literature has settled on and not one this book can fix on its own.