3  Embeddings: Words as Coordinates

Chapter status: drafted. The numbers and code outputs on this page come from real runs with gensim 4.4, scikit-learn 1.9 and NumPy 2.5 on Python 3.12. The code shown is abridged to explain the method; the full data preparation and reporting steps that produce the tables are in the chapter’s notebook. The skip-gram, analogy and subword explorers are interactive.

Learning objectives. After this chapter you can:

3.1 The problem, in one sentence

Chapter 2 ended in an uncomfortable place. Document vectors worked, but one-hot word vectors shared no similarity between words at all.

Every word was a one-hot column, so pump, pumps and impeller were three mutually orthogonal directions — each as unrelated to the others as pump is to Wednesday. Cosine similarity between any two distinct words was exactly zero. By construction. Always.

A good representation should be able to tell three relations apart: morphological closeness like pump–pumps, semantic or functional closeness like pump–impeller, and unrelatedness like pump–Wednesday.

An example shows what follows from that. In a linear model that uses only word counts, pump and pumps are separate features. Suppose your training data contains the pump failed and your test data contains the pumps failed. The weights learned for the and failed can be used; the weight learned for pump, however, does not carry over to the pumps feature by itself.

embedding — a learned numerical representation of a unit. In this chapter we mostly represent words with dense vectors of relatively few dimensions; in distributional methods, words used in similar contexts can end up close together in that space. Think of it as giving every word a set of map coordinates. ↩︎

In this chapter we move from the one-hot representation, which gives every word its own axis, to one in which all words are represented with the same small number of learned dimensions. In linear-algebra terms that means multiplying the one-hot vectors by a learned matrix.

For each word we get, say, 100 or 300 numbers, and we aim for words used in similar ways to end up near each other. That short, dense vector is an embedding. In a dense vector most components are usually non-zero; in the sparse count vectors of Chapter 2 most components were zero.

Three kinds of embedding, one chapter. The word “embedding” is used for three different things, and separating them now makes the rest of the chapter easier:

The three things “embedding” can mean
Kind Example Does the same word get the same vector everywhere?
Static word embedding word2vec, GloVe, fastText Yes — one vector per word form, whatever the context
Contextual token embedding BERT No — each token’s vector is computed from its context in the sequence given to the model; one word can consist of several tokens
Sentence or document embedding SBERT and its relatives Not the question: one vector for the whole sentence or document

This chapter is about static word embeddings. It also gives a short introduction to document vectors. Contextual representations are covered in Chapter 7, and the use of sentence and document representations in retrieval in Chapter 19.

Two reading layers. The chapter is long; on a first read this spine is enough:

  • On a first read: the distributional hypothesis → co-occurrence → word2vec → cosine → GloVe and fastText → the limits of static embeddings.
  • Deep dive (second read): the mathematics of PPMI and SVD → the gradient of negative sampling → the relationship between SGNS and PMI → the anisotropy and bias measurements.

Most of the derivations in the second layer sit in collapsible Deep dive boxes; the main line is complete without opening them.

What we will look at. We first build the space from counts; on a small example every step can be checked by hand. Then we write skip-gram with negative sampling from scratch, gradients included, and examine how that method relates to PMI. Finally we look at what embeddings are used for, at their geometric properties, and at the limits of a representation that does not change with context. On a first read you can skip the collapsible Deep dive boxes and the from-scratch implementation code.

One-hot encoding shows categories as separate coordinates. In some models that use it, independent parameters are learned for each category. Representing word–context interactions with two low-rank matrices, by contrast, gives those interactions a shared structure. That structure can let what is learned from similar distributions contribute to generalization. It should not be equated with a saturated model or with hierarchical partial pooling in the technical sense; the comparison is there for intuition only.

3.2 The distributional hypothesis

“You shall know a word by the company it keeps.” — J.R. Firth, 1957 (Firth 1957)

Firth was making a point about linguistic method; Zellig Harris had already made the operational version of it (Harris 1954). The claim is that a substantial amount of semantic and functional information about a word can be recovered from the distribution of contexts it appears in.

Put concretely: if two words keep appearing in the same slots — the ___ failed, replaced the ___, ___ pressure dropped — then their uses are systematically similar.

This is not a theorem but a hypothesis that is tested against data. Sharing similar contexts can help to identify topical or functional closeness. It does not by itself identify synonymy: antonyms such as hot and cold share many contexts too. Later in this chapter we look at that distinction with numerical examples.

An important advantage of this approach is that it needs no hand-labelled examples. The training signal is derived from the contexts in raw text. That makes it possible to learn representations from large unlabelled corpora.

What does “similar” mean here? Distributional similarity is not the same as synonymy. cat and dog are the same kind of thing; doctor and hospital are related; hot and cold are opposites that keep the same company. That distinction comes back later in this chapter, when we read cosine similarity.

3.3 Count first: co-occurrence, PPMI, SVD

A count-based approach does three things. It counts the word–context pairs inside a chosen context window (their co-occurrences), reweights the counts, and approximates the resulting matrix with a low-rank structure. Following those steps on a small example makes the later methods easier to understand. Skip-gram with negative sampling has also been shown, under certain assumptions, to implicitly factorize a matrix related to PMI, which is defined below (Levy and Goldberg 2014b).

The three steps in this section.

  1. Count. Which words appear in which contexts?
  2. Reweight. Compare the observed pair frequency with the value expected from the marginal frequencies of the word and the context.
  3. Compress. Represent the repeated patterns in fewer dimensions.

Here is a corpus of ten sentences, small enough to check every number.

import numpy as np

corpus = [
    "the king rules the castle",
    "the queen rules the castle",
    "the king eats an apple",
    "the queen eats a banana",
    "the royal castle is grand",
    "a banana is sweet fruit",
    "an apple is sweet fruit",
    "the man eats an apple",
    "the woman eats a banana",
    "the king and queen are royal",
]

tokens = [s.split() for s in corpus]
vocab = sorted({w for s in tokens for w in s})
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
print("sentences:", len(corpus), " vocabulary:", V)
print(vocab)
sentences: 10  vocabulary: 19
['a', 'an', 'and', 'apple', 'are', 'banana', 'castle', 'eats', 'fruit', 'grand', 'is', 'king', 'man', 'queen', 'royal', 'rules', 'sweet', 'the', 'woman']

Step 1 — count what appears near what

Here the context of a word consists of up to two tokens to its left and up to two to its right, within the same sentence. The centre token is not part of its own context. For every (centre word, context word) pair we add one to the corresponding cell; because the window is symmetric, the pair in the opposite direction is counted too, at its own centre position.

W = 2
C = np.zeros((V, V))
for sent in tokens:
    for i, w in enumerate(sent):
        for j in range(max(0, i - W), min(len(sent), i + W + 1)):
            if i != j:
                C[idx[w], idx[sent[j]]] += 1
### raw co-occurrence, a few rows
            eats   rules   royal   sweet     the  castle
king           1       1       0       0       4       0
queen          1       1       1       0       3       0
apple          2       0       0       1       0       0
banana         2       0       0       1       0       0

Already the structure is visible: king and queen have nearly the same row, and so do apple and banana. That is the distributional hypothesis working. The rows of \(C\) are word vectors — 19-dimensional ones — and unlike one-hots they are not orthogonal.

But they have chapter 2’s disease.

### the same problem as chapter 2: 'the' dominates
  the      total co-occurrences    22
  eats     total co-occurrences    16
  is       total co-occurrences    11
  queen    total co-occurrences    10
  king     total co-occurrences     9
  a        total co-occurrences     8

The largest count in the king row belongs to the. But the is frequent around many words, so that high count cannot by itself be read as an association specific to kings. Raw counts are affected by the association and also by how frequent the words are overall.

Step 2 — PPMI: from raw counts to unexpected co-occurrence

pointwise mutual information (PMI) — the logarithm of the ratio between the observed probability of a word–context pair and the product of its marginal probabilities. It is positive for pairs seen more often than the independence model expects and negative for pairs seen less often. ↩︎

We want to ask: do these two words co-occur more often than chance would predict? Pointwise mutual information asks exactly that.

\[ \mathrm{PMI}(w, c) = \log \frac{P(w, c)}{P(w)\,P(c)} \]

Read the fraction like this: the denominator (\(P(w)\,P(c)\)) is how often we would expect the two words to land next to each other if they were placed completely independently; the numerator (\(P(w, c)\)) is how often they actually do. So PMI is nothing but the log of observed ÷ expected. Three cases:

  • Ratio 1 — the observed probability of this pair equals the value the independence model predicts. Since \(\log 1 = 0\), PMI is zero. That does not mean there is no semantic relationship between the words.
  • Ratio above 1 — they co-occur more often than expected; they attract. The log is positive.
  • Ratio below 1 — they co-occur less often than expected; they avoid each other. The log is negative.

Particularly for rare words, observing only a few co-occurrences may not be enough to estimate a negative association reliably. Positive PMI (PPMI) sets negative PMI values to zero and so keeps only the pairs seen more often than the independence model expects. That can be a useful representational choice; it does not mean that every negative PMI value is meaningless.

\[ \mathrm{PPMI}(w, c) = \max\!\left(0,\ \log \frac{c_{wc}\, N}{c_{w\ast}\, c_{\ast c}}\right) \]

Here \(c_{wc}\) is how often the pair actually occurred, \(c_{w\ast}\) is the row total, \(c_{\ast c}\) the column total, and \(N\) the grand total of the matrix. That fraction is doing something very simple, and it is worth seeing it with actual numbers.

Work one cell by hand. We will run the formula on apple–sweet, but first pin down where the four numbers come from, because all four are read off the same matrix \(C\):

  • \(c_{wc} = 1\) — how many times apple and sweet actually landed in each other’s window. That is a single cell, visible in the table above.
  • \(c_{w\ast} = 7\) — the sum of the apple row, meaning “how many times was apple counted next to anything at all”. apple occurs three times in the corpus, with 2, 3 and 2 neighbours respectively: \(2 + 3 + 2 = 7\).
  • \(c_{\ast c} = 6\) — the same sum for sweet, down its column.
  • \(N = 144\) — the sum of every cell in the matrix: all the (word, neighbour) pairs counted in the corpus. You can check it by hand. Each token contributes one pair per neighbour in its window — 4 for a token in the middle of a sentence, 2 or 3 at the edges — so a five-token sentence yields 14 pairs. The corpus has nine five-token sentences and one six-token sentence (18 pairs): \(9 \times 14 + 18 = 144\).

Note that \(N\) is not the 51 tokens in the corpus; it is the number of pairs. All three probabilities — \(P(w, c)\), \(P(w)\) and \(P(c)\) — are estimated from the same pair table (the cell, the row sum and the column sum), so all three are divided by how many pairs were counted, not how many tokens exist.

Now the actual question: if the two words were sprinkled through the text completely independently, how many times would we expect them to land next to each other?

\[ \frac{c_{\text{apple},\ast} \times c_{\ast,\text{sweet}}}{N} = \frac{7 \times 6}{144} = 0.29 \]

We observed 1, which is \(144 / 42 = 3.43\) times more than expected. PMI is the log of that ratio: \(\log 3.43 = 1.23\) — the value in the table.

Now do the same for king–the, which occurs four times — exactly four times as often as apple–sweet. But the is everywhere: its column sums to 22, and the king row sums to 9. The independence model predicts \(9 \times 22 / 144 = 1.38\) co-occurrences for this pair.

Put the two calculations side by side:

Raw count and frequency relative to the independence model
observed expected ratio PMI
apple–sweet 1 \(7 \times 6 / 144 = 0.29\) 3.43 1.23
king–the 4 \(9 \times 22 / 144 = 1.38\) 2.91 1.07

The apple–sweet pair, observed once, gets the higher PMI because its observed-to-expected ratio is larger. The expected count is 1.38 for king–the and 0.29 for apple–sweet. Those expectations do not say how much of what we observed is causally “explained”; PMI compares two relative frequencies.

Think of a crowded party. Someone who talks to everyone tells you nothing when you see them with the guest of honour — you expected that. Someone who talks to only three people all evening, and one of them is the guest of honour, tells you a great deal. PMI is the arithmetic of that intuition: divide what you saw by what you had every reason to expect.

total = C.sum()
row = C.sum(axis=1, keepdims=True)
col = C.sum(axis=0, keepdims=True)
with np.errstate(divide="ignore", invalid="ignore"):
    pmi = np.log((C * total) / (row * col))
pmi[~np.isfinite(pmi)] = 0.0
PPMI = np.maximum(pmi, 0.0)
### PPMI for the same cells
            eats   rules   royal   sweet     the  castle
king        0.00    0.69    0.00    0.00    1.07    0.00
queen       0.00    0.59    1.06    0.00    0.67    0.00
apple       0.94    0.00    0.00    1.23    0.00    0.00
banana      0.94    0.00    0.00    1.23    0.00    0.00

Compare with the raw table. Among the columns shown, the apple–sweet count is 1, half the apple–eats count; the PPMI values are nevertheless 1.23 and 0.94. The reason is the lower marginal frequency of the sweet context. The transformation takes frequency into account, but it does not guarantee that an association resting on a single observation is reliable. The highest PPMI in the row is 2.04, for the an context; in this corpus an is followed by apple every time it occurs.

PPMI and IDF are not the same formula, but they share an engineering instinct: a high raw count is not information on its own. IDF measures how rare a term is across the documents of a collection, and contributes to its weight in a particular document once combined with TF. PMI checks whether two words meet often only because each of them is frequent.

Consider two words that each occur 10 times in a million-token corpus and never co-occur. Under independence, ignoring sentence boundaries and counting only the tokens immediately to the left and right, the expected directed co-occurrence count is about \(2 \times 10 \times 10 / 10^6 = 0.0002\). Observing zero in this example is unsurprising. Widening the window to two tokens on each side roughly doubles the expectation. The PMI of a zero count goes to \(-\infty\); the finite value obtained with smoothing depends on the method chosen.

The negative PMI estimates of rare pairs can be strongly affected by sampling uncertainty. Clipping at zero leaves those uncertain values out of the representation and makes the matrix sparser. Levy et al. (2015) showed that, on the similarity and analogy benchmarks they examined, the differences between methods mostly stayed small once hyperparameters were tuned comparably.

Step 3 — collect the repeated structure in a few dimensions

The PPMI rows already behave sensibly. (Neighbours here, and everywhere in this chapter, are ranked by the cosine similarity of Chapter 2, \(\cos(\mathbf a, \mathbf b) = \mathbf a \cdot \mathbf b \,/\, (\lVert \mathbf a \rVert\, \lVert \mathbf b \rVert)\): 1 means the two vectors point the same way, 0 means they are orthogonal.)

### nearest neighbours before factorization (PPMI rows, 19-D)
  king     queen 0.55, are 0.55, man 0.45, the 0.37
  apple    man 0.79, fruit 0.52, banana 0.40, is 0.31
  eats     a 0.55, an 0.55, man 0.34, woman 0.34

king → queen reflects the relationship we expected. apple → man, on the other hand, cannot be read as synonymy. In this small corpus the two rows carry high weights in the same columns, an and eats, and cosine measures that overlap. man occurs in only one sentence of the corpus; a context overlap that comes from a single example may not be a reliable semantic similarity.

singular value decomposition (SVD) — a way of factorizing a matrix into singular vectors and singular values. Keeping the \(d\) largest singular values and the corresponding directions gives the best rank-\(d\) approximation in the Frobenius norm. ↩︎

Truncated singular value decomposition gives you a regularisation and compression mechanism against overlaps of that kind, which rest on a single example. Keep only the \(d\) directions with the largest singular values and you force the model to spend its budget on the structure that recurs, leaving one-off coincidences only whatever room is left over. It is not a guarantee of correctness: on a small or biased corpus a low-rank model can generalise the wrong structure too.

In this small example 19 dimensions cost nothing to compute, so compression is not needed for speed here; the point is to watch what a low-rank representation does to the \(V\)-dimensional vectors of a real system.

\[ \mathrm{PPMI} \approx U_d \Sigma_d V_d^\top, \qquad \text{embedding} = U_d \Sigma_d \]

We use the rows of \(U_d \Sigma_d\) as each word’s coordinates in this chapter. The literature also uses \(U_d\), or variants that scale the singular values by other powers; the measured consequences of that choice come up just below.

Is that notation unfamiliar? Then here is the picture behind it.

Think of the PPMI matrix as a stack of transparencies. Each one shows a single recurring pattern — these words tend to sit near those words — and each is printed at a different darkness. Stack all 19 and you reproduce the original matrix exactly.

The singular values in \(\Sigma\) are those darknesses, sorted from boldest to faintest. \(U\) says how much each word participates in each pattern.

Truncating to \(d\) means keeping only the boldest few transparencies. The operation drops the components that belong to the small singular values; it can reduce some accidental detail, and it can also lose useful distinctions. What is kept is decided by the singular-value structure of the matrix, not by how many times something repeats.

It can help to think of a limited storage budget: with fewer components kept, some detail is lost. The method’s criterion, however, does not line up exactly with the detail a person finds important, or with noise.

d = 5
U, S, Vt = np.linalg.svd(PPMI, full_matrices=False)
emb = U[:, :d] * S[:d]

In this small example we compute the full SVD and take the first five components. For large sparse matrices a method that computes only the components needed is preferred.

Multiplying by S gives more weight to the coordinates that belong to large singular values. The scaling that minimizes reconstruction error, however, need not be the best choice for word similarity. In the family \(W = U_d \Sigma_d^{\,p}\) the exponent \(p\) can be tuned as well; Levy et al. (2015) show that on similarity tasks \(p = 0.5\) or \(p = 0\) can give better results than \(p = 1\). We use \(p = 1\) here to keep the computation visible; in practice \(p\) should be chosen on validation data.

### singular values (how much structure each dimension carries)
  5.75  4.76  4.03  3.71  3.64  3.08  3.03  2.80
  first 2 dimensions hold 36.8% of the sum of squares
  first 5 dimensions hold 65.4% of the sum of squares

### nearest neighbours after SVD to 5 dimensions
  king     queen 0.90, are 0.88, the 0.74, castle 0.65
  apple    banana 1.00, woman 0.86, man 0.85, fruit 0.74
  eats     an 0.95, a 0.95, man 0.64, woman 0.63
  castle   are 0.82, is 0.80, king 0.65, queen 0.61

Those percentages are shares of the sum of squared singular values. Five components carry about two thirds of the total energy and the remaining fourteen about one third. The ratios summarize the compression; they do not show that the discarded components are unimportant for a task.

Strictly the share is of the total sum of squares \(\lVert X \rVert_F^2\) and not of variance — the PPMI matrix is never centred, so these shares are not variance shares. (Uncentred data has a variance too; it is simply not what this quantity measures.) The two coincide only after centring, which is what makes the thirty-word PCA plot later in this chapter a genuinely different calculation.

The easiest way to see what happened is to put three cosines side by side, before and after factorization:

The same counts, compressed into five dimensions
word pair PPMI rows, 19-D SVD, 5-D what happened
apple – banana 0.40 1.00 never share a sentence in this corpus; the shared contexts already link them, and the compression is what makes the link decisive
king – queen 0.55 0.90 already right, and got stronger
apple – man 0.79 0.85 the one-sentence accident — not deleted, but no longer first

Look at the third row, because this step is usually described wrongly.

SVD does not delete the apple–man accident. That cosine rises, from 0.79 to 0.85. What changes is the ranking: before, apple’s nearest neighbour was man with banana third; now banana is first and man third.

So the compression does not break the wrong link. It strengthens the right one enough to overtake it — and that is what “preferring signal over noise” actually looks like. The ordering improves; the absolute values do not.

The number to watch is the first row. apple and banana never once occur in the same sentence in this corpus, so nothing links them directly. What links them is that they keep the same company — both follow eats, both precede is sweet fruit. The raw PPMI rows already register that as a cosine of 0.40, because shared contexts are shared coordinates; what the compression does is strengthen it to 1.00, enough to reorder the neighbour list. Call this shared-context generalisation rather than transitivity — cosine similarity is not a transitive relation in the mathematical sense — and note that low-rank compression is what makes it decisive rather than what makes it visible.

This is latent semantic analysis (Deerwester et al. 1990) applied to a word–context matrix; the original LSA applied the same truncated SVD to a term–document matrix, and it came before word2vec by more than twenty years.

### compression
  one-hot / count row : 19 numbers per word
  svd embedding       : 5 numbers per word
    different one-hots  : cosine 0.00 (always)
  king vs queen (2-D) : cosine 0.996
  king vs banana(2-D) : cosine 0.141

Read 0.996 with the caveat in the box below already in mind: in a two-dimensional space where every word shares the sign of the first coordinate, a high cosine is partly the shared direction talking, not the similarity.

A detail worth noticing, because it comes back at the end of the chapter. In this small example the first coordinate is negative for every word (between −0.70 and −2.14). That shared direction can contribute to cosine values; the magnitudes differ, however, so the dimension is not entirely without discriminating information. The figure below plots dimensions 2 and 3 in order to make other distinctions visible. That choice is no substitute for the similarities in the full space.

A space that is not spread evenly over directions is called anisotropic; a shared mean direction is one of the forms that anisotropy can take. We will measure it on GloVe later.

Ten sentences, reduced to two coordinates — dimensions 2 and 3 of the SVD

3.4 From counting to predicting: word2vec

So counting works. We built a space by hand, and royalty landed on one side and food on the other. Why were prediction-based methods like word2vec taken up so quickly?

Scalability was an important factor. A square co-occurrence matrix over 100,000 words has ten billion possible cells; sparse storage keeps only the non-zero ones. In the batch PPMI–SVD pipeline used here we first accumulate the counts and then factorize the matrix. word2vec, by contrast, learns vectors from local training pairs without building the co-occurrence matrix explicitly. Incremental versions of count-based methods are possible too, so this is not a distinction that holds for every method.

word2vec — a family of methods from 2013 that learn word vectors from context. Skip-gram predicts the context words from the centre word and CBOW predicts the centre word from the context words; the learned vectors can be used in other tasks. ↩︎

In 2013 Mikolov and colleagues published word2vec, a method that skipped the matrix entirely (Mikolov, Chen, et al. 2013; Mikolov, Sutskever, et al. 2013). Rather than counting co-occurrences and factorizing, train a classifier to predict context — and then use the weights it learned rather than its predictions.

Architecturally, that classifier is a shallow neural network: an input layer, a single hidden layer, an output layer. It is called shallow because there is only one hidden layer. Unusually, that layer also has no activation function — it is purely linear. In the skip-gram architecture the three layers can be read like this:

  1. The input layer is the centre word’s one-hot vector: as long as the vocabulary, a single 1, zeros everywhere else.
  2. The hidden layer multiplies that vector by a matrix \(\mathbf{W}\). Multiplying a one-hot by a matrix selects the row belonging to that word, so the hidden layer is a lookup table. The selected row is the word’s embedding. No further activation is applied, which is why this layer is often called a projection layer instead.
  3. The output layer scores the selected vector against candidate context words. For that we use a second table, \(\mathbf{W}'\), which holds one row per word under the full softmax and under negative sampling alike.

In CBOW the vectors of the context words are first combined: summed or averaged.

The critical point: the product is not the network you train, it is the weights of the network’s first layer. In classic word2vec use it is mostly the input matrix \(\mathbf{W}\) that is kept — one row per word, 300 numbers per row — and \(\mathbf{W}'\) is set aside. That is a common choice, not a requirement: some approaches use the output matrix, or the sum or average of the two, and GloVe’s authors recommend adding them (Pennington et al. 2014). Here we set up the prediction task in order to learn vectors we can reuse. The input embedding table in Chapter 7 works in a similar way; that table, however, usually has one row per token of the model’s vocabulary and not one row per word.

There are two ways to set up that prediction, and they are mirror images. Same window, same words, arrows reversed:

The two word2vec architectures
Architecture Input Predicts
Skip-gram the centre word the context words
CBOW the context words the centre word

This chapter follows skip-gram, which is also what negative sampling is most often paired with.

The two word2vec architectures, on one sentence
THE SUPERVISION SIGNAL — ONE WINDOW, CENTRE failed, WIDTH 2 the pump failed during the night shift CBOW context → centre the pump during the W average 300 numbers W′ failed 4 inputs one hidden vector 1 prediction One training example per window. Usually faster; combines the context vectors. Skip-gram centre → context failed W look up v 300 numbers W′ the pump during the 1 input its own vector 4 predictions Four training examples per window. More training pairs; can help on rare words.

Same window, same words, arrows reversed. Both architectures learn two matrices — W maps a word to its vector, W′ scores candidate partners. The pretrained vectors you download are mostly the rows of W; W′ is usually left unused.

Skip-gram is the variant most analyses and tutorials use, and the rest of this chapter follows it. (It is not the only survivor: CBOW is the default in both the original word2vec tool and gensim, because it is faster.) Slide a window over the corpus and every position yields training examples.

Skip-gram — type a sentence and watch the training pairs appear

Negative sampling with a logistic loss

First, what exactly is skip-gram asking? Take a centre word \(w\) with vector \(\mathbf{v}_w\) (its row of \(\mathbf{W}\)). Every candidate context word \(c\) gets a score: the dot product \(\mathbf{v}_w \cdot \mathbf{u}_c\) with its row of \(\mathbf{W}'\). A dot product is large when two vectors point the same way (and are long), so it works as a compatibility score. The original skip-gram turns these scores into a probability distribution over the whole vocabulary with a softmax, and training maximises the log-probability of the context words actually observed:

\[ P(c \mid w) = \frac{\exp(\mathbf{v}_w \cdot \mathbf{u}_c)} {\sum_{c'=1}^{V} \exp(\mathbf{v}_w \cdot \mathbf{u}_{c'})}, \qquad \text{maximise} \sum_{(w,\,c)\ \text{in the corpus}} \log P(c \mid w) \]

For a statistician this is multinomial logistic regression with one class per vocabulary word.

The full softmax computes and normalizes \(V\) context scores for every training pair. With \(d\)-dimensional vectors that costs about \(O(Vd)\). When \(V = 400{,}000\) (the size of the GloVe vocabulary we use later in this chapter), processing a large number of pairs that way is expensive.

Negative sampling changes the training objective: it replaces this multi-class task with smaller binary classification tasks. The scoring cost per pair becomes about \(O((K+1)d)\), where \(K\) is the number of negative samples drawn per pair.

negative sampling — for each observed word–context pair, drawing a small number of contexts from a noise distribution and using them as negatively labelled training examples. The negative examples need not be unseen in the corpus or semantically unrelated. ↩︎

The step that made word2vec practical replaces the 400,000-class softmax with a question a statistician will recognise (Mikolov, Sutskever, et al. 2013):

Did this training example come from the distribution of observed pairs (\(y = 1\)) or from the noise sampler (\(y = 0\))? The same word pair can be produced either way.

That is binary classification. For each observed pair \((w, c)\), draw \(K\) context words from a noise distribution — a frequency-based distribution over the vocabulary, whose exact form we return to in a few pages — and label them 0; that drawing is the negative sampling the method is named after. Then fit a logistic regression whose linear predictor is the dot product of the two words’ vectors:

\[ P(y = 1 \mid w, c;\, K, q) = \sigma(\mathbf{v}_w \cdot \mathbf{u}_c), \qquad \sigma(z) = \frac{1}{1 + e^{-z}} \]

Here \(q\) is the noise distribution the negative contexts are drawn from. This probability belongs to the training sampling scheme we set up; it is not the probability that two words occur together in the real world.

Unlike ordinary logistic regression the features are not fixed: the centre and context vectors are learned at the same time, which makes the problem non-convex once the two tables are considered together.

The rest is minimising the usual log loss:

\[ \mathcal{L} = -\log \sigma(\mathbf{v}_w \cdot \mathbf{u}_c) - \sum_{k=1}^{K} \log \sigma(-\mathbf{v}_w \cdot \mathbf{u}_{n_k}) \]

The term for the positive pair decreases as the dot product between the centre and context vectors grows. The term for a negative sample decreases as the corresponding dot product shrinks. The picture of pulling together and pushing apart describes that change in score; it does not say that the Euclidean distance between the vectors necessarily moves the same way.

Work one update by hand. Vectors are 300-dimensional in practice; use 4 so the arithmetic fits on a line. Say the centre word failed currently has \(\mathbf{v} = (0.4,\ -0.2,\ 0.7,\ 0.1)\), its true partner pump has \(\mathbf{u} = (0.5,\ 0.1,\ 0.6,\ -0.3)\), and a sampled fake opera has \(\mathbf{u} = (-0.3,\ 0.6,\ -0.2,\ 0.4)\).

For the real pair, the dot product is

\[ (0.4)(0.5) + (-0.2)(0.1) + (0.7)(0.6) + (0.1)(-0.3) = 0.57 \]

so \(\sigma(0.57) = 0.639\): under this training scheme the model gives the pair about a 64% probability of having arrived as a positive example. The contribution of the positively labelled example to the loss is \(-\log 0.639 = 0.448\).

For the fake pair the dot product is \(-0.34\), giving \(\sigma(0.34) = 0.584\) and a loss of \(-\log 0.584 = 0.538\). The model is already leaning the right way on both, but not confidently, so both will be nudged.

The update itself is where the whole thing becomes recognisable. For a single labelled term, the derivative with respect to the dot product \(z = \mathbf{v}_w \cdot \mathbf{u}_c\) is just

\[ \ell(z, y) = -y \log \sigma(z) - (1 - y) \log\bigl(1 - \sigma(z)\bigr), \qquad \frac{\partial \ell}{\partial z} = \sigma(z) - y \]

The derivative of the total SGNS (skip-gram with negative sampling) loss is found by adding up the derivatives of all the positive and negative terms involved. Here \(y\) is 1 for a positive example and 0 for a negative one. Prediction minus target — the same expression that falls out of every logistic regression you have ever fitted. For pump that is \(0.639 - 1 = -0.361\) (negative, so the dot product goes up); for opera it is \(0.416 - 0 = 0.416\) (positive, so the dot product goes down).

Both vectors in a pair move. The collapsed derivation below shows the two gradients, and each vector is dragged along the direction of the other.

To keep the arithmetic followable, let us hold the context vectors \(\mathbf{u}\) fixed and update only the centre vector \(\mathbf{v}\). Its gradient is the sum of both terms — that is what the loss above says, and what gv in the training code below computes in one einsum:

\[ g_{\mathbf v} = (0.639 - 1)\,\mathbf u_{\texttt{pump}} + (0.416 - 0)\,\mathbf u_{\texttt{opera}} = (-0.305,\ 0.213,\ -0.300,\ 0.275) \]

With a learning rate of 0.05 that moves failed·pump from 0.570 to 0.590 and failed·opera from \(-0.340\) to \(-0.359\).

Update the context vectors too, as a real implementation does, and the same step lands on 0.603 and \(-0.374\) — the same directions, moved roughly 1.7 times as far.

Those are tiny movements. Multiply them by tens of billions of pairs and you get the geometry that made these models famous. Do not draw the wrong conclusion from that: the mathematics of a single SGD update is surprisingly simple, but “word2vec is plain SGD and the rest does not matter” is not true. Much of what decides the quality is not in the update itself but in which pairs are shown, how often, and against what — the sampling and training strategies: negative sampling, subsampling of frequent words, the noise distribution and the context window. A few pages on we take those four one at a time.

Take the real-pair term alone, and write \(z = \mathbf{v}_w \cdot \mathbf{u}_c\). The single fact you need is that the sigmoid has an unusually tidy derivative:

\[ \sigma'(z) = \sigma(z)\,\bigl(1 - \sigma(z)\bigr) \]

Now differentiate \(-\log \sigma(z)\) with the chain rule:

\[ \frac{\partial}{\partial z}\bigl[-\log \sigma(z)\bigr] = -\frac{\sigma'(z)}{\sigma(z)} = -\frac{\sigma(z)(1 - \sigma(z))}{\sigma(z)} = \sigma(z) - 1 \]

The \(\sigma(z)\) cancels. That is a pleasant property, but it is not the main reason log loss is paired with the sigmoid; the main reason is about probability, and it is worth seeing step by step.

The label of every training example is 0 or 1. The probability distribution describing a single trial with a yes/no outcome is called the Bernoulli distribution; a coin flip is an example. We model the label with it. Its parameter \(p\) lies in the interval \([0, 1]\). Our model supplies that \(p\): for finite \(z\) the sigmoid produces \(p = \sigma(z)\) in the interval \((0, 1)\).

Now ask: how much probability did the model give to the answer we actually observed? If the answer was “yes” (\(y = 1\)) that probability is \(p\); if it was “no” (\(y = 0\)) it is \(1 - p\). This number is called the likelihood: how probable the model considered what in fact happened. A good model gives high probability to what happened, so the natural target of training is to make the likelihood large. Adding logarithms is easier than multiplying those probabilities across all pairs, and a minus sign turns “make large” into “make small”:

\[ -\log(\text{likelihood}) \;=\; -\bigl[\,y \log p + (1 - y)\log(1 - p)\,\bigr] \]

That expression is binary cross-entropy: the negative log-likelihood of the Bernoulli model. Making it small makes the likelihood of the observed labels large. Probability estimates can also be evaluated with a squared loss; that is a different criterion, however, and not the same as the Bernoulli log loss derived here.

In short, pairing the sigmoid with log loss buys two things at once: a clean probabilistic reading through the Bernoulli likelihood and a gradient as simple as the one above. The second is a gift of the first. For a negative sample the term is \(-\log \sigma(-z)\) and the same two lines give \(\sigma(z)\). Both cases collapse into \(\sigma(z) - y\).

Getting from there to the vectors is one more chain-rule step, because \(\partial z / \partial \mathbf{v}_w = \mathbf{u}_c\):

\[ \frac{\partial \mathcal{L}}{\partial \mathbf{v}_w} = \bigl(\sigma(z) - y\bigr)\, \mathbf{u}_c, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{u}_c} = \bigl(\sigma(z) - y\bigr)\, \mathbf{v}_w \]

The gradient of each vector is the scalar error \(\sigma(z) - y\) multiplied by the other vector. When the model gives the correct label a high probability, the scalar error becomes small. Vector norms can grow, however, and the contributions within a batch can add up, so this property alone does not guarantee stable training or convergence. The learning rate, the gradient scale and, where needed, clipping have to be considered separately.

Why negative sampling exists: 400,000 dot products, or six
The honest version softmax over the whole vocabulary failed v · u the of pump valve opera zinc tariff carbon ⋮ ⋮ 400,000 rows 400,000 dot products, then normalize — per pair. The version that shipped one real partner, K = 5 sampled fakes failed pump opera zinc tariff carbon helix real → push σ up fake → push σ down fake fake fake fake 6 dot products, no normalization — per pair.

The left panel scores all \(V = 400{,}000\) candidates and applies a softmax. The right panel computes six dot products, for one positive and five negative examples. In this example the number of dot products falls by a factor of about 67,000; the actual training speed-up depends on hardware, sampling and implementation.

The sigmoid and the log loss that SGNS uses for each labelled pair are familiar from logistic regression. But the vectors on both sides of the dot product are learned as well, so the model as a whole is not classical logistic regression with fixed features. A more accurate description is logistic matrix factorization, in which two vector tables are learned together. That distinction explains both the familiar form of the gradients and the non-convexity of the joint optimization.

Two matrices are learned, not one: \(\mathbf{v}_w\) for words as centres and \(\mathbf{u}_c\) for words as contexts. Almost everyone discards \(\mathbf{u}\) and keeps \(\mathbf{v}\). Adding them sometimes helps a little; GloVe’s authors recommend it (Pennington et al. 2014).

The objective — drag the dot product and watch the loss

Which vectors get updated in one step?

To understand the computational cost of negative sampling, look at which rows a single training pair contributes to.

For a single training pair, with \(K = 5\) negatives, there are seven row updates across the two matrices:

  • one row of \(\mathbf{W}\): the centre word’s vector \(\mathbf{v}_w\).
  • one row of \(\mathbf{W}'\): the true context word’s vector \(\mathbf{u}_c\).
  • \(K\) rows of \(\mathbf{W}'\): the vectors of the sampled fake words.

If the sampled negatives repeat, or collide with the true context word, the number of distinct rows touched is fewer than seven; the number of update contributions does not change.

The remaining rows — those of every other word in the vocabulary, in both matrices — do not enter the SGNS data loss of this pair, so their gradients are zero. Under the SGD update used here, which has no momentum and no weight decay, those rows do not change. With regularization, or with an optimizer that carries past updates forward, the parameter update has to be examined separately.

For scale, take the model we train from scratch below: two tables of 18,677 words and 100 dimensions hold 37,354 rows and 3,735,400 parameters in total. A single training pair with five negatives contributes to at most seven distinct rows, about 0.019% of them. That sparsity is one of the important reasons training is efficient.

You can see it directly in the update lines of the code below: targets in np.add.at(Wo, targets.ravel(), ...) holds only \(K+1\) indices per pair, so only that many rows of Wo are added into. The rest are untouched.

With a full softmax every output vector takes part in the scoring in the denominator and usually receives a non-zero gradient. In this example that is one centre row and 18,677 output rows, 18,678 rows in total. The ratio of about 2,668 against seven rows compares the number of rows affected; it is not a ratio of total computational cost or of running time.

In SGNS the table is accessed only through the selected indices, so the gradient of the data loss is sparse at the level of rows. PyTorch’s nn.Embedding(..., sparse=True) produces sparse gradients for accesses of this kind. If the embedding parameters are also used in other operations, such as a full softmax, the gradient of the whole parameter need not be sparse. Here the centre vector is updated directly when the word is selected as a centre, and the context vector when it is used as a positive or a negative context.

Four training choices that decide how well word2vec works

The update itself was simple. How good the model is depends largely on the four choices below; each one answers part of the question “which pairs do we show, how often, and against what?”

word2vec’s four training choices
choice what it sets typical value effect
Number of negatives (\(K\)) negatives drawn per positive pair 5–20 for a small corpus and 2–5 for a large one in the original work changes the balance between cost and training signal
Context window candidate neighbours and weighting by distance for example at most 5, with the dynamic-window option narrow windows can bring out functional relations more, wide ones topical relations
Subsampling frequent words (\(t\)) probability of keeping very frequent tokens for example \(t = 10^{-3}\) reduces the share of frequent words in training
Noise distribution (the 0.75 exponent) probability of drawing each negative \(q(w) \propto f(w)^{0.75}\) raises the relative sampling share of rare words compared with the raw unigram distribution

Negative sampling and \(K\). You have seen the method itself: for each real pair, \(K\) fake context words are drawn from a noise distribution and the model learns to tell real from fake. What is tuned here is \(K\). With a very small \(K\) the model sees too few “not this one” examples and the vectors do not separate enough; a large \(K\) makes every step more expensive. The original paper suggests 5–20 for small corpora and 2–5 for large ones (Mikolov, Sutskever, et al. 2013) — with plenty of data every word already gets many updates, so a few negatives are enough.

The context window. With a window of 5, up to five tokens to the left and up to five to the right of the centre, within the same input sequence, are candidate contexts. Narrow windows (1–2) mostly reflect close syntactic relations, and wide windows (5–10) reflect topical relations more. Those are tendencies; the corpus, the language and the preprocessing affect the outcome too. If sentence boundaries are kept, the window does not cross them.

word2vec adds one more refinement: it does not keep the window fixed. For each centre word it draws the effective width at random between 1 and the maximum (a dynamic window). The result is that near neighbours produce training pairs more often than distant ones — an implicit weighting by distance. You can change the window by hand in the counting method at the start of the chapter and see the difference yourself (exercise 1).

Subsampling frequent words. Frequent words produce a large number of training pairs. To reduce their relative influence, each token is kept with a probability that depends on its frequency. Below, \(f(w)\) is the relative frequency among the tokens left after the vocabulary filter, and \(t\) is a threshold on the same scale. The keep probability used is

\[ p_{\text{keep}}(w) = \min\!\left(1,\ \left(\sqrt{\frac{f(w)}{t}} + 1\right)\cdot\frac{t}{f(w)}\right), \qquad t \approx 10^{-3} \]

The formula looks arbitrary until you put numbers through it. The threshold \(t = 10^{-3}\) sets the scale: any word making up less than a thousandth of the corpus comes out with the inner expression above 1 and is simply always kept, which is what the \(\min\) is doing. (The exact break-even point is a little higher, at \(f \approx 2.6\,t\) — about 0.26% of the corpus — where the inner expression equals exactly 1.)

The original C implementation uses the keep formula above. The paper gives \(P(\text{discard}) = 1 - \sqrt{t/f}\); that expression is negative when \(f < t\), so as a probability it is read as \(P(\text{discard}) = \max(0,\ 1 - \sqrt{t/f})\), that is \(P(\text{keep}) = \min(1,\ \sqrt{t/f})\). The two formulas are not the same; when reproducing an experiment, state which implementation was used.

What subsampling actually throws away
share of corpus example \(p_{\text{keep}}\) discarded
5% the in English text 0.16 84%
1% of, and 0.42 58%
0.1% (\(=t\)) mid-frequency word 1.00 0%
0.01% and below everything in the tail 1.00 0%

So this is not a gentle reweighting. It deletes five out of every six occurrences of the, and it does not touch the tail at all — rare words are left alone while the top of the Zipf curve you plotted in Chapter 2 is aggressively thinned.

In the run below it discards 26.6% of the in-vocabulary tokens: 6,397,826 become 4,698,743. Almost all of what it throws away is the, of, and and their friends.

(Where does 6,397,826 come from? It is what is left of the run’s tokens read: 6794527 once everything below min_count=20 is dropped. The subsampling rate applies to that.)

When a token is removed, the words on either side of it move closer together in the remaining sequence. The window size does not change, but words that were further apart in the original text can now form new context pairs. That changes both the number of training examples and the context distribution.

The noise distribution: the 0.75 exponent. The noise distribution is the name for the distribution the fake context words are drawn from. Negative samples are not drawn from the unigram distribution but from \(P(w) \propto f(w)^{0.75}\). Raising every probability to a power below 1 pulls all of them toward 1, and it pulls the smallest ones furthest in relative terms: \(0.5^{0.75}\) is 19% larger than \(0.5\), while \(0.001^{0.75}\) is 5.6 times larger than \(0.001\). Renormalising afterwards therefore moves probability mass out of the head of the distribution and into the tail.

Concretely: if a common word is 5,000 times more frequent than a rare one, then after the exponent it is only \(5000^{0.75} \approx 595\) times more likely to be drawn as a negative. The gap narrows by a factor of about eight, and it narrows in the rare word’s favour.

Negative samples make the context vectors receive updates from pairs drawn from the noise distribution as well as from the observed positive pairs. The 0.75 exponent raises the relative share of rare contexts compared with raw unigram sampling; it does not give them a kind of signal that did not exist before. The centre vectors are affected indirectly too, through those changing context vectors. The exponent also changes the noise distribution in the implicit objective of SGNS; a later Deep dive box examines that connection. 0.75 is an empirical choice, not a universal theoretical optimum. Levy et al. (2015) show that hyperparameters of this kind can affect comparison results substantially.

Deep dive: writing word2vec from scratch

For the main storyline, the training loop is six steps:

  1. take a real (centre, context) pair from the corpus;
  2. draw \(K\) fake context words from the noise distribution;
  3. compute the dot products between the centre vector and those \(K+1\) vectors;
  4. apply the sigmoid;
  5. take the error, prediction minus label;
  6. update the corresponding rows of the two tables with that error.

The NumPy example below shows the training pipeline used with the first 40 million characters of the text8 corpus (a cleaned dump of English Wikipedia). The data preparation and the update steps are visible; the lines that report the loss, the time and the neighbours printed underneath are not part of this abridged listing (the full code is in the chapter notebook). The basic gradient comes from the logistic loss above; the gradient clipping below changes the update rule. If the numerical details — batching, sampling, gradient clipping — are not what you are here for, skip to the results.

import gzip, time
from collections import Counter
import numpy as np
import gensim.downloader as api

rng = np.random.default_rng(0)

with gzip.open(api.load("text8", return_path=True), "rt") as f:
    raw = f.read(40_000_000).split()

MIN_COUNT = 20
counts = Counter(raw)
vocab = [w for w, c in counts.most_common() if c >= MIN_COUNT]
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)

total = sum(counts[w] for w in vocab)
freq = np.array([counts[w] / total for w in vocab])
keep = np.minimum(1.0, (np.sqrt(freq / 1e-3) + 1) * (1e-3 / freq))

ids = np.array([idx[w] for w in raw if w in idx], dtype=np.int32)
ids = ids[rng.random(len(ids)) < keep[ids]]          # subsampling

noise = freq ** 0.75                                  # the 0.75 exponent
noise /= noise.sum()
cum = np.cumsum(noise)

D, K, WINDOW, EPOCHS, LR0 = 100, 5, 5, 3, 0.02
Wc = ((rng.random((V, D)) - 0.5) / D).astype(np.float32)   # centre vectors
Wo = np.zeros((V, D), dtype=np.float32)                    # context vectors

# Two simplifications. (1) The window is fixed at 5; no dynamic window is used.
# (2) text8 is processed as a single token stream. The original sentence and
# document boundaries are not kept, so artificial context pairs can form
# across them. With separate documents, build the pairs within each document.
centres, contexts = [], []
for off in range(1, WINDOW + 1):
    centres.append(ids[:-off]); contexts.append(ids[off:])
    centres.append(ids[off:]); contexts.append(ids[:-off])
centres = np.concatenate(centres)
contexts = np.concatenate(contexts)
N = len(centres)

BATCH = 2048
n_batches = N // BATCH

def sigmoid(x):
    # Overflow-safe sigmoid; it does not artificially restrict the score to [-8, 8].
    return np.exp(-np.logaddexp(0.0, -x))

for epoch in range(EPOCHS):
    for b in range(n_batches):
        # drawn WITH replacement: "3 epochs" means three times as many
        # pair-draws as there are pairs, not three passes. About 5% of pairs
        # are never drawn and some are drawn six times, so do not compare
        # EPOCHS=3 here with epochs=3 in gensim.
        sl = rng.integers(0, N, BATCH)
        c, o = centres[sl], contexts[sl]
        negs = np.searchsorted(cum, rng.random((BATCH, K))).astype(np.int32)
        targets = np.concatenate([o[:, None], negs], axis=1)   # (B, K+1)
        labels = np.zeros((BATCH, K + 1), dtype=np.float32)
        labels[:, 0] = 1.0                       # first column is the real pair

        v = Wc[c]                                              # (B, D)
        u = Wo[targets]                                        # (B, K+1, D)
        score = sigmoid(np.einsum("bkd,bd->bk", u, v))         # (B, K+1)
        err = (score - labels).astype(np.float32)              # the negated residual

        lr = LR0 * max(1e-4, 1 - (epoch * n_batches + b) / (EPOCHS * n_batches))
        gv = np.einsum("bk,bkd->bd", err, u)
        gu = err[:, :, None] * v[:, None, :]
        np.clip(gv, -1.0, 1.0, out=gv)
        np.clip(gu, -1.0, 1.0, out=gu)
        np.add.at(Wc, c, -lr * gv)
        np.add.at(Wo, targets.ravel(), (-lr * gu).reshape(-1, D))

E = Wc / (np.linalg.norm(Wc, axis=1, keepdims=True) + 1e-9)

Look at the line err = score - labels. That is predicted minus observed — the residual of a logistic regression (observed minus predicted) with its sign flipped, which is the form the log-loss gradient uses directly. Every gradient below it is that difference times the other vector. If the statistics feels familiar, it should.

tokens read: 6794527
vocabulary (min_count=20): 18677
tokens after subsampling: 4698743
training pairs: 46987400
### training
  epoch 1    0.0%  loss  4.159
    epoch 1  100.0%  loss  2.409
  epoch 2  100.0%  loss  2.368
  epoch 3  100.0%  loss  2.350
  trained in 611s

### nearest neighbours, from ~60 lines of NumPy
  king      consort 0.75, kings 0.75, isabella 0.75, aram 0.74, regent 0.73
  france    spain 0.79, italy 0.75, netherlands 0.74, switzerland 0.73, portugal 0.72
  physics   mechanics 0.82, quantum 0.80, bose 0.80, chemistry 0.78, bohr 0.76
  monday    friday 0.87, thursday 0.85, weekend 0.84, tuesday 0.84, wednesday 0.83
  car       racing 0.74, cars 0.74, driver 0.70, automobile 0.69, motor 0.68
  three     four 0.95, five 0.93, two 0.91, six 0.91, one 0.89
  pump      ignition 0.88, pedals 0.87, injected 0.87, compartment 0.86, oven 0.86

The loss at the end of each epoch is the mean over that epoch’s last 1,000 batches; the time depends on the machine.

Real word2vec updates one pair at a time, so a frequent word’s vector is refreshed between updates. Batching breaks that: np.add.at accumulates every gradient for a repeated index and applies the sum in one step. In a batch of 2048 pairs, a common word can appear a hundred times, so it receives a hundred stacked updates computed from a vector that never moved.

Without clipping this run diverges to nan within half an epoch — which is exactly what happened the first time it was run for this chapter, at learning rate 0.05.

Here the clip is applied before the contributions of repeated indices are summed. The total gradient of the centre example and the gradient of each context example are restricted to \([-1, 1]\) per coordinate. If a row receives a hundred contributions, each of its coordinates can change by at most \(100 \times \text{lr}\), and its L2 norm in \(d\) dimensions by at most \(100 \times \text{lr} \times \sqrt{d}\). The operation bounds individual contributions; it does not directly bound the accumulated row update.

One option is to divide the gradient of the whole batch by the batch size and choose the learning rate accordingly. Another is to divide each row’s accumulated gradient by the number of contributions its index has in the batch; np.bincount can compute those counts. The second option scales the effective learning rates of the rows differently and is not the same as the standard batch mean. Neither reproduces sequential SGD, which updates the parameters again at every example.

Gradient clipping is an optimization choice made in this implementation, and it can affect the vectors that are learned. Vectorization does not necessarily require clipping; the scaling of the batch loss, the learning rate and the update scheme have to be tuned together. The divergence reported in this example belongs to the run made with the settings chosen.

The link between count-based and prediction-based methods

For a while counting (LSA, PPMI+SVD) and predicting (word2vec) were seen as rival approaches; Baroni et al. (2014) compared the two and reported better results for prediction-based methods. Then Levy and Goldberg (2014b) showed how skip-gram with negative sampling relates to matrix factorization. If the negative contexts are drawn from the context marginal of the training pairs, and if the dot product of every pair can be chosen independently, then for pairs with a positive count the optimal target is the shifted PMI value below. That does not mean every version of word2vec is identical to PPMI–SVD:

\[ \mathbf{v}_w \cdot \mathbf{u}_c = \mathrm{PMI}(w, c) - \log K \]

So prediction-based SGNS and matrix-based methods are closer than they looked: under those conditions the optimal dot products target a shifted PMI. The low-rank constraint, different losses and sampling mean they still do not produce the same embedding in practice.

Note that this is PMI, not the PPMI of route A. When building an explicit matrix we can clip the PMI values that are negative or go to infinity and obtain the shifted positive PMI that Levy and Goldberg propose, \(\mathrm{SPPMI}(w,c) = \max(\mathrm{PMI}(w,c) - \log K,\ 0)\). SGNS does not do that explicitly. A pair whose positive count is zero can still be drawn as a negative; with \(q(c) > 0\) and a positive centre probability, the logistic optimum for its free score lies at the limit \(z \to -\infty\). The actual model works with finite parameters, shared low-dimensional vectors and a finite number of updates, so it does not realize all those limits exactly.

SGNS does not build this matrix explicitly; it makes updates from sampled pairs. Shared low-dimensional vectors may not be able to meet the free optima of all pairs at once. The noise distribution with the 0.75 exponent changes the target as well. For those reasons not every dot product of a converged implementation equals \(\mathrm{PMI} - \log K\) directly.

When \(K\) grows, with the other distributions held fixed, the free dot-product targets shift down by the same constant. That is not in general equivalent to adding the same vector to every word vector. Adding a constant to the target matrix raises its rank by at most one; it can be represented with an extra coordinate if needed, but the same fit may not be preserved at a fixed \(d\). The effect of \(K\) on a task has to be assessed experimentally.

Fix a single pair \((w, c)\) and treat \(z = \mathbf{v}_w \cdot \mathbf{u}_c\) as a free number, not as something built from vectors. Over the whole corpus that pair contributes two terms to the objective: one proportional to its positive count, and one proportional to how often the sampler would produce it as a negative. Assuming the negatives are drawn from the context marginal \(P(c)\), those two weights are \(P(w,c)\) and \(K \cdot P(w)P(c)\). The pair’s contribution to the objective to be maximized is the following; taking the negative of the same expression gives the loss to be minimized:

\[ P(w,c)\,\log \sigma(z) \;+\; K\,P(w)P(c)\,\log \sigma(-z) \]

Differentiate with respect to \(z\), using \(\frac{d}{dz}\log\sigma(z) = 1 - \sigma(z)\) and \(\frac{d}{dz}\log\sigma(-z) = -\sigma(z)\), and set the result to zero:

\[ P(w,c)\bigl(1 - \sigma(z)\bigr) = K\,P(w)P(c)\,\sigma(z) \]

Solve for \(\sigma(z)\), then convert back through \(z = \log \frac{\sigma(z)}{1-\sigma(z)}\):

\[ z = \log \frac{P(w,c)}{K\,P(w)P(c)} = \log \frac{P(w,c)}{P(w)P(c)} - \log K = \mathrm{PMI}(w,c) - \log K \]

This derivation assumes that the negatives are drawn from \(P(c)\) and that \(z\) can be chosen independently for every pair. In the actual low-dimensional model all the dot products are built from shared vectors. Under that constraint SGNS optimizes a logistic loss weighted by sampling. PPMI–SVD, by contrast, minimizes the squared reconstruction error of a matrix whose negative values have been clipped. So the two can differ in their targets as well as in their losses, because of the clipping and the noise distribution.

The assumption that negatives are drawn from \(P(c)\), the raw unigram distribution, does not match the code above. word2vec draws them from \(P_\alpha(c) \propto f(c)^{0.75}\) instead — the exponent from two sections ago — and carrying that through gives

\[ \mathbf{v}_w \cdot \mathbf{u}_c = \log \frac{P(w,c)}{P(w)\,P_\alpha(c)} - \log K \]

which is PMI computed against a smoothed context distribution. Levy et al. (2015) call this context distribution smoothing, and show it transfers straight to PPMI+SVD, where it is worth several points. So the 0.75 exponent is not a sampling detail bolted onto the algorithm: it changes which matrix the algorithm factorizes.

Two routes, one destination
the same corpus ROUTE A · COUNT ROUTE B · PREDICT count co-occurrences in a ±2 word window reweight with PPMI log (joint ÷ independent) truncated SVD keep 300 directions cut into pairs (centre, context) logistic regression real pair, or sampled fake? stochastic gradient descent billions of tiny updates one 300-number vector per word dense, and comparable by cosine at the optimum: v·u = PMI(w, c) − log K

Count-based methods and SGNS use distributional information in different ways. When the negatives are drawn from the context marginal and the dot products are free, the target of SGNS is \(\mathrm{PMI} - \log K\). The PPMI–SVD route in the figure additionally clips at zero and uses a squared loss; it is not expected to give the same vectors at low rank. With 0.75-exponent sampling, the actual noise distribution \(q(c)\) has to replace \(P(c)\) in the equality under the figure.

Levy et al. (2015) followed up by showing that once you give the count-based methods the same hyperparameters word2vec quietly uses — subsampling, context distribution smoothing, dynamic windows — the performance gap mostly disappears.

The lesson generalises well beyond embeddings, and it is worth carrying into every paper you read: when a new method beats an old one, some of the improvement is the method and some is the tuning, and the split is rarely reported.

Which leaves the practical question of when to take which route. Since their targets are close and they land close enough that Levy et al. (2015) could not separate them once the hyperparameters matched, the choice is mostly about engineering:

Close targets, different engineering
Counting (PPMI + SVD) Predicting (word2vec)
Memory Large corpora need a sparse co-occurrence matrix, the low-rank vectors and the solver’s working memory. The small example in this chapter uses a dense matrix for teaching purposes In a streaming implementation the two vector tables take \(O(Vd)\) space; the sampler and buffers need memory too. The NumPy example here stores every pair, so its total memory also depends on corpus size
Corpus size Comfortable on small corpora, where every count is precious Needs a lot of text before the gradient noise averages out
Reproducibility The solver and the environment are fixed; randomized SVD needs a stated seed. The signs of singular vectors, and some choices of basis in subspaces with equal singular values, are not unique Controlling the seed, data order, thread count and environment improves reproducibility; different initializations can give different solutions
New data Counts can be updated incrementally; the reweighting and the factorization need a suitable update method Training can continue if the two tables, the vocabulary and the necessary training state have been kept. Released word vectors alone may not be enough
Inspectability Counts and weighted cells can be inspected directly The sampling distribution, gradients, norms, loss and neighbourhoods can be inspected
Tuning burden Window size, PPMI variant, \(d\) Window, \(d\), \(K\), subsampling \(t\), learning rate, epochs

One more difference does not fit in that table, and it is statistical. Truncated SVD gives the best rank-\(d\) approximation in Frobenius norm (the Eckart–Young–Mirsky theorem), which means it gives an equal coefficient to the squared reconstruction error of every cell of the PPMI matrix. The zeros in PPMI cover pairs that were never observed and also observed pairs whose PMI is zero or negative. The sheer number of zeros does not by itself show that most of the error, or of the model’s capacity, goes to them. SGNS uses a logistic loss under weights set by the positive pair frequencies and the negative-sampling distribution. GloVe chooses yet another weighting, one that depends on the counts, as the next section shows. Those differences affect how the methods behave; they do not prove an unconditional advantage for any of them on rare words.

The choice should weigh corpus size, the need for updates, memory, inspectability and the target task together. PPMI–SVD makes the steps of the computation visible on small examples; streaming SGNS can be convenient for training on large text that keeps arriving. How well either approach does depends on the implementation and the settings used.

3.5 GloVe: counting, with a loss function

GloVe accumulates the co-occurrence counts explicitly and learns the vectors with a weighted loss function defined on those counts. In that way it combines a count-based representation with gradient-based learning.

GloVe’s input is a word–context co-occurrence table. The matrix \(C\) at the start of the chapter added one for every pair inside the window. In the original GloVe implementation distant contexts can be counted with a lower weight, so the cells of \(X\) need not be integers. Both approaches accumulate information from local windows across the corpus.

GloVe learns vectors and bias terms for every word and every context. It asks the sum \(\mathbf{v}_w \cdot \mathbf{u}_c + b_w + b_c\) to be close to the logarithm of the positive co-occurrence count. Closeness is measured by the squared error multiplied by a weight that depends on the pair’s count: weighted least squares. Because the two vector tables are learned together, this is not the same optimization problem, in its parameters, as ordinary linear regression.

More formally: GloVe factorizes co-occurrence counts accumulated from local windows across the whole corpus, using a weighted least-squares objective.

The “accumulated from local windows” part matters, because it corrects a common wrong intuition. It is true that GloVe uses “global statistics”, but the global table does not come from nowhere: it is built from the same local context windows word2vec sees, added up over the entire corpus. The difference is not whether windows are used but when the learning happens: word2vec sees context examples one at a time and makes an update from each; GloVe counts and totals them all first, then optimises against the totals. The original paper presents the method as a combination of the two: bringing together the advantages of global matrix factorization and local context-window methods (Pennington et al. 2014).

Putting the three methods side by side: PPMI+SVD transforms the whole co-occurrence table first and factorizes it afterwards. word2vec does an implicit factorization through local training pairs. GloVe keeps the counts gathered from the windows explicitly and optimises low-dimensional vectors to account for those counts directly.

GloVe (Pennington et al. 2014) took that synthesis seriously. Build the global co-occurrence matrix, then fit vectors so that dot products reproduce log co-occurrence counts, weighting rare pairs down:

\[ \mathcal{L} = \sum_{(w,c)\,:\,X_{wc} > 0} f(X_{wc}) \left(\mathbf{v}_w \cdot \mathbf{u}_c + b_w + b_c - \log X_{wc}\right)^2 \]

This is weighted least squares on log counts, with a bias term per word — about as classical as statistics gets. Take it apart piece by piece, because every term is there for a reason.

Why \(\log X_{wc}\) and not \(X_{wc}\)? Co-occurrence counts can span a very wide range: some pairs are frequent and a great many are rare. A squared error on raw counts can give the pairs with large counts far too much prominence. The logarithm turns multiplicative ratios into additive differences and reduces that difference in scale.

Why the bias terms? The bias terms can model part of the overall usage tendencies of the word and of the context. That carries an intuition similar to the way PMI takes marginal frequencies into account. GloVe learns these terms jointly from the data, however, and they need not equal the log row and column totals. So GloVe’s dot products cannot be read as exactly PMI.

Why \(f(X_{wc})\)? Two problems need solving at once. A pair that never co-occurs has \(\log 0 = -\infty\), and a pair that co-occurs constantly would swamp everything else. GloVe’s weighting function handles both:

\[ f(x) = \begin{cases} (x / x_{\max})^{3/4} & x < x_{\max} \\ 1 & x \geq x_{\max} \end{cases} \qquad x_{\max} = 100 \]

GloVe’s weighting function, at \(x_{\max} = 100\)
\(X_{wc}\) \(f(X_{wc})\) effect on that pair
0 0.00 never enters the objective — no \(\log 0\) problem
1 0.03 counted, but barely
10 0.18 growing weight
100 1.00 full weight
100,000 1.00 also full weight — the cap has bitten

Because cells with a zero count are left out of the sum, \(\log 0\) is never computed. For positive counts \(f(x)\) lowers the weight of rare pairs and stays at 1 for \(x \geq 100\). The weight therefore does not grow without bound as the count grows; the squared residual and the gradient can still be large. In its aim of controlling the influence of frequent pairs this is related to subsampling; it is not the same operation. The 0.75 exponent that appears in both methods does different jobs as well.

For the rest of the chapter we use GloVe’s pretrained vectors as an easily obtained example of a static representation.

import numpy as np
import gensim.downloader as api

g = api.load("glove-wiki-gigaword-100")   # Wikipedia 2014 + Gigaword 5, 6 billion tokens
print("vectors:", len(g.index_to_key), " dimensions:", g.vector_size)
v = g["king"]
print("king[:8] =", np.round(v[:8], 3))
print("norm:", round(float(np.linalg.norm(v)), 3))
vectors: 400000  dimensions: 100
king[:8] = [-0.323 -0.876  0.22   0.253  0.23   0.739 -0.38  -0.353]
norm: 6.118

Every word is now represented by one hundred numbers. No labels such as “royalty” or “gender” were assigned to these coordinates in advance. Some coordinates may turn out to be related to particular properties; such readings depend on the chosen representation, however, and are usually not preserved when the coordinate system changes.

for w in ["king", "pump", "turkey", "python", "bank"]:
    print(f"  {w:<8} " + ", ".join(f"{a} {b:.2f}"
          for a, b in g.most_similar(w, topn=6)))
  king     prince 0.77, queen 0.75, son 0.70, brother 0.70, monarch 0.70, throne 0.69
  pump     pumps 0.84, pumping 0.77, pumped 0.71, gasoline 0.64, fuel 0.63, petrol 0.63
  turkey   ankara 0.75, turkish 0.72, greece 0.71, syria 0.70, bulgaria 0.69, russia 0.69
  python   monty 0.69, php 0.59, perl 0.58, cleese 0.54, flipper 0.51, ruby 0.51
  bank     banks 0.81, banking 0.75, credit 0.70, investment 0.69, financial 0.68, securities 0.67

Three of these deserve a second look.

Among the neighbours of turkey the words about the country stand out, and among those of bank the words about finance. For python, neighbours related to the comedy troupe (monty, cleese) and to programming languages (php, perl) appear together. Which usage relations the flipper result comes from cannot be determined from this list alone. The lists show that a single word vector can carry the influence of different usages together; they are not a complete inventory of all the senses.

This is the defining limitation of a static embedding: one vector per word form, whatever the context.

The Turkish word yüz can mean face, the number hundred, or the imperative swim. A static word table gives that form a single vector; the different usages influence what is learned, but the result is not an explicitly computed arithmetic mean of those senses. Querying the vector by itself cannot select which sense is in use in the sentence at hand. A model that processes this vector together with the surrounding words, on the other hand, can make use of the context.

That limitation is one of the important reasons for the token representations that change with context, covered in Chapter 7.

3.6 Word analogies and the conditions of evaluation

With the king − man + woman example we look at what vector arithmetic shows and at how the result depends on choices made in the computation.

The result that made embeddings famous outside NLP:

res = g.most_similar(positive=["king", "woman"], negative=["man"], topn=5)
  queen        0.7699
  monarch      0.6843
  throne       0.6756
  daughter     0.6595
  princess     0.6521

\(\text{king} - \text{man} + \text{woman} \approx \text{queen}\). Among the candidates allowed, queen is the word with the highest cosine similarity to the target this operation builds. The result indicates that some relations can be represented by similar vector differences. An exact parallelogram is sufficient for the arithmetic to produce the queen vector itself; it is not necessary for queen to be selected as the nearest neighbour. Nor can we conclude from this example that the differences between man and woman, and between king and queen, consist of a single gender direction only.

gensim.most_similar normalizes each input vector to unit length separately and removes the input words from the candidates. To see the effect of the filter alone, rank the same scores with two candidate sets; topn=None returns the unfiltered scores for the whole vocabulary:

sims = g.most_similar(positive=["king", "woman"], negative=["man"], topn=None)
inputs = {"king", "man", "woman"}
order = np.argsort(-sims)
for label, excluded in [("unfiltered", set()), ("filtered", inputs)]:
    chosen = [i for i in order if g.index_to_key[i] not in excluded][:6]
    print(label)
    for i in chosen:
        print(f"  {g.index_to_key[i]:<12} {sims[i]:.4f}")
unfiltered
  king         0.8361
  queen        0.7699
  monarch      0.6843
  throne       0.6756
  daughter     0.6595
  princess     0.6521
filtered
  queen        0.7699
  monarch      0.6843
  throne       0.6756
  daughter     0.6595
  princess     0.6521
  prince       0.6517

With the scores held fixed, the unfiltered nearest neighbour is king (0.8361) and queen is second (0.7699). That is the result of an unrestricted neighbour search; it does not show that the expected answer to the analogy is king. Excluding the input words defines a different candidate set, and in that set queen comes first. The candidate set is part of the definition of the task and has to be stated with the result. If the inputs are added without normalizing them first, the scores change a little (king 0.8552, queen 0.7834); the top of the ranking stays the same.

This scoring is called 3CosAdd: it combines the candidate’s cosines with the three inputs using two plus signs and one minus sign. Excluding the input words is a separate evaluation rule. 3CosMul shifts the cosines suitably and combines them as a multiplicative ratio; it gives better results in some evaluations (Levy and Goldberg 2014a). The scoring method and the candidate filter should be reported separately.

The deeper objection is Linzen (2016)’. He shows that a baseline which ignores the relation entirely — return the nearest neighbour of king, never look at man or woman — already recovers a large share of the reported accuracy on some categories. On this example it would be wrong — king’s nearest neighbour is prince (0.77) — but the GloVe neighbour list above shows how much of the work the structure has already done: queen is already king’s second-nearest neighbour (0.75) before any arithmetic, so the offset only has to move it up one place, and in some categories the nearest neighbour is already the answer. So the headline number measures neighbourhood structure at least as much as it measures a relation. Nissim et al. (2020) show that excluding the input words can also affect how bias analogies are read. In the examples they examine, doctor can become a candidate again once the filter is removed; the model, the normalization and the candidate set a result was obtained with should be stated together. Try a few for yourself:

Analogy explorer — real GloVe results, with and without the filter

Here is what the arithmetic looks like geometrically, projected from 100 dimensions down to 2:

Thirty words, projected to two dimensions

3.7 fastText: words made of pieces

Everything so far shares one assumption we have not questioned: that a word is an indivisible unit with its own row in a table. Which raises a question you already know the answer to if you read Chapter 2 — what happens when a word is not in the table?

out-of-vocabulary (OOV) — a form that is not in the model’s word vocabulary. It may never have been seen in training, or it may have been dropped from the vocabulary for a reason such as a frequency threshold. In languages that produce many inflected forms, such as Turkish, the coverage of a word-level vocabulary matters all the more. ↩︎

For a form that is not in the word table — an out-of-vocabulary word — no row can be fetched directly. The gensim word-vector lookup used here raises a KeyError in that case; other systems may use a special unknown-unit representation. The variety of forms in Turkish makes this coverage problem especially visible.

Here are two Wikipedia samples of comparable size, one Turkish and one English:

### corpora
  turkish   103040 lines    1458213 tokens    138578 types
  english    44079 lines    1828840 tokens     78140 types

Turkish produced 138,578 distinct word forms from 1.46 million tokens; English produced 78,140 from more tokens — 1.83 million. Turkish agglutination means ev, evim, evimde, evimdeki, evlerimizdekiler are five separate entries in a word-level vocabulary, each needing its own vector, each estimated from its own sparse handful of occurrences.

### vocabulary growth: new word forms per million tokens
  turkish  250k:45688  500k:73435  750k:97412  1000k:112644  1250k:126389
  english  250k:22869  500k:34016  750k:42854  1000k:52347  1250k:60178

### the tail: how much of the vocabulary is seen exactly once
  turkish   50.0% of types are hapax (69296 of 138578)
  english   46.7% of types are hapax (36503 of 78140)

### out-of-vocabulary rate: train on 90%, test on the last 10%
  turkish   7.33% of tokens unseen, 28.99% of types unseen
  english   4.72% of tokens unseen, 23.29% of types unseen

In this sample one million tokens contain 112,644 Turkish forms and 52,347 English forms, and new forms keep arriving over the range shown. In the held-out Turkish portion about one token in fourteen falls outside the training vocabulary. That about half the observed types occur once does not mean half the vectors are learned from a single example: a frequency filter can leave those types out, and a single token can produce several context pairs. These numbers belong to one sample, one tokenization choice and one way of choosing the held-out portion; they change on another sample.

subword — a piece smaller than a word. Storing pieces instead of whole words means an unseen word can still be assembled from parts you do know. ↩︎

fastText’s answer (Bojanowski et al. 2017) is to stop treating a word as atomic. Represent it as the sum of its character n-grams — its subwords — plus a vector for the whole word:

\[ \mathbf{v}_w = \mathbf{1}_{\{w \in \mathcal{V}\}}\,\mathbf{z}_w + \sum_{g \in G(w)} \mathbf{z}_g \]

Here \(\mathcal{V}\) is the whole-word vocabulary and \(G(w)\) is the multiset of the selected character n-grams. The separate whole-word component exists only for words in the vocabulary; an out-of-vocabulary word is built from its subword components. The sum describes the composition used in training; how the exported vector is scaled depends on the implementation.

Shared character n-grams can receive updates from examples in different words: <pu is also contained in pump, pumps, pumped, pumping and punch. That lets rare words benefit from forms that are seen more often. Not every piece is learned from plentiful data, however. fastText maps the n-grams to a fixed number of hash-table buckets, and different pieces can share a bucket. That a vector can be produced for an out-of-vocabulary word does not by itself show that the vector is semantically reliable.

The sum in the paper and the mean of the same learned components, divided by their number (what the fastText and gensim implementations do), give the same direction for a non-zero word vector; the word-to-word cosines of the two versions are identical. That should not be read as saying that using a sum or a mean in training, or combining different words into a document vector, is equivalent under all conditions.

Shared n-gram vectors tie together the representations of words that are spelled alike. That trade-off can provide morphological generalization, and it can also create unwanted closeness between unrelated words. The separate whole-word component of an in-vocabulary word can learn information apart from that effect. The contribution of a given n-gram is shared among the words that contain it.

fastText: a word is the sum of its pieces
pumps — seen in training <pu pum ump mps ps> <pumps> 5 n-grams + the word + v(pumps) word2vec gets here too pumpage — never seen, not in any vocabulary <pu pum ump mpa pag age ge> <pumpage> no whole-word vector — the n-grams carry it alone + v(pumpage) word2vec: KeyError The three teal pieces are shared. That overlap is the whole trick: an unseen word inherits meaning from the words it is spelled like.

Every word is a bag of character n-grams plus, if the model ever saw it, the word itself. Rare and unseen words are no longer a special case — which is why this is the default choice for morphologically rich languages like Turkish. Drawn with 3-grams so it stays readable; fastText’s default is every n-gram of length 3 to 6, so a real pumpage decomposes into 22 pieces, not 7.

Character n-grams — see the pieces of the chosen length, and the shared ones

Trained on the same Turkish Wikipedia sample, word2vec and fastText behave very differently on words the corpus never contained:

### a word the corpus never saw
  kitaplarımızdaki   in word2vec vocab: False   in fastText vocab: False
    fastText still answers: kitapları 0.91, kitapların 0.91, kitaplıkları 0.90, kitaplarda 0.90
    word2vec raises: KeyError
  bilgisayarcılık    in word2vec vocab: False   in fastText vocab: False
    fastText still answers: bilgisayara 0.96, bilgisayarlı 0.96, bilgisayar 0.96, bilgisayarı 0.96
    word2vec raises: KeyError

Shared character pieces can be useful for some typos as well. In the examples below the out-of-vocabulary forms get vectors close to the expected words; that result is not guaranteed for every typo.

  philosphy       in vocab: False
    philosophy 0.96, philosophie 0.95, philosophies 0.94, philology 0.94, philo 0.93
  governmentt     in vocab: False
    government 0.98, governments 0.94, governmental 0.93, governs 0.91, govern 0.91

The price of subwords

Character n-grams make it possible to share some stem and suffix patterns without an explicit morphological analyser. In return, the representations of words that are spelled alike but mean different things can move closer together too.

### morphology: does fastText put inflected forms together?
  ft  kitap    kitapta 0.96, kitaba 0.95, kitab 0.94, kitabın 0.94, kitabı 0.93
  w2v kitap    eseri 0.81, makale 0.80, eseriyle 0.80, kitabın 0.80, kitaplar 0.79

  ft  yaz      yao 0.78, yağ 0.77, yağdı 0.77, yayoi 0.76, yay 0.75
  w2v yaz      kış 0.86, olimpiyatları 0.84, aylarında 0.83, gündönümü 0.80

For kitap (book), fastText returns inflections of kitap; word2vec returns eseri (his/her work), makale (article) — semantic neighbours. Which one you want depends entirely on your task.

In this run the word2vec neighbours of yaz (summer, and also “write”) reflect the seasonal usage more: kış (winter), olimpiyatları (olympics), gündönümü (solstice). In the fastText list, forms with similar spelling come to the front: yao, yağ (oil), yayoi, yay (bow). The observation shows that sharing characters can create unwanted closeness. To say that the difference comes only from the word being three letters long, however, one would have to compare n-gram settings, frequency and different training seeds.

### the price, measured
  kitap    kitle    fastText +0.667   word2vec +0.446

In this example the cosine of kitap (book) and kitle (mass, crowd) is 0.667 in fastText and 0.446 in word2vec. The cosine distributions of the two spaces can differ, so the ratio cannot be read as “half again as much semantic similarity”. The score distributions of unrelated or hard-negative pairs would have to be compared as well.

The rule to remember. fastText is a strong candidate for tasks where out-of-vocabulary forms and inflectional variety matter. Character sharing can also create unwanted closeness, however; evaluate the method by coverage and by task performance, not by accepting or rejecting it unconditionally according to word length. The idea it shares with BPE, which comes later (Section 8.3.1), is to benefit from rare wholes through their pieces. The mechanism differs: BPE determines the tokenization units, whereas fastText builds a word vector by combining the vectors of overlapping character n-grams.

3.8 Documents as vectors

Chapter 2 classified 20 Newsgroups posts with TF-IDF. Now we have word vectors — do they help? The question is how to produce a coordinate for a document when what we have is a coordinate per word. The obvious answer is the mean, the centroid of the word vectors in a document, which gives a 100-dimensional document vector instead of a 32,000-dimensional sparse one.

def average(docs):
    # docs: a list of documents, each one a list of tokens (not a raw string).
    # Out-of-vocabulary tokens are skipped; a document with no vector stays a zero row.
    out = np.zeros((len(docs), g.vector_size), dtype=np.float32)
    for i, tokens in enumerate(docs):
        if isinstance(tokens, str):
            raise TypeError("each document must be a list of tokens, not a raw string")
        vs = [g[w] for w in tokens if w in g.key_to_index]
        if vs:
            out[i] = np.mean(vs, axis=0)
    return out

Tokenization and case conversion have to be applied consistently to the training and the test documents. Cosine similarity is undefined for zero vectors, so a search application also needs an explicit empty-representation rule. In the experiment below, 79 of the 2,327 training documents and 65 of the 1,550 test documents stay zero rows.

A step up from the plain mean is to weight it by IDF:

\[ \mathbf{v}_d = \frac{\sum_{i \in I_d} \mathrm{idf}(w_i)\, \mathbf{v}_{w_i}} {\sum_{i \in I_d} \mathrm{idf}(w_i)} \]

Here \(I_d\) is the set of token positions in the document that have a word vector; repeated words contribute at every position. IDF is computed from the training documents only and held fixed at test time. If the denominator is zero, an explicit empty-representation rule applies, as for the plain mean. As in Chapter 2, this weighting reduces the relative contribution of words that occur in many documents. The variant appears in the table below as idf-weighted averaged glove.

doc2vec (Le and Mikolov 2014) assigns a learnable vector to the document id as well. Training is based on predicting target words; output objectives such as negative sampling or hierarchical softmax can be used. It has two basic architectures:

The two doc2vec variants
Variant Input Predicts
PV-DM document vector + context words the target word
PV-DBOW the document vector alone words sampled from the document

Loosely, they are the CBOW and skip-gram of the document world — the input–output columns show they are not literally the same models.

The original work favours PV-DM, while Lau and Baldwin (2016) found that PV-DBOW can also be strong, and computationally convenient, on the tasks they examined. In gensim, dm=1 (the default) selects PV-DM and dm=0 selects PV-DBOW. On reproducibility, a follow-up written with one of the original authors is instructive (Mesnil et al., 2014). Its final version (v7) reports 88.73% accuracy with paragraph vectors alone, against the 92.58% IMDB accuracy (7.42% error) of the original paper, and states that 92.6% appeared only when the training and test data were not shuffled. That is the result of one particular replication; it is not a verdict on every replication of the method.

doc2vec: give the document its own vector, then reuse word2vec
PV-DM = CBOW + a document slot doc #42 the pump failed average 300 numbers during document + context next word The document vector is just another input row, trained by the same gradient. It ends up holding whatever the context words did not. PV-DBOW = skip-gram, document as centre doc #42 look up d 300 numbers pump failed shift night document only words drawn from it No word inputs and no averaging at all — cheaper, and often the stronger of the two. Word order is discarded, exactly as in a bag.

Every training document is assigned a learnable vector. For a new document the model’s word and output parameters are held fixed and the new document vector is estimated with a few optimization steps. That requires the trained model, which can run inference, and not only the saved document vectors. This extra computation is an important difference from averaging word vectors directly.

Same four categories, same train/test split, same logistic regression on top.

### documents as vectors: five representations
tf-idf (chapter 2 baseline)        accuracy 0.853   dimensions   32014
averaged glove                     accuracy 0.820   dimensions     100
idf-weighted averaged glove        accuracy 0.822   dimensions     100
doc2vec (PV-DBOW, 40 epochs)       accuracy 0.839   dimensions     100
averaged glove + doc2vec           accuracy 0.845   dimensions     200

A single accuracy computed from 1,550 documents carries uncertainty. For example, the standard error of an accuracy near 0.83 is about 0.010 under the binomial approximation; that value is not, however, the standard error of the difference between the accuracies of two models. To decide whether gaps of 0.2, 1.9 or 3.3 points are significant, the predictions on the same documents have to be examined as pairs and, where possible, results reported on different splits. The note below gives that paired comparison.

McNemar’s test can be used to compare the accuracies of two models that classify the same test documents. The test compares the counts, in the two directions, of documents that one model gets right and the other gets wrong. Predicting different wrong classes does not by itself enter that disagreement. An effect size and a paired uncertainty interval should be reported too, and the variability across different splits examined separately.

The counts measured on this split:

Paired comparison on the same 1,550 test documents
Comparison Only the first right Only the second right Exact McNemar \(p\)
TF-IDF – averaged GloVe 135 84 0.0007
TF-IDF – IDF-weighted averaged GloVe 129 81 0.0011
IDF-weighted – plain averaged GloVe 66 63 0.86

The Bonferroni threshold for three comparisons is \(0.05 / 3 \approx 0.017\); the first two differences fall below it and the 0.2-point gap between the two GloVe averages does not. We cannot give the same calculation for the doc2vec rows: gensim’s infer_vector derives its starting vector from a hash value that changes in every Python session, so the doc2vec predictions behind the table cannot be reproduced exactly. With a fixed hash, the chapter notebook finds doc2vec accuracies between 0.830 and 0.836 over three seeds.

On this split the highest accuracy is in the TF-IDF row. The dense representations reach accuracies close to the 32,014-dimensional sparse one using 100 or 200 coordinates. The paired predictions show that the gap between TF-IDF and the two GloVe averages is hard to explain by chance on this split (the note above); the same calculation could not be made for the gaps to the doc2vec rows. A reduction in the number of coordinates does not translate directly into the same ratio of memory saved; sparse storage, running time and the search method to be used have to be measured as well. A small classification task and large-scale document search can call for different choices.

In topic classification the mean of the word vectors can be a useful starting representation. The SIF approach of Arora et al. (2017) uses weights \(a / (a + p(w))\) that depend on word frequency and removes a common dominant component. That is not the same method as the IDF-weighted mean above; it is a separate baseline that can be compared on tasks such as sentence similarity.

Two more measurements before leaving this. First, coverage:

### coverage: how much of the corpus does glove actually know?
  tokens covered: 98.6%
  types covered : 84.5%  (23395 of 27681)

In this measurement GloVe covers 98.6% of the tokens and 84.5% of the types. The forms it does not cover make up 1.4% of the token mass, which shows that the group is rare overall. How many of them are typos, proper names or domain terms has to be examined separately. The averaging method used here leaves those forms out.

Second, the failure that no amount of averaging can fix:

a = "the pump failed and the valve is fine".split()
b = "the valve failed and the pump is fine".split()
va = np.mean([g[w] for w in a], axis=0)
vb = np.mean([g[w] for w in b], axis=0)
### what averaging destroys
  cosine of the two averaged vectors: 1.0000
  identical: True

The same tokens occur the same number of times in the two sentences, so the unordered mean is mathematically the same vector. In floating-point arithmetic the order of addition can produce very small rounding differences; bit-level equality is not guaranteed in every implementation. The identical: True here is the result of an np.allclose comparison; in the same run the bit-level comparison (np.array_equal) gives False, and the largest difference is \(1.2 \times 10^{-7}\). The basic limitation does not change: averaging does not keep the word order that determines which part failed. This is chapter 2’s bag-of-words problem, now with 100 dimensions instead of 32,000.

Word order needs a model that reads the sequence. That is Chapter 6 and Chapter 7.

3.9 The geometry has problems

Reading static embeddings means keeping distributional relatedness, the distribution of directions and the effect of frequency apart. After that we look at the arbitrariness of the coordinate system and at how consistent different training runs are.

In this space, cosine reflects relatedness more than similarity

  good       bad        0.770
  hot        cold       0.725
  increase   decrease   0.881
  cat        dog        0.880
  cat        car        0.311
  teacher    student    0.808

The pair increase–decrease scores 0.881 and the pair cat–dog 0.880. We do not have enough evidence to read that small difference as a meaningful ordering. The important observation here is that the antonym pair gets a high cosine value too.

This is not a bug in GloVe; it is the distributional hypothesis doing exactly what it promised. Antonyms are largely substitutable in context — sales increased sharply and sales decreased sharply are both fluent — so they keep identical company and land in the same place.

This is called the antonym problem, and its one-sentence definition is: words with opposite meanings do not sit far apart in embedding space; they sit close together. Take hot and cold. Intuition says “opposite meaning, opposite direction”; but the model never sees meaning, only which words appear around a word. hot and cold both appear next to water, weather, very, too — almost the same circle of friends — and to the model, having the same circle is what being close means. The practical cost: a system searching for “hot” may return documents containing “cold” among its most similar results.

Task-specific labels, for contradiction or synonymy for example, can help a model learn this distinction, and what is learned can generalize to pairs that were not seen directly in training. Success still depends on the coverage of the data and on the training objective; using such labels does not by itself guarantee that every antonym pair will be separated.

The rule to remember. Words that keep the same company end up close distributionally, so proximity does not mean synonymy. Cosine itself is a geometric measure — what it carries depends on the objective that trained the embedding. In this space it mostly carries relatedness, and similarity and oppositeness are both kinds of it.

Steck et al. (2024) show that, in certain linear matrix-factorization models, the cosines can change while the training predictions are preserved. The result draws attention to the fact that cosine is not a scale of meaning independent of the learned representation. The distribution over random pairs gives a useful basis for comparison; assessing similarity for a task additionally requires suitable labelled examples. Do not treat a value of 0.8 in two models as directly equivalent.

Do not read the raw cosine of static vectors directly as a measure of sentiment or polarity. Evaluation data measure different relations as well: WordSim-353 (Finkelstein et al. 2002) contains judgements that also reflect broad semantic relatedness, whereas SimLex-999 (Hill et al. 2015), developed later, specifically aims to separate similarity from relatedness. The scores models get on the two sets come up in the evaluation section.

The space is not isotropic

The word itself first. Isotropic means “the same in every direction”. Picture every word vector as an arrow leaving a common centre. In an isotropic space those arrows spread evenly in all directions, like the spines of a hedgehog: as many point north as south, as many up as down. Pick two arrows at random and the angle between them is, on average, a right angle, so their cosine averages zero. That is what we expect of “two unrelated words”.

Anisotropic means that the distribution of the vectors is not the same in every direction. A shift toward a common direction is one form of it and can raise the mean cosine. Such a shift is not required for anisotropy, however: the vectors can concentrate in the two opposite directions of one axis and still have a zero mean.

The cosine distribution over random pairs helps to judge how unusual a particular score is. A fixed value such as 0.3 does not automatically indicate a semantic relation, even in an isotropic space; the dimension and the task matter too. Below we examine the pairs among 4,000 vectors drawn from the first 50,000 words of the GloVe vocabulary.

rng = np.random.default_rng(0)
M = g.get_normed_vectors()                       # (400000, 100), unit rows
sample = rng.choice(50_000, size=4000, replace=False)   # top 50k only
S = M[sample]
vals = (S @ S.T)[np.triu_indices(4000, k=1)]
  mean cosine between random word pairs : 0.029
  std                                   : 0.119
  share of pairs with cosine > 0        : 56.9%
  norm of the mean vector               : 0.171
  after subtracting the mean, mean cosine: -0.000

In this sample 56.9% of the pairs have a positive cosine, the mean cosine is 0.029 and the norm of the mean of the unit vectors is 0.171. Those values are consistent with a shared mean direction; they do not by themselves show that the whole space is squeezed into a narrow cone. The mean cosine reported after centring is approximately zero; that it should be exactly zero is not a general identity. The results belong to the sample drawn from the first 50,000 words. The practical consequence: do not read raw cosine values as if they were on the same scale across models.

Be precise about the null, because the obvious phrasing is wrong: for independent unit vectors drawn uniformly in \(d\) dimensions the angles do not spread evenly — they concentrate near 90°, with mean cosine 0 and variance \(1/d\). Isotropy means no direction is special, not that every angle is equally likely.

The 56.9% figure proves little on its own: under an isotropic null the share would be 50%, and the 8 million pairs behind it come from 4,000 vectors, so they are nowhere near independent observations.

The mean cosine and the norm of the mean vector are not two independent pieces of evidence. For the distinct pairs among \(n\) unit vectors the identity \(\bar{c} = (n\lVert\bar{v}\rVert^2 - 1)/(n-1)\) holds. With \(n = 4000\) and the rounded \(\lVert\bar{v}\rVert = 0.171\) the result is approximately 0.029. These summaries do not by themselves determine every detail of the distribution of directions. Note also what centring does not buy: a zero mean removes the shared direction, it does not make the covariance spherical.

Centring removes the mean direction, and removing the top few components can reduce the dominant directions as well. The all-but-the-top approach (Mu et al. 2018) combines those operations. Useful information can be thrown away too, so task performance should be measured before and after. Ethayarajh (2019) observes strong anisotropy in certain layers of the contextual models he examines; that result should not be read as a fixed ordering for every method of contextual representation.

Cosine between 8 million random word pairs, before and after removing the mean

Frequency leaks in

An embedding is supposed to encode meaning. It also encodes how often the word appeared: the training objective cannot fully separate frequency from geometry. Frequent words take part in more co-occurrence events and therefore receive more, and denser, training signal.

### mean vector norm by frequency rank (GloVe, 400k words)
  rank      0-100     mean norm  6.19
  rank    100-1000    mean norm  5.85
  rank   1000-10000   mean norm  5.45
  rank  10000-100000  mean norm  4.99
  rank 100000-400000  mean norm  3.62
  spearman(log rank, norm) over the top 100k words: -0.313

Average norm falls steadily across frequency bands as words get rarer — a trend across bands, not a monotone relation for every individual word. This table shows that the norm can be related to frequency; it does not show that cosine is the best measure for every task. A vector has two separate properties: its direction and its length (norm). Cosine leaves the norm out and compares directions; Euclidean distance is affected by both direction and length. For unit-norm \(a\) and \(b\), \(\lVert a - b \rVert^2 = 2 - 2\cos(a, b)\) (the identity from Chapter 2), so the two measures give the same ranking. Whether keeping the norm or removing it helps has to be assessed on the task.

Raw dot products and Euclidean distance can be affected by differences in norm. A model that takes unnormalized vectors can also make use of this frequency-related information; how far it does so depends on the model’s training objective and its parameters.

Look at the neighbour lists of rare words as well; your domain vocabulary mostly lives in this tail:

  water        rank    430  natural 0.70, dry 0.68, salt 0.68, clean 0.68, drinking 0.67
  turbine      rank  15747  turbines 0.80, compressor 0.78, generator 0.72, engine 0.70
  cavitation   rank 109044  aliasing 0.56, barotrauma 0.54, attenuation 0.54, breakage 0.53
  impeller     rank 124835  armature 0.62, evaporator 0.61, nozzle 0.60, centrifugal 0.59
  eutectic     rank 159022  azeotrope 0.61, azeotropic 0.55, tiwai 0.54, bifurcation 0.52

The neighbours in the turbine list look related to equipment and energy systems. For the cavitation and eutectic lists, which relations are useful for the intended task has to be judged with domain knowledge. Being related within the same physical process is not the same as being synonymous; no general quality ranking can be drawn from these short lists.

Evaluate domain terms separately. Words that are frequent in a general corpus may be better represented than the rare terms specific to your domain; that is not a rule for every word, however. Examining the neighbours of your important domain terms is a good first check. After that, compare pretrained, domain-adapted and subword representations with task metrics; a few neighbour lists do not settle the choice of method by themselves.

The arbitrariness of the coordinate system and the variability between training runs

We train two models on the same text8 slice, changing only the random seed. sents is the 6,794,527 tokens the from-scratch model also used, cut into 1,000-token pieces. workers=1 reduces the variability that comes from thread ordering. Exact reproduction also requires fixing, along with the seed, the data order, PYTHONHASHSEED, the library versions and the execution environment.

from gensim.models import Word2Vec
from scipy.linalg import orthogonal_procrustes

def train(seed):
    return Word2Vec(sents, vector_size=100, window=5, min_count=20, sg=1,
                    negative=5, sample=1e-3, epochs=3, workers=1, seed=seed).wv

a, b = train(1), train(2)                    # same corpus, same settings, two seeds
A, B = a.get_normed_vectors(), b.get_normed_vectors()

raw = np.mean(np.sum(A * B, axis=1))         # same word, same coordinates?
R, _ = orthogonal_procrustes(A, B)           # best orthogonal alignment; may include a reflection
aligned = np.mean(np.sum((A @ R) * B, axis=1))

Q, _ = np.linalg.qr(rng.standard_normal((100, 100)))   # a random orthogonal transformation
rotated = A @ Q
gensim 4.4.0; tokens read: 6,794,527
  seed 1: vocab 18,677, 29 s
  seed 2: vocab 18,677, 29 s

### top-10 neighbour overlap between two seeds
  most frequent 1,000    mean count     4,892   overlap  62.2%
  ranks 5,000-6,000      mean count       116   overlap  58.2%
  rarest 1,000           mean count        20   overlap  46.6%

### are the coordinates comparable?
  mean cosine, same word, raw coordinates      : -0.016
  mean cosine, same word, after one rotation   : +0.951

### a rotation changes every coordinate and no cosine
  king, first 4 coordinates, before : [-0.004 -0.045  0.063 -0.044]
  king, first 4 coordinates, after  : [ 0.086 -0.046 -0.114  0.104]
  cos(king, queen) before 0.6826   after 0.6826
  largest change in any of 4,000,000 cosines: 5.5e-07

The code above is the core of the experiment; the lines that print the output and compute the neighbour overlap are left out. The script that produced the whole output is this:

"""Two word2vec runs that differ only in the seed: how different are they?"""
import gzip, time
from collections import Counter
import numpy as np
import gensim, gensim.downloader as api
from gensim.models import Word2Vec
from scipy.linalg import orthogonal_procrustes

with gzip.open(api.load("text8", return_path=True), "rt") as f:
    raw = f.read(40_000_000).split()
print(f"gensim {gensim.__version__}; tokens read: {len(raw):,}")
sents = [raw[i:i + 1000] for i in range(0, len(raw), 1000)]

def train(seed):
    t0 = time.time()
    m = Word2Vec(sents, vector_size=100, window=5, min_count=20, sg=1, negative=5,
                 sample=1e-3, epochs=3, workers=1, seed=seed)
    print(f"  seed {seed}: vocab {len(m.wv):,}, {time.time() - t0:.0f} s")
    return m.wv

a, b = train(1), train(2)
vocab = a.index_to_key                      # sorted by frequency, same for both
assert vocab == b.index_to_key
A, B = a.get_normed_vectors(), b.get_normed_vectors()

def overlap(words, k=10):
    out = []
    for w in words:
        na = {x for x, _ in a.most_similar(w, topn=k)}
        nb = {x for x, _ in b.most_similar(w, topn=k)}
        out.append(len(na & nb) / k)
    return float(np.mean(out))

rng = np.random.default_rng(0)
V = len(vocab)
bands = [("most frequent 1,000", range(0, 1000)),
         ("ranks 5,000-6,000", range(5000, 6000)),
         ("rarest 1,000", range(V - 1000, V))]
print("\n### top-10 neighbour overlap between two seeds")
for name, r in bands:
    ws = [vocab[i] for i in rng.choice(list(r), size=300, replace=False)]
    cnt = [a.get_vecattr(w, "count") for w in ws]
    print(f"  {name:<22} mean count {np.mean(cnt):>9,.0f}   overlap {100 * overlap(ws):5.1f}%")

print("\n### are the coordinates comparable?")
same = float(np.mean(np.sum(A * B, axis=1)))
print(f"  mean cosine, same word, raw coordinates      : {same:+.3f}")
R, _ = orthogonal_procrustes(A, B)          # best orthogonal alignment; may include a reflection
aligned = float(np.mean(np.sum((A @ R) * B, axis=1)))
print(f"  mean cosine, same word, after one rotation   : {aligned:+.3f}")

print("\n### a rotation changes every coordinate and no cosine")
Q, _ = np.linalg.qr(rng.standard_normal((100, 100)))
i, j = vocab.index("king"), vocab.index("queen")
print(f"  king, first 4 coordinates, before : {np.round(A[i][:4], 3)}")
print(f"  king, first 4 coordinates, after  : {np.round((A @ Q)[i][:4], 3)}")
print(f"  cos(king, queen) before {A[i] @ A[j]:.4f}   after {(A @ Q)[i] @ (A @ Q)[j]:.4f}")
S = rng.choice(V, size=2000, replace=False)
d = np.abs(A[S] @ A[S].T - (A @ Q)[S] @ (A @ Q)[S].T).max()
print(f"  largest change in any of 4,000,000 cosines: {d:.1e}")

A single dimension has no meaning by itself. Look at the last block of the output. We multiplied every vector by the same random rotation matrix: every coordinate of king changed, but the cosine between king and queen stayed the same to four decimal places, and the largest change in any of four million cosines was \(5.5 \times 10^{-7}\) — floating-point error. The reason is simple: rotating a space does not change the angles between its arrows. For an embedding matrix \(E\), \(ER\) (for any orthogonal \(R\)) preserves every dot product and every cosine.

To preserve every dot product of SGNS we multiply the centre table and the context table by the same orthogonal transformation: \((WR)(UR)^\top = WU^\top\). The training objective is then unchanged. For that reason the name of a single coordinate is not something the model is forced to determine. A coordinate may turn out to be related to a property in a particular training run; that reading may not survive an equivalent change of coordinates. Meaning is not in the individual coordinates but in where the vectors sit relative to one another — in angles and distances.

Coordinates from two training runs cannot be compared directly. The mean direct cosine of the same words in the two runs is −0.016; after orthogonal (Procrustes) alignment it is +0.951. That shows the choice of coordinates accounts for an important part of the difference; it does not show that the two spaces are merely rotated copies of each other. The neighbour overlaps indeed show that local differences remain. When comparing different periods or corpora, the alignment should use shared anchor words that are as stable as possible, and care is needed not to absorb the change under study into the alignment itself.

Embedding training is stochastic. Same corpus + different seed → geometrically different but functionally similar spaces. The first block shows what that costs: of the 10 nearest neighbours the two runs give for a word, 62.2% are shared on average for the 1,000 most frequent words, and only 46.6% for the 1,000 rarest. The overall shape is stable (+0.951), but the fine local ordering is not — and less so for rare words.

What not to do: build an argument on one neighbour list from one run, especially for a word that occurs rarely. If a neighbourhood matters to you, train with a few seeds and look at which neighbours come back every time.

3.10 Bias is in the data, and the geometry records it

Project occupation words onto the woman − man axis:

axis = g["woman"] - g["man"]
axis /= np.linalg.norm(axis)
occupations = ["doctor", "nurse", "engineer", "teacher",
               "programmer", "homemaker", "scientist", "receptionist"]
for w in occupations:
    proj = float(g[w] @ axis / np.linalg.norm(g[w]))
    print(f"  {w:<13} {proj:+.3f}  ->  {'woman' if proj > 0 else 'man'}")
  doctor        +0.109  ->  woman
  nurse         +0.329  ->  woman
  engineer      -0.123  ->  man
  teacher       +0.152  ->  woman
  programmer    -0.083  ->  man
  homemaker     +0.363  ->  woman
  scientist     -0.046  ->  man
  receptionist  +0.281  ->  woman

The sign shows the direction of the projection onto the difference direction we chose. It does not by itself measure every dimension of social bias, or the real-world distribution of an occupation.

nurse, homemaker and receptionist sit on the woman side; engineer, programmer and scientist on the man side. No one put this in. The model learned these associations from its training data, Wikipedia and news text from 2014; the numbers are what those associations look like when projected onto the one axis we chose. Bolukbasi et al. (2016) estimate a direction, or a subspace, from several defining word pairs and not from a single pair. The single-pair calculation here is a teaching simplification of the method. Caliskan et al. (2017), with WEAT, find relations similar to various human association patterns in the word sets they examine. Reaction times in people and relations between vectors are different measurements; the effect sizes should not be read as the same quantity.

Three lessons for applications follow from this example.

  1. This is usually not a training error. Systematic associations in the training data can carry over into the geometry of the embedding. Used deliberately, embeddings are a quantitative instrument for studying bias in text at scale — but what you measure is a projection produced jointly by the corpus, the model and the axis you chose.
  2. It can carry over into decisions. A system that uses these vectors can be affected by the unwanted associations the representation carries. The effect depends on the downstream model’s training objective, on the other inputs and on the decision rule; the outputs of the application have to be evaluated along with the test on the representation.
  3. A single intervention does not remove every indicator of bias. The method of Bolukbasi and colleagues includes neutralize and equalize steps for selected words (Bolukbasi et al. 2016). Gonen and Goldberg (2019) show that gender-related information can remain in the neighbourhood and classification structure after interventions of this kind. That does not say the information was redistributed by the operation; it says that a fall in the chosen metric does not show that all the related structure has been erased.

No, and the reasoning is the useful part. This test measures one direction defined by one word pair. Bias that is not aligned with woman − man is invisible to it, and Gonen and Goldberg (2019) demonstrated exactly that failure mode for debiased vectors: they pass the projection test and fail a clustering test on the same data.

The general principle is that passing a single fairness metric is evidence only about the kind of bias that metric can detect; it does not show that other forms of bias are absent. Report which test you ran, what it can detect, and what it structurally cannot.

3.11 How do you know an embedding is any good?

Intrinsic and extrinsic evaluation measure different aspects of a representation. Each can be summed up in one question.

Two questions.

  • Intrinsic: does the geometry of the space itself agree with human judgements?
  • Extrinsic: does this representation improve the task I actually have?

Intrinsic evaluation asks whether the space matches human judgements directly: correlation with human similarity ratings, or accuracy on the analogy set. It is fast and requires no downstream task.

Extrinsic evaluation asks whether the embedding improves the thing you actually care about — accuracy on your classifier, recall of your search engine. It is slow, and it is the only one that pays.

from gensim.test.utils import datapath
# gensim defaults: restrict_vocab=300000, case_insensitive=True.
# In the word-pair evaluation, pairs containing a word outside that limit
# are skipped by default; no search for a candidate answer takes place.
# In the analogies, both the questions covered and the candidate vocabulary
# searched are limited. Report the number of examples scored and the coverage
# together with the results.
pear, spear, oov = g.evaluate_word_pairs(datapath("wordsim353.tsv"))
pear_s, spear_s, oov_s = g.evaluate_word_pairs(datapath("simlex999.txt"))
score, sections = g.evaluate_word_analogies(datapath("questions-words.txt"))
### two similarity benchmarks that disagree on purpose
  WordSim-353 (relatedness)    spearman 0.533   pearson 0.548   oov 0.0%
  SimLex-999 (similarity)      spearman 0.296   pearson 0.324   oov 0.1%

### the Google analogy set, 19544 questions in 14 categories
  overall accuracy: 0.633
    capital-common-countries    0.939  (506 questions)
        capital-world               0.889  (4524 questions)
    currency                    0.152  (808 questions)
    city-in-state               0.308  (2467 questions)
    family                      0.816  (506 questions)
    gram1-adjective-to-adverb   0.244  (992 questions)
        gram2-opposite              0.201  (812 questions)
    gram3-comparative           0.791  (1332 questions)
    gram4-superlative           0.543  (1122 questions)
    gram5-present-participle    0.695  (1056 questions)
    gram6-nationality-adjective 0.879  (1599 questions)
    gram7-past-tense            0.554  (1560 questions)
    gram8-plural                0.720  (1332 questions)
    gram9-plural-verbs          0.584  (870 questions)

### the same benchmarks on the vectors trained from scratch above
  WordSim-353                  spearman 0.627   oov 11.9%
  SimLex-999                   spearman 0.268   oov 13.4%
  Google analogies             accuracy 0.200
  (trained on 6.8M tokens; glove saw 6 billion)

The category counts add up to 19,486, whereas the file has 19,544 questions. All 58 questions in between contain the word denar. It is in GloVe’s 400,000-word vocabulary (frequency rank 394,434), but it falls outside gensim’s default restrict_vocab=300000 limit, so those questions are skipped. Being skipped therefore does not mean the word is missing from the model’s full vocabulary.

The per-category breakdown shows differences that the overall score hides: the capital categories are high (capital-common-countries 0.939, capital-world 0.889), while currency (0.152) and gram2-opposite (0.201) get lower accuracy. Those differences can be affected by corpus coverage, the type of relation and the scoring method; they do not by themselves prove which cause dominates. The overall analogy accuracy should be reported together with the category scores, the coverage and simple baselines. Even the scores in the low categories are far above what random choice from a set of 300,000 candidates would give.

The same dataset name does not mean the same evaluated examples. The NumPy model scores 0.627 on WordSim-353 and GloVe 0.533; but 11.9% of the pairs are skipped for the NumPy model, while GloVe covers all of them. The two scores were therefore not computed on the same sample. Measured on the 311 pairs (88.1%) that both models cover, GloVe scores 0.549 and the NumPy model still 0.627: the coverage difference moves GloVe’s score by 0.016 and does not close the gap. On the 42 pairs the NumPy model skips, GloVe’s correlation is 0.473, so those pairs are somewhat harder than average for GloVe. This result on a single relatedness set does not show that the small model is better in general; the analogy numbers below say the opposite.

In the analogy evaluation the default dummy4unknown=False skips questions that contain an out-of-coverage word. The denominators of 0.633 and 0.200 therefore differ: GloVe is scored on 19,486 questions and the NumPy model on only 9,282. dummy4unknown=True counts the skipped questions as wrong and so reports performance on the full question set; it does not give the model vectors for unknown words. With that setting GloVe scores 0.631 (12,334 of 19,544 questions correct) and the NumPy model 0.095 (1,852 of 19,544 correct). The candidate vocabulary, case sensitivity and the input filter should be reported in the same way as well. The score on the full question set and the comparison on the questions both models cover give different, complementary information.

The rule to remember. Give the coverage and the number of examples scored together with the evaluation score. Skipped examples can change the results; the direction of the change depends on the nature of the examples skipped. Do not compare scores computed on different subsets of examples as if they had the same coverage.

The WordSim-353 and SimLex-999 correlations differ here (Spearman 0.533 and 0.296); but the coverage of the datasets and the kinds of judgement they contain differ too, so the gap cannot be explained by antonymy alone. Intrinsic evaluations are useful for comparing particular properties of a representation and for eliminating candidates. For choosing what to deploy, performance and cost on the target task should decide; intrinsic rankings have been shown not to predict extrinsic rankings every time (Schnabel et al. 2015).

3.12 Choosing between them

Four methods, all producing a dense vector per word, all comparable by cosine. The differences only matter in specific circumstances — so here are the circumstances.

Static word embedding methods
Strengths Weaknesses Reach for it when
PPMI + SVD The counts and the transformations can be inspected directly; an instructive starting point on small corpora. With a fixed solver and environment the results are reproducible, though sign and equal-singular-value subspace ambiguities remain. Needs the sparse count matrix plus the low-rank factors and the solver’s working memory. Which of them is the largest object depends on the data. No row is directly available for out-of-vocabulary forms. You want to inspect the steps of the computation, or to set up a count-based baseline on a modest vocabulary.
word2vec Can be implemented as a streaming method; the two basic tables are of size \(O(Vd)\). Skip-gram gives good results on rare words in some tasks. Performance depends on sampling, training settings and coverage. Keeping only the final vectors is not enough to continue training. The NumPy example in this chapter holds every pair in memory. You want to learn word vectors from a domain corpus — measuring against fastText, with the language’s morphology and the OOV rate in view.
GloVe Accumulates co-occurrences and learns with a weighted loss; provides accessible pretrained vectors. Training requires the co-occurrence data to be prepared. The coverage of a released vocabulary varies with language and domain. You want a quick start with released static vectors that cover the language and domain you need.
fastText By sharing character pieces it can produce representations for out-of-vocabulary and rarely seen forms; robust to some spelling variation. The subword table needs extra memory. Building an OOV vector requires combining pieces; in-vocabulary vectors can be precomputed. Similar spelling can create unwanted closeness (kitap/kitle 0.667). The OOV rate, inflectional variety or spelling variability is high — evaluating the coverage gain on task performance and hard negatives (see the measurement in Chapter 4).

The document-level choice is separate, and narrower than the literature suggests:

Turning a document into one vector
Strengths Weaknesses
Averaged word vectors With word vectors already available, no extra training of a document representation is needed; computed directly for new documents. Loses word order and syntactic relations; does not reliably keep the scope of negation. OOV behaviour and weighting matter.
doc2vec Gives higher accuracy than averaged GloVe in this experiment (0.839 against 0.820); the uncertainty of the difference has to be measured with a paired evaluation. The vector of a new document is estimated by optimization; inference cost and training settings have to be assessed.

On the single split of this experiment doc2vec scored 1.9 points above averaged GloVe, and none of the dense methods examined beat the TF-IDF row. That finding does not cover every document classification or retrieval task. Whether dense representations help with vocabulary mismatch has to be measured on the target task. The comparison in Chapter 4, with a different language, domain and amount of data, will provide further evidence for those conditions.

3.13 Applications in production systems

Word2vec’s real legacy is not the “word”: it is the idea of learning an embedding for a discrete entity from co-occurrence. Every example below is a variation on that one theme. Static embeddings are not a historical curiosity; the technique — learn a dense vector for a discrete item from co-occurrence, then do arithmetic on it — became infrastructure far beyond text.

  • Airbnb search ranking. Listing embeddings trained with skip-gram on click sequences, treating a user’s session as a sentence and listings as words, with booked listings as a global context. Deployed to production search and reported as a substantial booking gain (Grbovic and Cheng 2018).
  • YouTube recommendations. The embeddings of watched videos and of search tokens are averaged separately; those features are passed, with other user information, to a deep candidate-generation network. The network’s output is the user representation used to score candidate videos (Covington et al. 2016).
  • Item2vec and its descendants. Skip-gram-like objectives can also be applied to item co-occurrences such as purchase baskets (Barkan and Koenigstein 2016). It should not be assumed, however, that every method named *2vec uses the same training objective.
  • fastText for classification. The supervised fastText classifier combines the vectors of word features and selected word n-gram features and predicts the class with a linear output layer (Joulin et al. 2017). These word n-grams are not the same unit as the character n-grams described earlier. It is a baseline worth testing for its balance of speed and accuracy; training time and performance depend on the data and the settings.
  • Retrieval with vectors. Chapter 19 and Chapter 20 also cover systems that retrieve by representing texts as vectors. Such systems can use suitable sentence or document encoders and cosine, dot-product or distance measures. RAG can also be built on lexical or hybrid retrieval; not every RAG pipeline uses the same representation and measure.

An embedding layer maps discrete identities to learnable continuous vectors. What is differentiated is not the product ID or the token itself but the vector parameters that stand for it. In a standard table every identity has its own row; the shared training objective makes those rows be learned in a related space. In structures such as fastText there is real parameter sharing as well, through the subword pieces. The resulting vectors can be used in downstream models that have a suitable input dimension and structure.

3.14 What static embeddings cannot do

Collecting the failures we have accumulated, because together they are exactly the specification for the next part of the book.

What a static embedding cannot represent
Limitation Evidence from this chapter Common way to address or mitigate it
One static vector per word form For python, neighbours from different areas of use appear together Token representations computed from the context of use (Chapter 7)
Relations lost in unordered combination The same tokens in a different order give the same mathematical vector under averaging Combination that takes order and syntax into account, or sequence models (Chapter 6)
Vocabulary coverage In this gensim lookup OOV forms raise KeyError; the OOV token rate on the Turkish sample given is 7.3% Subword representations; also an evaluation of coverage on the task
Similarity and oppositeness mixed together The increase/decrease cosine is 0.881 Evaluation with labels suited to the task and with comparison models
Unevenness in the distribution of directions Mean cosine 0.029 in the sample drawn from the first 50,000 words Test centring or suitable post-processing on the task; centring does not remove all anisotropy
Uncertainty on rare or out-of-domain words The neighbours of some domain terms do not give the expected relation Domain evaluation, adaptation and subword options
Social associations Some occupations project differently on the chosen axis (homemaker +0.363) Evaluation with several representation and application metrics; testing suitable data, training and decision interventions

The first two are the important ones, but they do not share a root cause, and keeping them apart helps.

The rule to remember. A static embedding is a type-level lookup table: word form → one vector.

A static table maps a word form to the same vector; different usages can influence that vector, but no sense is selected anew in each sentence. A contextual model computes the token’s representation from the surroundings it can access. Those surroundings depend on the model’s attention pattern: bidirectional models use both sides, causal models only the preceding tokens they are allowed to see. Chapter 7 covers those details.

The second row is a different problem: word order is about how you compose individual vectors into a sentence or document vector, not about the vectors themselves. Averaging destroys order; an order-preserving composer would ease that half of the problem even with non-contextual vectors.

ELMo (Peters et al. 2018) was an important turning point in the spread of this approach in 2018, using the internal states of a bidirectional language model as word representations that change with context. The idea of contextual representation existed earlier; ELMo is an effective example of producing general-purpose, transferable representations from it. The idea of computing a representation from context continues in the models of Chapter 7.

3.15 Intuition checks

Answer before expanding.

There is no universal optimum at 300. With fewer dimensions the capacity of the representation can be limited; with more, memory and computation cost rise, and how well detail learned from little data generalizes has to be assessed separately. A low dimension does not force all unrelated words onto the same point, however, and a high dimension does not necessarily produce noise. We choose the dimension from data coverage and validation performance on the target task, taking the storage and latency budget into account together.

Measured on the same benchmarks used above, going from 50 to 100 dimensions raises all three scores:

  glove-wiki-gigaword-50   d=50   wordsim 0.503  simlex 0.263  analogy 0.464
  glove-wiki-gigaword-100  d=100  wordsim 0.533  simlex 0.296  analogy 0.633

The dimension sweeps in the GloVe paper (Pennington et al. 2014) show that, in particular experiments, the gain can slow down as the dimension grows. That can make a few hundred dimensions a good starting range; it is not evidence for a universal default. On the real task, compare the dimension options for performance, memory and latency.

The scoring cost of the full softmax is about \(O(Vd)\) per pair, and that of scoring with \(K\) negatives \(O((K+1)d)\). The sampling method has a cost of its own; the binary search over the cumulative distribution used here takes \(O(\log V)\) steps per sample. Negative sampling does not merely compute the same softmax probability faster: it changes the training objective. With \(q = P(c)\) and free dot products, the shifted-PMI target is obtained (Levy and Goldberg 2014b). Hierarchical softmax produces a normalized probability through the decisions along a tree path; for a balanced tree its scoring cost is about \(O(d \log V)\).

First measure the number of tokens and types in the support tickets, and the OOV rate on a train/test split. The inflectional variety of Turkish makes fastText a strong candidate; but the 7.3% OOV rate on Wikipedia is not a lower bound for support tickets. The vocabulary of a narrow, repetitive support domain can be smaller, too. Compare word2vec and fastText under the same preprocessing and evaluation conditions, and weigh the coverage gain of the subword method together with the unwanted spelling-based closeness (kitap/kitle).

From the value 0.95 alone we cannot conclude that they are about the same thing. Averaging loses order; opposite relations can be built from the same words. Nor does the shared component have to grow under averaging: the independent, varying contributions shrink, and the relative effect of the common direction can rise.

We measured this on the 20 Newsgroups training documents of this chapter: over all pairs of the 2,248 documents that have a vector, the cosine between averaged GloVe vectors has mean 0.944 and median 0.961, and 65.6% of the pairs are above 0.95. Even among pairs from different newsgroups that share is 63.8%. In this corpus, then, 0.95 is below what a typical random pair scores.

Examine the distribution over random pairs in your own corpus, but evaluate the decision threshold with document pairs labelled as similar, unrelated and, especially, hard negatives. A percentile is not by itself a probability of semantic equivalence.

Look at two axes: the size of your domain corpus, and how well the pretrained vocabulary covers your terms.

Fine-tune or train from scratch?
Domain corpus Pretrained coverage Recommendation
Small High Use pretrained vectors; freeze them downstream if needed
Small Low Subword model, domain vocabulary, or a contextual encoder
Medium–large High Test continuing training from pretrained
Medium–large Low Consider domain-adaptive training or training from scratch
Very large Domain-specific vocabulary Training from scratch can make sense

Domain shift and coverage matter alongside the amount of data. Examining the neighbours of domain terms is a first check; the choice has to be confirmed on the target task. When you say “continue from pretrained”, separate two operations: giving released vectors to a downstream supervised model as its initialization, or continuing the original word-embedding training. The second may require the output parameters and the training state as well, and not only the KeyedVectors.

It depends on the task. On the single split here TF-IDF had the highest accuracy (0.853); its gap to averaged GloVe is hard to explain by chance in the paired test, and its gap to doc2vec we could not measure. Lexical retrieval (BM25, for example) can be useful for exact term matching, and dense retrieval for relations between different wordings. Hybrid retrieval combines those two kinds of information, but its performance has to be evaluated. Embedding vectors do not remove every use of TF-IDF or of one-hot encoding; selecting an embedding row can in fact be written mathematically as a matrix product with a one-hot vector.

3.16 Interview questions

A note on sourcing. The questions below were written for working through the concepts and the practical decisions of this chapter; no claim is made that they were asked at particular companies. As general preparation resources, see Chip Huyen’s Introduction to Machine Learning Interviews (Huyen 2021) and the AIMLInterviews repository. For the production examples, the relevant system papers are cited separately.

The questions fall into two groups. The first three sets rest directly on what this chapter covered; if you worked through it, you can answer them. The last set steps outside the chapter into production system design and touches concepts not defined here (approximate nearest-neighbour indexes, cold start, probe sets) — read those answers as a map of where to look next rather than as knowledge this chapter requires.

From this chapter: testing whether you understand the objective

Q: Explain word2vec to someone who knows regression but not neural networks.

Take skip-gram with negative sampling. Observed pairs are labelled positive and pairs from the noise sampler negative. The score is the dot product of two learned vectors; the probability is \(\sigma(v \cdot u)\), and the derivative of a single term with respect to the score is \(\sigma(v \cdot u) - y\). The form of the loss is shared with logistic regression, but both vector tables are learned, so the joint problem is not convex. SGNS is also related to the target \(\mathrm{PMI} - \log K\) when the negatives are drawn from the context marginal and every dot product is left free (Levy and Goldberg 2014b). The actual low-rank model and PPMI–SVD use different losses and different transformations of the target; they need not give the same result.

Q: What do skip-gram and CBOW each predict, and when would you prefer each?

Skip-gram predicts the context words from the centre word, and CBOW predicts the centre word from the combined context vectors. CBOW is usually fast because it works with fewer training predictions; skip-gram can have an advantage on rare words under some conditions. The differences in speed and quality are set by the window, the number of negatives, the corpus and the implementation. The choice should be measured on the target task, by the frequency groups that matter in particular.

Q: Why is the negative sampling distribution raised to the power 0.75?

The 0.75 exponent flattens the unigram distribution and so raises the relative sampling share of rare contexts. The negative label belongs to the pair having come from the sampler; it does not mean the pair never co-occurs in the corpus. The exponent is an empirical choice and it also changes the implicit matrix target (Levy et al. 2015), so it should not be treated in evaluation as a fixed, unimportant detail.

Q: How can subsampling frequent words affect speed and the quality of the representation?

It reduces the share of very frequent words in the training pairs and lowers the computational load. Because some words move closer together in the remaining sequence, the distribution of context pairs changes too. That can improve quality on some tasks; it is not a general guarantee. In the run reported in this chapter the number of in-vocabulary tokens fell by 26.6%.

From this chapter: testing whether you understand the geometry

Q: king − man + woman ≈ queen. What is actually going on, and what is wrong with the demo?

Analogy arithmetic relies on some relations being representable by roughly consistent vector differences; for the nearest neighbour to be the expected word, an exact parallelogram is not required. The library removes the input words from the candidates. To see the effect of the filter, the normalization and the scores have to be held fixed: with the same scores and no filter, the nearest vector is king itself at 0.836, and queen is second at 0.770. The candidate set is part of the definition of the task. Compare category results and baselines, not a single successful or failed analogy (Nissim et al. 2020; Linzen 2016).

Q: Two unrelated words have cosine 0.3. Is that high or low?

In the sample drawn from the first 50,000 GloVe words the mean cosine is 0.029 and the standard deviation 0.119; 0.3 is about 2.3 standard deviations above that mean. That says the score is high relative to the sample; it does not imply a normal distribution, a particular p-value or a guaranteed semantic relation. Examine the empirical percentiles of your own data and the performance on labelled pairs. Centring removes the shared mean (Mu et al. 2018); it does not remove all anisotropy.

Q: Your semantic search returns antonyms as top matches. Why, and what do you do?

Distributional representations can give high similarity to antonyms that share the same contexts (increase/decrease 0.881). First examine the wrong matches and which relations the task needs to exclude. Lexical rules, features that handle the scope of negation, sentence encoders adapted to the task (Reimers and Gurevych 2019) and cross-encoders that assess the pair jointly are the options. Their cost and performance have to be measured; training on natural language inference is not by itself a guarantee of adequacy. More raw text does not necessarily solve the problem, but neither can we generalize that the distribution holds no discriminating information at all.

From this chapter: testing depth

Q: What is the connection between PPMI+SVD, word2vec and GloVe?

PPMI–SVD weights the co-occurrences with PMI, sets negative values to zero and approximates the resulting matrix at low rank under a squared error. SGNS uses a logistic loss on sampled pairs; its free dot-product target is, in general, \(\log\bigl[P(w,c) / (P(w)\,q(c))\bigr] - \log K\). GloVe models the positive log-counts with two vector tables and bias terms under a count-weighted squared loss. Despite that shared distributional basis the targets are not the same. The comparison by Levy et al. (2015) shows that, on the tasks they tried, hyperparameters strongly affect the ranking of the methods; it is not a claim of equal performance on every task.

Advanced system design: testing whether you have shipped something

Q: You need document vectors for 50 million documents with sub-100 ms retrieval. Walk me through the decision.

First establish whether the 100 ms limit covers query encoding, search, filtering and re-ranking; define the latency percentile, the traffic volume and the memory budget as well. Document vectors can be computed in advance. Compare the candidate representations on labelled retrieval data; do not use a universal ratio for encoder cost or vector dimension. Measure approximate nearest-neighbour indexes such as HNSW or IVF-PQ for recall, memory and latency. If exact identifiers matter (the P-101 problem from Chapter 2), include lexical or hybrid retrieval in the same evaluation.

Q: You trained embeddings on your own corpus and they are worse than pretrained. What happened?

First check the amount and distribution of the data, the coverage rate, the tokenization, the training settings (min_count, number of epochs, subsampling) and the evaluation protocol. Few observations can weaken rare words, but there is no universal token threshold. A low score on a general similarity set does not entail being worse on your domain task; nor can it be said that a domain model must fall behind there. Compare using released vectors as the initialization, or continuing training where the necessary training state is available, with training from scratch on the same task.

Q: How would you detect that your production embedding model has gone stale?

On live traffic, track the rate of out-of-vocabulary forms, the distributions of tokens and of domain terms, and task performance on examples that can be labelled. The OOV rate does not rise before every drop in quality, and the fact that a subword model can produce a vector does not guarantee coverage of meaning. For a frozen model and a fixed vocabulary, the neighbours of fixed terms do not change. A fixed probe set is useful for comparing retrained model versions; measuring current data drift requires fresh traffic samples and task labels that are updated over time.

Q: You have 100,000 items and click logs. Design an item embedding.

You can treat sessions as sequences of items and learn item vectors with a skip-gram-like method. The Airbnb example (Grbovic and Cheng 2018) uses the booked listing as an additional positive context and adds within-market negatives to the general negatives in order to learn local distinctions. Session boundaries, the definition of a negative example and the time split used for evaluation have to be set up explicitly. For new items (cold start), a representation can be produced from the title, the category and other content features; it should not be assumed that splitting arbitrary item IDs into character pieces carries meaning.

3.17 Where to go deeper

Each entry carries a level, so you start in the right place: Beginner assumes only this chapter, Intermediate assumes comfort with the maths in it, and Advanced goes into derivations and empirical detail.

Papers, in reading order. All free.

Watch.

  • CS224N Lecture 1 — Intro & Word Vectors, Stanford. (Intermediate) word2vec derived at the board by the group that wrote much of the follow-up literature.
  • Visualizing High-Dimensional Space, Google. (Beginner) Four minutes that repair the intuition for what “close in embedding space” actually means — useful before this chapter’s section on the geometry, which is about how that intuition misleads.

Textbooks and courses.

  • Speech and Language Processing, Jurafsky & Martin (Jurafsky and Martin 2026), Chapter 5, Embeddings. (Intermediate) In the August 19, 2026 draft, chapter 5 covers word representations; for PPMI see also Appendix J in the table of contents. The book is free.
  • Stanford CS224N. (Intermediate) The notes of the word-vectors lecture are a useful reference for skip-gram and the derivatives used in learning. The assignments change from year to year: on the Winter 2026 page the first assignment is an introduction to word vectors and the second covers neural network foundations, tensor derivatives and dependency parsing. If you are looking for a word2vec implementation assignment, check the year in the course archive.
  • NLP Course For You, Elena Voita (Voita 2020). (Beginner) Explains skip-gram and negative sampling with visuals; the animations make it easier to follow the roles of the centre and context tables.

Hands on.

3.18 Exercises

The label on each exercise marks its level: Basic consolidates a concept, Applied makes you measure something on your own data, Advanced goes past this chapter, and Research is open-ended.

  1. Examine the small example with a different window. (Basic) Compare the ten-sentence PPMI–SVD example for windows 1 and 4, keeping the other settings fixed. Which neighbourhoods change? Interpret the changes in terms of topical and functional relations; if the expected tendency does not show on the small corpus, report that too.
  2. Examine the limits of analogies. (Applied) Choose three examples that give the expected answer and three that do not. Keep the input normalization and the candidate filter fixed, and compare with the nearest-neighbour baseline. Propose explanations for the failures in terms of coverage, type of relation or scoring; do not settle on a cause from a single ranking.
  3. Audit your own domain. (Applied) Choose twenty domain terms in the language of the model. Decide in advance which criterion you use to judge the neighbours: “synonymy”, “being of the same kind” or “topical relation”. Grade the lists good / plausible / weak, preferably without seeing the frequency rank. Count out-of-vocabulary terms separately; for the rest, report the Spearman correlation between frequency rank and the ordinal grades. State that a twenty-term result is exploratory.
  4. Compare vocabulary coverage in Turkish and English. (Applied) Choose two corpora that are as similar as possible in topic, period and preprocessing. Set aside a fixed test set for each language; measure types, hapaxes and the test OOV rate at increasing numbers of training tokens. From the OOV learning curves, compare the amount of data needed to reach the same level of coverage. If you extrapolate beyond the measured range, state the uncertainty; do not derive an exact token requirement from type/token and hapax ratios alone.
  5. Show the limit of averaging. (Basic) In the language of the embedding model you use, write two sentences that contain the same tokens the same number of times but whose operational meaning differs through a change of order. Explain why their mathematical means are the same; evaluate any floating-point differences with a tolerance. Then write a paragraph on which relation this representation loses and why the decision model needs extra order or context information.
  6. Compare TF-IDF with dense representations. (Advanced) Set comparable validation budgets for TF-IDF and the dense candidates. Choose the doc2vec settings, different dimensions, weighted representations such as SIF (Arora et al. 2017), or combinations, on the validation set. Use the test set only for the final evaluation. If you obtain a gain, report its size and its stability across different splits; if you do not, discuss that as a valid result as well.
  7. Implement CBOW yourself. (Advanced) Extend the NumPy example in this chapter with CBOW. Compare the two methods with the same data, window, negative-sampling scheme and an explicitly stated training budget. Report performance and time by frequency group; write down the difference you expect before the experiment. If you use an additional assignment, give the relevant year in the CS224N archive and the direct assignment link.
  8. Explore an association direction. (Research) Decide in advance on the word pairs that define a direction relevant to your domain and on the terms you will evaluate. Report the projections together with the limitations of the direction you chose. After removing that direction, compare the neighbourhood and clustering structure. Do not produce the group labels by looking at the outcome of the operation; if you use a classifier, keep training and evaluation words apart. Present your small experiment as an exploration inspired by Gonen and Goldberg (2019)’s study, not as a replication of it.

These items are not an answer key; they are checkpoints for interpreting your results:

  1. Show which neighbourhoods the window size changes; on the ten-sentence example the topical/functional distinction need not be clear.
  2. For the analogies, separate the effects of normalization, the candidate filter and the baseline; do not assign a definite cause to a single failure.
  3. Report the direction of the frequency–quality relationship, the coverage and the limits of the small sample; finding no strong relationship is a result too.
  4. Interpret the Turkish–English OOV gap with the measured learning curves; do not assume in advance that the data ratio needed is several-fold.
  5. The mean of the same multiset of tokens is mathematically the same; explain the relation that is lost.
  6. It is possible that no candidate beats TF-IDF. Report the comparison with a validation/test split and with uncertainty.
  7. The ordering of CBOW and skip-gram depends on the data and the training budget; show frequent and rare groups separately.
  8. Other relations can remain after a direction is removed. Whether or not that is observed in a small experiment, it gives no definite verdict about social bias as a whole.

3.19 What to remember

If you can say the following twelve sentences in your own words, you have the core of this chapter.

  1. One-hot word vectors represent a word’s identity but define no semantic structure shared between words.
  2. The distributional hypothesis rests on the assumption that words used in similar contexts carry similar or related information.
  3. The rows of a word–context co-occurrence matrix are already distributional word representations.
  4. Raw co-occurrence counts confound the marginal frequency of frequent words with the real association signal; PMI and PPMI compare what is observed against what independence would predict.
  5. Truncated SVD compresses high-dimensional distributional structure into a low-rank space and can bring repeated patterns to the front.
  6. Skip-gram predicts the context from the centre word. Negative sampling replaces the full softmax with a training objective that separates observed pairs from pairs that come from the noise sampler; the same pair can arise either way.
  7. The SGNS used here learns two tables of word vectors, for the centre and context roles. In word2vec’s hierarchical softmax option the output parameters correspond to the inner nodes of a tree, not to words.
  8. There is a strong mathematical link between skip-gram with negative sampling and PMI-based matrix factorization, but at the low dimensionality you actually train, they are not the same model.
  9. GloVe uses corpus-wide co-occurrence counts directly inside a weighted regression objective.
  10. fastText builds a word vector from character n-gram pieces, which makes it more robust to out-of-vocabulary words and morphological variation at the cost of letting spelling similarity bleed into semantic similarity.
  11. A static word embedding produces a single, context-independent vector for a form. It can carry traces of different usages, but it does not by itself select a different representation for the sense in each new sentence.
  12. Evaluating embedding quality by nearest neighbours, analogies or cosine alone is not enough; coverage and downstream task performance must be measured together.

In Chapter 4 we look at how the classification result that favoured TF-IDF here changes with a different language, domain and amount of data, along a learning curve.