Embeddings: Words as Coordinates

Chapter 3 · Applied Natural Language Processing

Dr. Batuhan Bardak

After this lecture you can

• turn the distributional hypothesis into a matrix, reweight it with PPMI and compress it with truncated SVD
• write skip-gram with negative sampling from scratch and explain how two vector tables are learned together under a logistic loss
• choose between word2vec, GloVe, fastText and doc2vec for a given problem, and defend the choice
• examine how the space is not spread evenly over directions (anisotropy), how frequency affects the representation, and how cosine reflects relations broader than synonymy
• explain the context-independent representation of static embeddings and discuss why the contextual representations of Chapter 7 are needed

The chapter’s learning objectives, condensed to five. The PPMI objective in full: a reweighting that compares the observed pair probability with what the independence model predicts and sets negative PMI values to zero.

The route of the lecture: we first build the space from counts; on a small example every step can be checked by hand. Then we write skip-gram with negative sampling from scratch, gradients included, and examine how that method relates to PMI. Finally: what embeddings are used for, their geometric properties, and the limits of a representation that does not change with context.

The code in the chapter is abridged to explain the method; the numbers come from real runs and the full code is in the chapter notebook.

A dense chapter. If it is split over two lectures, the natural break is after the slide “Where the −log K comes from”.

The problem, in one sentence

• Chapter 2 built document vectors that work — and word vectors that do not
• Every word was a one-hot column: pump, pumps and impeller are three mutually orthogonal directions
• The cosine between two different one-hots is exactly zero
• In a linear model that uses only word counts, pump and pumps are separate features: train on the pump failed, test on the pumps failed, and the weight learned for pump does not carry over to pumps by itself

In linear-algebra terms: multiply the one-hot vectors by a learned matrix. Each word gets 100 or 300 numbers, and we aim for words used in similar ways to end up close. That short, dense vector is an embedding.

This is where Text as Data (Chapter 2) ended. The two sentences share the and failed, so those weights can be used; the weight learned for pump does not pass to the pumps feature by itself.

Definition of an embedding: a learned numerical representation of a unit. In this chapter we mostly represent words with dense vectors of relatively few dimensions; in distributional methods, words used in similar contexts can end up close in that space. In a dense vector most components are usually non-zero; in Chapter 2’s sparse count vectors most components were zero. The map-coordinates picture helps here.

For statisticians, in careful form: one-hot is an encoding; in some models that use it, independent parameters are learned per category. Representing word–context interactions with two low-rank matrices gives those interactions a shared structure. Do not equate that with a saturated model or with partial pooling in the technical sense; the comparison is for intuition only.

Three kinds of embedding, one chapter

Kind	Example	Does a word get the same vector everywhere?
Static word embedding	word2vec, GloVe, fastText	Yes — one vector per word form
Contextual token embedding	BERT	No — each token’s vector is computed from its context in the sequence; one word can be several tokens
Sentence / document embedding	SBERT and similar	One vector for the whole sentence or document

This chapter is about static word embeddings, with a short introduction to document vectors. Contextual representations belong to Chapter 7, and the use of sentence and document representations in retrieval to Chapter 19.

“Embedding” is used for three different things; separate them in the first five minutes so students have the chapter’s mental map from the start. We come back to this table at the end, at the limits of static embeddings: the “Yes” in the first row is exactly the constraint that has to be addressed.

The reading advice can go here too. Backbone for a first read: distributional hypothesis → co-occurrence → word2vec → cosine → GloVe and fastText → limits of static embeddings. Second read (Deep dives): the mathematics of PPMI and SVD, the gradient of negative sampling, the SGNS–PMI relation, anisotropy and the bias measurements.

The distributional hypothesis

“You shall know a word by the company it keeps.” — J.R. Firth, 1957

• Harris (1954) had already put the working version: the meaning of a word can be recovered from the distribution of contexts it appears in
• Words that fill the same blanks — the ___ failed, replaced the , pressure — mean something similar
• Not a theorem: a hypothesis tested against data. Shared contexts help identify topical or functional closeness; they do not by themselves identify synonymy — hot and cold share many contexts too
• No hand-labelled examples are needed: the training signal is derived from the contexts in raw text

That makes it possible to learn representations from large unlabelled corpora.

Firth (1957) was making a point about linguistic method; Zellig Harris (1954) had already stated the working version.

Concretely: if two words show up in the same blanks, then whatever their meanings are, they mean something similar.

State the limit of the hypothesis up front: antonyms share contexts too. Later in the lecture we look at that distinction with numbers: the cosine of increase/decrease is 0.881.

Count first: co-occurrence, PPMI, SVD

Ten sentences

corpus = [
    "the king rules the castle",
    "the queen rules the castle",
    "the king eats an apple",
    "the queen eats a banana",
    "the royal castle is grand",
    "a banana is sweet fruit",
    "an apple is sweet fruit",
    "the man eats an apple",
    "the woman eats a banana",
    "the king and queen are royal",
]

Step 1 · count within ±2 tokens

	eats	rules	royal	sweet	the
king	1	1	0	0	4
queen	1	1	1	0	3
apple	2	0	0	1	0
banana	2	0	0	1	0

• king ≈ queen and apple ≈ banana: the hypothesis is already working
• The largest count in the king row is the. Raw counts are affected by the association and also by how frequent the words are overall.

A count-based approach does three things: it counts the word–context pairs inside a chosen window (co-occurrences), reweights the counts, and approximates the resulting matrix with a low-rank structure. Following those steps on a small example makes the later methods easier to understand. Skip-gram with negative sampling has also been shown, under certain assumptions, to implicitly factorize a matrix related to PMI (Levy and Goldberg 2014).

The corpus has 10 sentences and a 19-word vocabulary.

Step one: the context of a word is up to two tokens to its left and up to two to its right, within the same sentence. The centre token is not part of its own context. For every (centre, context) pair we add one to the cell; the window is symmetric, so the pair in the opposite direction is counted at its own centre position. The rows of this matrix are already 19-dimensional word vectors and, unlike one-hots, they are not orthogonal. (The castle column is all zeros for these four rows; it is left off the slide.)

Total co-occurrences: the 22, eats 16, is 11, queen 10, king 9, a 8. the is frequent around many words, so the high count in the king row cannot by itself be read as an association specific to kings.

Step 2 · PMI is observed ÷ expected

\[\mathrm{PMI}(w, c) = \log \frac{P(w, c)}{P(w)\,P(c)}\]

• Ratio 1: the observed probability of the pair equals what the independence model predicts, PMI 0 — that does not mean there is no semantic relationship
• Above 1: more often than expected, positive. Below 1: less often than expected, negative
• For rare words, a few co-occurrences may not be enough to estimate a negative association reliably
• Positive PMI (PPMI) sets negative values to zero; a useful representational choice, not a claim that every negative value is meaningless:

\[\mathrm{PPMI}(w, c) = \max\!\left(0,\ \log \frac{c_{wc}\, N}{c_{w\ast}\, c_{\ast c}}\right)\]

Pointwise mutual information (PMI): the logarithm of the ratio between the observed probability of a word–context pair and the product of its marginal probabilities. Positive for pairs seen more often than the independence model expects, negative for pairs seen less often.

Read the fraction like this: the denominator is how often we would expect the two words to land next to each other if they were independent; the numerator is how often they actually do. So PMI is the log of observed ÷ expected.

In the PPMI formula c_wc is how often the pair occurred, c_w* the row total, c_*c the column total and N the grand total of the matrix.

PPMI and IDF are not the same formula, but they share an engineering instinct: a high raw count is not information on its own. IDF measures how rare a term is across documents; PMI checks whether two words meet often only because each of them is frequent.

Work one cell by hand

All four numbers are read off \(C\): the pair count \(c_{wc}\), the row total \(c_{w\ast}\), the column total \(c_{\ast c}\) and \(N = 144\) — not the 51 tokens but all (word, neighbour) pairs.

pair	observed	expected	ratio	PMI
apple–sweet	1	\(7 \times 6 / 144 = 0.29\)	3.43	1.23
king–the	4	\(9 \times 22 / 144 = 1.38\)	2.91	1.07

• The pair observed once gets the higher PMI, because its observed-to-expected ratio is larger
• The expectations do not say how much of what we observed is “explained”; PMI compares two relative frequencies
• The transformation does not guarantee that an association resting on a single observation is reliable

Where the numbers come from. c_wc = 1: how many times apple and sweet landed in each other’s window. c_w* = 7: the sum of the apple row. apple occurs three times, with 2, 3 and 2 neighbours: 2 + 3 + 2 = 7. c_*c = 6: the same sum for the sweet column. N = 144: the sum of every cell. Each token contributes one pair per neighbour in its window — 4 in the middle of a sentence, 2 or 3 at the edges — so a five-token sentence yields 14 pairs. Nine five-token sentences and one six-token sentence (18 pairs): 9 × 14 + 18 = 144.

N is the number of pairs, not tokens: all three probabilities are estimated from the same pair table.

The independence model predicts 7 × 6 / 144 = 0.29 meetings for apple–sweet; we observed 1, which is 3.43 times that. log 3.43 = 1.23. For king–the the prediction is 1.38; we observed 4, 2.91 times.

PPMI for the same cells: king has rules 0.69 and the 1.07; queen has rules 0.59, royal 1.06, the 0.67; apple and banana have eats 0.94 and sweet 1.23. The apple–sweet count is half the apple–eats count; the PPMI values are 1.23 and 0.94. The reason is the lower marginal frequency of the sweet context. The highest PPMI in the row is 2.04, for an: in this corpus an is followed by apple every time it occurs.

Quick check · PPMI

Two words each occur 10 times in a million-token corpus and never co-occur. Why clip the negative PMI values instead of keeping them?

• Counting only the tokens immediately to the left and right, the expected directed co-occurrence count is about \(2 \times 10 \times 10 / 10^6 = 0.0002\); observing zero is unsurprising
• Widening the window to two tokens on each side roughly doubles the expectation
• The PMI of a zero count goes to \(-\infty\); the finite value obtained with smoothing depends on the method chosen
• The negative PMI estimates of rare pairs can be strongly affected by sampling uncertainty; clipping at zero leaves those values out of the representation and makes the matrix sparser

Give the room a minute, then reveal the answer (one click).

Levy, Goldberg and Dagan (2015) showed that, on the similarity and analogy benchmarks they examined, the differences between methods mostly stayed small once hyperparameters were tuned comparably. We come back to that paper several times today.

Step 3 · Truncated SVD

• The PPMI rows already behave sensibly: king → queen 0.55. But apple → man 0.79: the two rows carry high weights in the an and eats columns, and man occurs in a single sentence
• Keep only the \(d\) directions with the largest singular values — the best rank-\(d\) approximation in the Frobenius norm:

\[\mathrm{PPMI} \approx U_d \Sigma_d V_d^\top, \qquad \text{embedding} = U_d \Sigma_d\]

• Here five components carry 65.4% of the sum of squares — not of the variance, since PPMI is never centred
• Truncation can reduce some accidental detail, and it can also lose useful distinctions; what is kept is decided by the singular-value structure, not by how often something repeats

Nearest neighbours before factorization (19-D PPMI rows): king → queen 0.55, are 0.55, man 0.45, the 0.37. apple → man 0.79, fruit 0.52, banana 0.40, is 0.31. apple → man cannot be read as synonymy: cosine measures the overlap of weights in shared columns, and a context overlap that comes from a single example may not be a reliable semantic similarity.

The picture behind the notation. Think of the PPMI matrix as a stack of transparencies. Each shows a single recurring pattern, printed at a different darkness. Stack all nineteen and you reproduce the matrix exactly. The singular values are those darknesses; U says how much each word takes part in each pattern. Truncating to d keeps the boldest few. A storage budget is a helpful picture: with fewer components some detail is lost; but the method’s criterion does not line up exactly with the detail a person finds important, or with noise.

Code: d = 5; in this small example we compute the full SVD and take the first five components. For large sparse matrices a method that computes only the components needed is preferred.

Singular values: 5.75, 4.76, 4.03, 3.71, 3.64, 3.08, 3.03, 2.80. The first two dimensions hold 36.8% of the sum of squares, the first five 65.4%. Those ratios summarize the compression; they do not show that the discarded components are unimportant for a task.

Scaling. Multiplying by S gives more weight to coordinates that belong to large singular values; the scaling that minimizes reconstruction error need not be the best choice for word similarity. In the family W = U_d Σ_d^p the exponent can be tuned; Levy, Goldberg and Dagan (2015) show that p = 0.5 or p = 0 can give better results than p = 1 on similarity tasks. Here p = 1 keeps the computation visible; in practice p is chosen on validation data.

What the compression actually did

word pair	PPMI rows, 19-D	SVD, 5-D	what happened
apple – banana	0.40	1.00	never share a sentence; shared contexts link them, and compression makes the link decisive
king – queen	0.55	0.90	already right, and got stronger
apple – man	0.79	0.85	the one-sentence accident — not deleted, but no longer first

SVD does not break the wrong link; it strengthens the right one enough to overtake it. The ordering improves; the absolute values do not.

Deerwester et al. 1990 — latent semantic analysis, more than twenty years before word2vec

Look at the third row, because this step is usually described wrongly. SVD does not delete the apple–man accident: that cosine rises from 0.79 to 0.85. What changes is the ranking. Before, apple’s nearest neighbour was man with banana third; now banana is first and man third. That is what “preferring signal over noise” actually looks like.

The number to watch is the first row. apple and banana never once occur in the same sentence, so nothing links them directly. What links them is that they keep the same company — both follow eats, both precede is sweet fruit. The raw PPMI rows already register that as 0.40, because shared contexts are shared coordinates; the compression strengthens it to 1.00, enough to reorder the neighbour list. Call this shared-context generalisation rather than transitivity — cosine is not a transitive relation — and note that low-rank compression makes it decisive rather than visible.

Other neighbours after SVD to 5 dimensions: king → queen 0.90, are 0.88; eats → an 0.95, a 0.95; castle → are 0.82, is 0.80.

This is latent semantic analysis (Deerwester et al. 1990).

Ten sentences, two coordinates

• Royalty words on the left, food words on the right
• man and woman are close in this view; their contexts are not identical
• The first coordinate is negative for every word (−0.70 … −2.14): a shared direction, which is why we plot dimensions 2 and 3
• Not spread evenly over directions: anisotropy, measured on GloVe later

A static snapshot of the chapter’s interactive plot: the SVD coordinates of the small corpus, dimensions 2 and 3. That choice makes other distinctions visible; it is no substitute for the similarities in the full space.

Read the compression printout with that in mind: in two dimensions king vs queen has a cosine of 0.996 and king vs banana 0.141. In a space where every word shares the sign of the first coordinate, part of a high cosine comes from the shared direction. Between two different one-hots the cosine is always 0.00.

From counting to predicting

• Scalability was an important factor: ten billion possible cells at 100,000 words; batch PPMI–SVD counts first and factorizes afterwards
• word2vec (Mikolov et al. 2013): learns vectors from local training pairs without building the matrix explicitly
• In skip-gram, a shallow network: the centre word’s one-hot → that row of \(\mathbf{W}\) (a lookup, no activation) → \(\mathbf{W}'\) scores candidate contexts. In CBOW the context vectors are combined first
• Mostly \(\mathbf{W}\) is kept; not a requirement

We set up the prediction task to learn vectors we can reuse. A transformer’s input embedding table (Chapter 7) works similarly; its rows belong to tokens.

Counting works: royalty landed on one side and food on the other. Scalability was an important factor in how quickly prediction-based methods were taken up; it is not a distinction that holds for every count-based method.

Definition of word2vec: a family of methods from 2013 that learn word vectors from context. Skip-gram predicts the context words from the centre word, and CBOW predicts the centre word from the context words.

Three layers in skip-gram. The input layer is the centre word’s one-hot vector. The hidden layer multiplies it by W; multiplying a one-hot by a matrix selects that word’s row. So the hidden layer is a lookup table, often called a projection layer. No further activation is applied. The output layer scores the selected vector against candidate context words; for that we use a second table, W′, which holds one row per word under the full softmax and under negative sampling alike.

Two matrices are learned: v for words as centres, u for words as contexts. In classic word2vec use it is mostly v that is kept — a common choice, not a requirement. Some approaches use the output matrix, or the sum or average of the two; GloVe’s authors recommend adding them (Pennington et al. 2014).

CBOW and skip-gram: arrows reversed

One window on the pump failed during the night shift: centre failed, width 2.

CBOW · context → centre

the pump during the → average → failed

• One training example per window
• Usually faster; combines the context vectors

Skip-gram · centre → context

failed → its own vector → the pump during the

• Four training examples per window
• More training pairs; can help on rare words

Both learn \(\mathbf{W}\) (word → vector) and \(\mathbf{W}'\) (scores partners); the pretrained vectors you download are mostly rows of \(\mathbf{W}\). The rest of the chapter follows skip-gram, which is also what negative sampling is most often paired with.

A static copy of the chapter’s two-architecture figure. CBOW combines the vectors of the surrounding words and predicts the centre. Skip-gram takes the centre word and predicts each surrounding word in turn. (CBOW is the default in the original word2vec tool and in gensim.)

Skip-gram pairs for the centre failed: (failed, the), (failed, pump), (failed, during), (failed, the). In the book’s interactive version students type a sentence and move the centre and the window.

No hand annotation is needed; in the next step we give training labels to the positive pairs and to sampled negative ones. Six billion tokens with a window of five on each side produce at most 60 billion pairs; sentence boundaries, subsampling and dynamic windows reduce that number.

Negative sampling with a logistic loss

• The full softmax computes and normalizes \(V\) context scores for every training pair: about \(O(Vd)\). Negative sampling changes the objective; the scoring cost is about \(O((K+1)d)\)
• The new question: did this training example come from the observed pairs (\(y=1\)) or from the noise sampler (\(y=0\))? The same pair can be produced either way. For each observed pair, \(K\) contexts are drawn from a noise distribution \(q\)

\[P(y = 1 \mid w, c;\, K, q) = \sigma(\mathbf{v}_w \cdot \mathbf{u}_c), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}\]

\[\mathcal{L} = -\log \sigma(\mathbf{v}_w \cdot \mathbf{u}_c) - \sum_{k=1}^{K} \log \sigma(-\mathbf{v}_w \cdot \mathbf{u}_{n_k})\]

• The positive pair’s term falls as the dot product grows; a negative sample’s term falls as its dot product shrinks
• \(K = 5\), \(V = 400{,}000\): six dot products per pair in place of 400,000 — about 67,000 times fewer; the actual speed-up depends on hardware and implementation

Mikolov et al. 2013

Definition of negative sampling: for each observed word–context pair, drawing a small number of contexts from a noise distribution and using them as negatively labelled training examples. The negatives need not be unseen in the corpus or semantically unrelated.

What the probability means: it belongs to the training sampling scheme we set up; it is not the probability that two words occur together in the real world.

The pull-together, push-apart picture describes the change in score; it does not say that the Euclidean distance between the vectors necessarily moves the same way.

The relation to logistic regression, in careful form: the sigmoid and the log loss that SGNS uses for each labelled pair are familiar from logistic regression. But the vectors on both sides of the dot product are learned too, so the model as a whole is not classical logistic regression with fixed features. A more accurate description is logistic matrix factorization, in which two vector tables are learned together. That explains both the familiar form of the gradients and the non-convexity of the joint optimization.

The chapter’s figure: the left panel scores all 400,000 candidates and applies a softmax; the right panel computes six dot products, for one positive (pump) and five negatives (opera, zinc, tariff, carbon, helix).

The loss widget in the book: what is optimized directly is the word–context score, not word–word similarity. Two words seen with similar contexts receive similar updates and so move closer together.

Work one update by hand

Four dimensions in place of 300. Centre failed: \(\mathbf{v} = (0.4,\ -0.2,\ 0.7,\ 0.1)\). Positive context pump: \(\mathbf{u} = (0.5,\ 0.1,\ 0.6,\ -0.3)\). Sampled negative opera: \(\mathbf{u} = (-0.3,\ 0.6,\ -0.2,\ 0.4)\).

pair	\(y\)	\(\mathbf{v} \cdot \mathbf{u}\)	\(\sigma(\mathbf{v} \cdot \mathbf{u})\)	loss	\(\sigma - y\)
failed–pump	1	0.57	0.639	0.448	−0.361
failed–opera	0	−0.34	0.416	0.538	0.416

• The derivative of one labelled term with respect to the dot product is \(\sigma(z) - y\): prediction minus target
• Update only \(\mathbf{v}\) at learning rate 0.05: failed·pump 0.570 → 0.590; failed·opera −0.340 → −0.359
• Update the context vectors too: 0.603 and −0.374 — the same directions, roughly 1.7 times as far

For the positive pair: (0.4)(0.5) + (−0.2)(0.1) + (0.7)(0.6) + (0.1)(−0.3) = 0.57. So σ(0.57) = 0.639: under this training scheme the model gives the pair about a 64% probability of having arrived as a positive example. Its contribution to the loss is −log 0.639 = 0.448.

The negative pair’s dot product is −0.34; its term uses σ(−z), so σ(0.34) = 0.584 and the loss is −log 0.584 = 0.538.

For pump, 0.639 − 1 = −0.361: negative, so the dot product goes up. For opera, 0.416 − 0 = 0.416: positive, so the dot product goes down.

With the context vectors fixed, the gradient of v is the sum of both terms: (0.639 − 1)·u_pump + (0.416 − 0)·u_opera = (−0.305, 0.213, −0.300, 0.275). gv in the training code computes it in one einsum.

The mathematics of one update is simple; quality is decided largely by which pairs are shown, how often and against what — negative sampling, subsampling, the noise distribution and the context window. Two slides on.

One pair touches at most seven rows

\[\frac{\partial \mathcal{L}}{\partial \mathbf{v}_w} = \bigl(\sigma(z) - y\bigr)\,\mathbf{u}_c, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{u}_c} = \bigl(\sigma(z) - y\bigr)\,\mathbf{v}_w\]

• The gradient of each vector is the scalar error \(\sigma(z) - y\) times the other vector. When the correct label gets a high probability the scalar error is small; that alone does not guarantee stable training
• One pair with \(K = 5\): one row of \(\mathbf{W}\) and \(1 + K\) rows of \(\mathbf{W}'\) — at most seven distinct rows
• The remaining rows do not enter this pair’s data loss; their gradients are zero. Under SGD with no momentum and no weight decay those rows do not change
• The model we train from scratch has 37,354 rows; seven is about 0.019%. A full softmax affects 18,678 rows, 2,668 times as many — a ratio of rows, not of running time

For those who want the derivation. Write z = v·u. σ′(z) = σ(z)(1 − σ(z)). By the chain rule the derivative of −log σ(z) is σ(z) − 1. For a negative sample the term is −log σ(−z) and the same two lines give σ(z). Both collapse into σ(z) − y.

The probabilistic reading: the label of every training example is 0 or 1, modelled with a Bernoulli distribution. Its parameter p lies in [0, 1]; for finite z the sigmoid produces p = σ(z) in (0, 1). −[y log p + (1 − y) log(1 − p)] is binary cross-entropy, the negative log-likelihood of the Bernoulli model. Probability estimates can also be evaluated with a squared loss; that is a different criterion.

Stability: vector norms can grow and the contributions within a batch can add up. The learning rate, the gradient scale and, where needed, clipping have to be considered separately.

The count: two tables of 18,677 words and 100 dimensions hold 37,354 rows and 3,735,400 parameters. If the sampled negatives repeat, or collide with the positive context, fewer than seven distinct rows are touched. That sparsity is one of the important reasons training is efficient.

Practical consequence: in SGNS the table is accessed only through the selected indices, so the gradient of the data loss is sparse at the level of rows. PyTorch’s nn.Embedding(..., sparse=True) produces sparse gradients for accesses of this kind. If the embedding parameters are also used in other operations, such as a full softmax, the gradient of the whole parameter need not be sparse.

Four training choices that decide how well word2vec works

choice	what it sets	typical value	effect
Number of negatives (\(K\))	negatives per positive pair	5–20 for a small corpus, 2–5 for a large one in the original work	changes the balance between cost and training signal
Context window	candidate neighbours and weighting by distance	for example at most 5, with a dynamic window	narrow windows can bring out functional relations more, wide ones topical relations
Subsampling (\(t\))	probability of keeping very frequent tokens	for example \(t = 10^{-3}\)	reduces the share of frequent words in training
Noise distribution	probability of drawing each negative	\(q(w) \propto f(w)^{0.75}\)	raises the relative sampling share of rare words

The update itself is simple. What decides quality: which pairs you show, how often, and against what.

Number of negatives: with a very small K the model sees too few negative examples; a large K makes every step more expensive. The original paper suggests 5–20 for small corpora and 2–5 for large ones (Mikolov et al. 2013).

The context window: with a window of 5, up to five tokens to the left and right of the centre, within the same input sequence, are candidate contexts. Narrow windows (1–2) mostly reflect close syntactic relations and wide windows (5–10) topical relations more. Those are tendencies; the corpus, the language and the preprocessing affect the outcome too. If sentence boundaries are kept the window does not cross them. word2vec does not keep the window fixed either: for each centre it draws the effective width at random between 1 and the maximum, so near neighbours produce pairs more often. Exercise 1 checks this by hand on the counting method.

Subsampling and the noise distribution, with numbers, on the next slide.

Two of the choices, in numbers: subsampling and the 0.75 exponent

Subsampling

share of corpus	example	kept	discarded
5%	the	0.16	84%
1%	of, and	0.42	58%
0.1% (\(= t\))	mid-frequency	1.00	0%
0.01% and below	the tail	1.00	0%

In our run 26.6% of the in-vocabulary tokens go: 6,397,826 → 4,698,743.

Negatives are drawn with \(f(w)^{0.75}\)

• A word 5,000 times more frequent is only about 595 times more likely to be drawn — the gap narrows about eightfold
• The exponent raises the relative share of rare contexts; it adds no new kind of signal
• An empirical choice, not a theoretical optimum

Levy et al. (2015): hyperparameters of this kind can affect comparison results substantially.

Subsampling. Frequent words produce a large number of training pairs; to reduce their relative influence each token is kept with a probability that depends on its frequency: min(1, (sqrt(f/t) + 1) · t/f). Here f is the relative frequency among the tokens left after the vocabulary filter, and t a threshold on the same scale.

Paper and code. The original C implementation uses that keep formula. The paper gives P(discard) = 1 − sqrt(t/f); that expression is negative when f < t, so as a probability it is read as max(0, 1 − sqrt(t/f)). The two formulas are not the same; when reproducing an experiment, state which implementation was used.

Almost all of what the run discards is the, of, and and similar words. (6,397,826 is what is left of the 6,794,527 tokens read once everything below min_count=20 is dropped.)

When a token is removed, the words on either side move closer together in the remaining sequence. The window size does not change, but words that were further apart in the original text can form new context pairs. That changes both the number of examples and the context distribution.

The exponent. Raising probabilities to a power below 1 enlarges the smallest ones most in relative terms: 0.5^0.75 is 19% larger than 0.5, and 0.001^0.75 is 5.6 times 0.001. 5000^0.75 ≈ 595. Negative samples make the context vectors receive updates from pairs drawn from the noise distribution as well as from observed positive pairs; the centre vectors are affected indirectly.

Writing it from scratch

NumPy, no framework; trained on the first 40 million characters of text8. The update step:

v = Wc[c]                                              # (B, D)
u = Wo[targets]                                        # (B, K+1, D)
score = sigmoid(np.einsum("bkd,bd->bk", u, v))         # (B, K+1)
err = (score - labels).astype(np.float32)              # predicted − observed
gv = np.einsum("bk,bkd->bd", err, u)
gu = err[:, :, None] * v[:, None, :]
np.clip(gv, -1.0, 1.0, out=gv)
np.clip(gu, -1.0, 1.0, out=gu)
np.add.at(Wc, c, -lr * gv)
np.add.at(Wo, targets.ravel(), (-lr * gu).reshape(-1, D))

err = score - labels is predicted minus observed; every gradient is that difference times the other vector. After three epochs: monday → friday 0.87, france → spain 0.79, physics → mechanics 0.82.

The full code is in the chapter and the notebook: a vocabulary with min_count 20 (18,677 words), subsampling, the 0.75 noise distribution, D = 100, K = 5, window 5, 3 epochs, learning rate 0.02, batches of 2,048. The basic gradient comes from the logistic loss; the gradient clipping changes the update rule.

sigmoid is written in an overflow-safe form: np.exp(-np.logaddexp(0.0, -x)). It does not artificially restrict the score to an interval.

The run: 6,794,527 tokens read; 4,698,743 after subsampling; 46,987,400 training pairs; loss 4.159 → 2.350 (the loss at the end of an epoch is the mean of its last 1,000 batches); trained in 611 s. Other neighbours: king → consort 0.75, kings 0.75. three → four 0.95. car → racing 0.74, cars 0.74.

Two simplifications stated in the code: the window is fixed at 5, with no dynamic window; and text8 is processed as a single token stream — the original sentence and document boundaries are not kept, so artificial context pairs can form across them. Batches are drawn with replacement; “3 epochs” here cannot be compared with epochs = 3 in gensim.

The chapter’s intuition check: why does np.add.at need gradient clipping here? Batching accumulates every gradient of a repeated index and applies the sum in one step; in a batch of 2,048 a common word can appear a hundred times. Without clipping this run diverged to nan within half an epoch at learning rate 0.05. The clip is applied per coordinate, to [−1, 1], before the contributions are summed: it bounds individual contributions, not the accumulated row update. Options: divide the gradient of the whole batch by the batch size, or divide each row’s accumulated gradient by the number of contributions of its index; neither reproduces sequential SGD. Clipping is an optimization choice and can affect the vectors that are learned.

The link between counting and predicting

Levy and Goldberg (2014): how skip-gram with negative sampling relates to matrix factorization.

\[\mathbf{v}_w \cdot \mathbf{u}_c = \mathrm{PMI}(w, c) - \log K\]

• The conditions: negatives come from the context marginal and every dot product can be chosen independently; then this is the optimal target for pairs with a positive count
• Not every version of word2vec is identical to PPMI–SVD: shared low-dimensional vectors may not meet all the optima, and the 0.75 exponent changes the target
• For an explicit matrix: SPPMI \(= \max(\mathrm{PMI} - \log K,\ 0)\)

Levy et al. (2015): with matched hyperparameters, on the tasks they examined, the performance gap largely closes.

Baroni et al. 2014 · Levy & Goldberg 2014 · Levy, Goldberg & Dagan 2015

For a while counting (LSA, PPMI+SVD) and predicting (word2vec) were seen as rival approaches; Baroni, Dinu and Kruszewski (2014) compared the two and reported better results for prediction-based methods. Then Levy and Goldberg showed the relation.

Unobserved pairs: a pair whose positive count is zero can still be drawn as a negative; the logistic optimum for its free score lies at the limit z → −∞. The actual model works with finite parameters, shared low-dimensional vectors and a finite number of updates, so it does not realize those limits exactly.

The −log K term: when K grows, with the other distributions held fixed, the free dot-product targets shift down by the same constant. That is not in general equivalent to adding the same vector to every word vector. Adding a constant to the target matrix raises its rank by at most one; it can be represented with an extra coordinate, but the same fit may not be preserved at a fixed d. The effect of K on a task has to be assessed experimentally.

The caption of the chapter’s figure: the PPMI–SVD route additionally clips at zero and uses a squared loss; it is not expected to give the same vectors at low rank.

The hyperparameters in the follow-up paper: subsampling, context distribution smoothing, dynamic windows. The lesson generalizes: when a new method beats an old one, some of the improvement is the method and some is the tuning, and the split is rarely reported.

Where the −log K comes from

Fix one pair and treat \(z = \mathbf{v}_w \cdot \mathbf{u}_c\) as a free number. If the negatives are drawn from the context marginal \(P(c)\), the pair’s contribution to the objective to be maximized is:

\[P(w,c)\,\log \sigma(z) \;+\; K\,P(w)P(c)\,\log \sigma(-z)\]

Set the derivative to zero, \(P(w,c)\bigl(1 - \sigma(z)\bigr) = K\,P(w)P(c)\,\sigma(z)\), and solve:

\[z = \log \frac{P(w,c)}{K\,P(w)P(c)} = \mathrm{PMI}(w,c) - \log K\]

The derivation assumes negatives drawn from \(P(c)\) and a \(z\) that can be chosen independently for every pair. In the actual model the dot products are built from shared vectors; SGNS minimizes a sampling-weighted logistic loss, PPMI–SVD the squared error of a clipped matrix. Both the losses and the targets can differ.

The derivatives used: d/dz log σ(z) = 1 − σ(z) and d/dz log σ(−z) = −σ(z). Solve for σ(z), then convert back through z = log(σ/(1 − σ)). The negative of the same expression is the loss to be minimized.

The assumption that negatives come from the raw unigram distribution does not match the code: word2vec draws them from f(c)^0.75, which gives PMI computed against a smoothed context distribution. Levy, Goldberg and Dagan (2015) call this context distribution smoothing and show it transfers to PPMI+SVD. So the 0.75 exponent changes which matrix the algorithm factorizes.

The statistical difference. Truncated SVD gives the best rank-d approximation in Frobenius norm (Eckart–Young–Mirsky): an equal coefficient on the squared error of every cell of the PPMI matrix. The zeros in PPMI cover pairs never observed and also observed pairs whose PMI is zero or negative. The number of zeros does not by itself show that most of the error or capacity goes to them. SGNS uses a logistic loss under weights set by the positive pair frequencies and the negative-sampling distribution; GloVe chooses yet another count-dependent weighting. Those differences affect behaviour; they do not prove an unconditional advantage for any method on rare words.

The choice weighs corpus size, the need for updates, memory, inspectability and the target task together.

If the lecture is split in two, this is the place.

GloVe: counting, with a loss function

\[\mathcal{L} = \sum_{(w,c)\,:\,X_{wc} > 0} f(X_{wc}) \left(\mathbf{v}_w \cdot \mathbf{u}_c + b_w + b_c - \log X_{wc}\right)^2\]

• Accumulates co-occurrence counts explicitly; learns vectors and biases with a count-weighted squared error
• Why \(\log X\)? Counts span a wide range; logs turn ratios into differences
• Why the biases? They can model overall usage tendencies; the dot products are not exactly PMI
• Why \(f\)? Down-weights rare pairs, capped at 1 for \(x \geq 100\); zero counts are outside the sum

\(f(x) = (x/x_{\max})^{3/4}\) for \(x < x_{\max} = 100\), else 1

\(X_{wc}\)	\(f(X_{wc})\)
1	0.03
10	0.18
100	1.00
100,000	1.00

Pennington et al. 2014

GloVe accumulates the co-occurrence counts explicitly and learns the vectors with a weighted loss defined on those counts; it combines a count-based representation with gradient-based learning.

Input: a word–context co-occurrence table. The matrix C at the start of the chapter added one for every pair in the window. In the original GloVe implementation distant contexts can be counted with a lower weight, so the cells of X need not be integers.

The loss: the sum v·u + b_w + b_c is asked to be close to the logarithm of the positive co-occurrence count (weighted least squares). The two vector tables are learned together, so this is not the same optimization problem as ordinary linear regression.

Correct a common wrong intuition: GloVe uses “global statistics”, but that table is built from the same local windows word2vec sees. The difference is when the learning happens: word2vec sees examples one at a time; GloVe counts and totals them first, then optimizes against the totals.

The bias terms carry an intuition similar to the way PMI accounts for marginal frequencies; but GloVe learns them jointly from the data, and they need not equal the log row and column totals.

f: the weight does not grow without bound as the count grows; the squared residual and the gradient can still be large. In its aim of controlling the influence of frequent pairs it is related to subsampling; it is not the same operation. The 0.75 exponent in the two methods does different jobs too.

What a word is now

Pretrained GloVe: 400,000 vectors, 100 dimensions, trained on Wikipedia 2014 + Gigaword 5 (6 billion tokens).

query	nearest neighbours	what it shows
king	prince 0.77, queen 0.75, son 0.70	—
pump	pumps 0.84, pumping 0.77, pumped 0.71	pumps is a neighbour, not an orthogonal axis
turkey	ankara 0.75, turkish 0.72, greece 0.71	words about the country stand out
python	monty 0.69, php 0.59, perl 0.58, flipper 0.51	neighbours from different usages together
bank	banks 0.81, banking 0.75, credit 0.70	words about finance stand out

The defining limitation of a static embedding: one vector per word form, whatever the context. That limitation is one of the important reasons for the context-dependent token representations of Chapter 7.

For the rest of the chapter we use GloVe’s pretrained vectors as an easily obtained example of a static representation.

king starts −0.323, −0.876, 0.22, 0.253, … and has norm 6.118. No labels such as “royalty” or “gender” were assigned to these coordinates in advance. Some coordinates may turn out to be related to particular properties; such readings depend on the chosen representation and are usually not preserved when the coordinate system changes.

For python, neighbours related to the comedy troupe (monty, cleese) and to programming languages (php, perl) appear together. Which usage the flipper result comes from cannot be determined from this list alone. The lists show that a single vector can carry the influence of different usages together; they are not an inventory of all the senses.

The Turkish example: yüz can mean face, the number hundred, or the imperative swim. A static table gives that form a single vector; the different usages influence what is learned, but the result is not an explicitly computed mean of those senses. Querying the vector by itself cannot select the sense in use in a sentence; a model that processes it together with the surrounding words can make use of the context.

king − man + woman ≈ queen?

Inputs removed

word	cosine
queen	0.7699
monarch	0.6843
throne	0.6756
daughter	0.6595
princess	0.6521

The same scores, unfiltered

word	cosine
king	0.8361
queen	0.7699
monarch	0.6843
throne	0.6756
daughter	0.6595

gensim normalizes the inputs and removes them from the candidates. With the scores fixed, the unfiltered nearest neighbour is king; queen is first only without the inputs. The candidate set is part of the task.

The result that made embeddings famous outside NLP. It indicates that some relations can be represented by similar vector differences. An exact parallelogram is sufficient for the arithmetic to produce the queen vector itself; it is not necessary for queen to be selected as the nearest neighbour. Nor can we conclude from this example that the differences consist of a single gender direction only.

The rule of the comparison: to see the effect of the filter alone, hold the scores fixed. The code takes the scores for the whole vocabulary with most_similar(..., topn=None) and ranks them with two candidate sets. The unfiltered king is the result of an unrestricted neighbour search; it does not show that the expected answer to the analogy is king.

If the inputs are added without normalizing them first, the scores change a little (king 0.8552, queen 0.7834); the top of the ranking stays the same.

Evaluating analogies

• The scoring is called 3CosAdd: it combines the candidate’s cosines with the three inputs using two plus signs and one minus. Excluding the inputs is a separate evaluation rule. 3CosMul (Levy and Goldberg 2014) gives better results in some evaluations
• Linzen (2016): a nearest-neighbour baseline that ignores the relation reaches a large share of the reported accuracy in some categories
• In five of the book’s eight selected examples the unfiltered nearest candidate is an input; that proportion does not measure analogy accuracy in general
• mice − mouse + goose: the expected geese is third in the unfiltered list and second, behind cows, once the inputs are removed
• madrid − spain + germany: munich comes ahead of berlin. The expected answer is not first; that does not prove the model carries no information about the capital relation

The scoring method and the candidate filter should be reported separately.

Linzen’s point: the headline analogy number measures neighbourhood structure as much as it measures a relation. On this example the baseline would be wrong — king’s nearest neighbour is prince (0.77) — but queen is already the second-nearest neighbour (0.75) before any arithmetic. Nissim, van Noord and van der Goot (2020) show that excluding the input words can also affect how bias analogies are read; the model, the normalization and the candidate set a result was obtained with should be stated together.

The eight analogies in the book’s interactive tool (real GloVe results; scores computed the same way as in the code above): king − man + woman, paris − france + italy, walking − walk + swim, bigger − big + small, madrid − spain + germany, windows − microsoft + google, mice − mouse + goose, einstein − physics + biology. Unfiltered top results: walking − walk + swim → swim 0.8385; windows − microsoft + google → windows 0.8330; einstein − physics + biology → einstein 0.7982. paris − france + italy gives rome 0.8190.

The chapter also projects thirty words to two dimensions. Read it with care: those are the first two principal components of those thirty vectors, not of all 400,000, and they keep only 37.5% of the variance. The GloVe similarities in this section are computed with the original 100-dimensional vectors; the projection is for visualization only.

Turkish makes more than twice the word forms

Two Wikipedia samples of comparable size:

	Turkish	English
tokens	1,458,213	1,828,840
distinct word forms	138,578	78,140
forms seen after 1 million tokens	112,644	52,347
forms seen exactly once (hapax)	50.0%	46.7%
test tokens outside the training vocabulary	7.33%	4.72%

ev, evim, evimde, evimdeki, evlerimizdekiler are five separate vocabulary entries. In the held-out Turkish portion about one token in fourteen falls outside the training vocabulary. The numbers belong to one sample and one tokenization choice.

Out-of-vocabulary (OOV): a form that is not in the model’s word vocabulary. It may never have been seen in training, or it may have been dropped for a reason such as a frequency threshold. For a form that is not in the table no row can be fetched directly; the gensim lookup used here raises a KeyError, and other systems may use a special unknown-unit representation.

Vocabulary growth: Turkish 45,688 at 250k, 73,435 at 500k, 97,412 at 750k, 112,644 at 1 million, 126,389 at 1.25 million; English 22,869, 34,016, 42,854, 52,347, 60,178. New forms keep arriving over the range shown.

Hapax: 69,296 of 138,578 types in Turkish; 36,503 of 78,140 in English. That half the types occur once does not mean half the vectors are learned from a single example: a frequency filter can leave those types out, and a single token produces several context pairs.

Out-of-vocabulary rate (train on 90%, test on the last 10%): 7.33% of tokens and 28.99% of types in Turkish; 4.72% and 23.29% in English. The rate also depends on how the held-out portion is chosen; it changes on another sample.

fastText: a word is built from its pieces

\[\mathbf{v}_w = \mathbf{1}_{\{w \in \mathcal{V}\}}\,\mathbf{z}_w + \sum_{g \in G(w)} \mathbf{z}_g\]

• The separate whole-word component exists only for words in the vocabulary; an out-of-vocabulary word is built from its subword components
• Shared character n-grams receive updates from examples in different words: <pu inside pump, pumps, pumped, pumping, punch … Not every piece is learned from plentiful data; the n-grams are mapped to a fixed number of hash buckets

pumps, seen in training: <pu pum ump mps ps> <pumps>

pumpage, never seen: <pu pum ump mpa pag age ge> — word2vec: KeyError

Drawn with 3-grams so it stays readable; fastText’s default is every n-gram of length 3 to 6, so a real pumpage decomposes into 22 pieces, not 7.

fastText (Bojanowski et al. 2017) does not treat a word as atomic: it represents a word as the sum of its character n-grams plus, if the word is in the vocabulary, a vector for the whole word. V is the whole-word vocabulary and G(w) the multiset of the selected character n-grams.

The bold pieces are shared. pumpage has no whole-word vector; only the n-grams carry it. That a vector can be produced for an out-of-vocabulary word does not by itself show that the vector is semantically reliable.

Sum and mean: the sum in the paper and the mean that fastText and gensim compute give the same direction for a non-zero word vector; word-to-word cosines are identical. Do not read that as saying a sum or a mean in training, or in combining words into a document vector, is equivalent under all conditions.

The angle brackets mark the start and the end of the word; they separate the boundary form of a letter sequence from its word-internal form. That does not by itself analyse stems or suffixes. The default example in the book’s tool is kitaplarımızdaki against kitaplar.

The trade-off: shared n-gram vectors tie together the representations of words spelled alike. That can provide morphological generalization, and it can also create unwanted closeness between unrelated words. The whole-word component of an in-vocabulary word can learn information apart from that effect.

What subwords buy

Out-of-vocabulary words: two Turkish forms the Turkish Wikipedia sample never contained, then two typos. word2vec raises KeyError; fastText still gives a vector:

word	fastText’s nearest neighbours
kitaplarımızdaki	kitapları 0.91, kitapların 0.91, kitaplıkları 0.90
bilgisayarcılık	bilgisayara 0.96, bilgisayarlı 0.96, bilgisayar 0.96
philosphy	philosophy 0.96, philosophie 0.95, philosophies 0.94
governmentt	government 0.98, governments 0.94, governmental 0.93

Shared character pieces help with some typos as well: in these examples the out-of-vocabulary forms get vectors close to the expected words. That result is not guaranteed for every typo.

The two Turkish rows come from word2vec and fastText trained on the same Turkish Wikipedia sample: both words are missing from both vocabularies and word2vec raises KeyError on each. The last two rows are typos; they are not in the vocabulary either.

Fourth neighbours, for completeness: kitaplarımızdaki → kitaplarda 0.90; bilgisayarcılık → bilgisayarı 0.96; philosphy → philology 0.94; governmentt → governs 0.91.

What subwords cost

query	fastText	word2vec
kitap (book)	kitapta 0.96, kitaba 0.95, kitab 0.94	eseri 0.81, makale 0.80, eseriyle 0.80
yaz (summer; also “write”)	yao 0.78, yağ 0.77, yağdı 0.77	kış 0.86, olimpiyatları 0.84, aylarında 0.83
cosine of kitap–kitle	0.667	0.446

• kitap: inflections or semantic neighbours? It depends on your task. yaz: spelling neighbours come to the front in fastText
• The ratio of 0.667 to 0.446 cannot be read as “half again as similar”: the cosine distributions of the two spaces can differ

fastText is a strong candidate where out-of-vocabulary forms and inflection matter. Character sharing can also create unwanted closeness: evaluate by coverage and task performance.

Character n-grams make it possible to share some stem and suffix patterns without an explicit morphological analyser. In return, the representations of words that are spelled alike but mean different things can move closer together too.

For kitap, word2vec returns semantic neighbours such as eseri (his/her work) and makale (article). For yaz, the word2vec neighbours reflect the seasonal usage more (kış winter, olimpiyatları, gündönümü solstice); fastText returns yao, yağ (oil), yayoi, yay (bow).

The rule: do not accept or reject the method unconditionally according to word length. Compare the score distributions of unrelated or hard-negative pairs as well. The idea shared with BPE (Chapter 8) is to benefit from rare wholes through their pieces; the mechanism differs: BPE determines the tokenization units, fastText builds a word vector by combining overlapping character n-gram vectors.

Documents as vectors

Chapter 2’s four 20 Newsgroups categories, same split, same logistic regression on top:

representation	accuracy	dimensions
TF-IDF (Ch. 2 baseline)	0.853	32,014
averaged GloVe	0.820	100
idf-weighted averaged GloVe	0.822	100
doc2vec (PV-DBOW, 40 epochs)	0.839	100
averaged GloVe + doc2vec	0.845	200

On this split TF-IDF has the highest accuracy. In the paired test its gap to averaged GloVe is hard to explain by chance: 135 documents against 84, p = 0.0007.

The standard error of one accuracy (≈ 0.010) is not that of the difference between two models. The two GloVe averages: 66 against 63 (p = 0.86). No such calculation for the doc2vec rows: their predictions cannot be reproduced exactly.

The plain mean: the average of the word vectors in a document. The function expects a list of tokens; out-of-vocabulary tokens are skipped, and a document with no vector stays a zero row (79 of the 2,327 training documents, 65 of the 1,550 test documents). In the IDF-weighted mean, IDF is computed from the training documents only.

doc2vec (Le and Mikolov 2014) assigns a learnable vector to the document id as well; negative sampling or hierarchical softmax can be used. PV-DM predicts the target word from the document vector and context words; PV-DBOW predicts words sampled from the document from the document vector alone. Lau and Baldwin (2016) found that PV-DBOW can also be strong and computationally convenient on the tasks they examined; in gensim, dm=1 (the default) is PV-DM and dm=0 is PV-DBOW. On reproducibility: the final version (v7) of Mesnil et al. reports 88.73% with paragraph vectors alone, against the 92.58% IMDB accuracy of the original paper, and says 92.6% appeared only when the data were not shuffled. That is the result of one particular replication.

A new document: the model’s word and output parameters are held fixed and the new document vector is estimated with a few optimization steps; that requires the trained model.

McNemar’s test compares the counts, in the two directions, of documents that one model gets right and the other gets wrong. The Bonferroni threshold for three comparisons is 0.017. infer_vector derives its starting vector from a hash value that changes in every Python session, so the doc2vec predictions behind the table cannot be reproduced exactly; with a fixed hash the notebook finds 0.830–0.836.

A reduction in the number of coordinates is not the same ratio of memory saved; sparse storage, running time and the search method have to be measured as well.

SIF (Arora, Liang and Ma 2017): weights a/(a + p(w)) that depend on word frequency, and removal of a common dominant component. Not the same method as the IDF-weighted mean; a separate baseline that can be compared on tasks such as sentence similarity.

What averaging destroys

• Coverage: GloVe covers 98.6% of the tokens and 84.5% of the types; the uncovered forms are 1.4% of the token mass, and averaging leaves them out

a = "the pump failed and the valve is fine"
b = "the valve failed and the pump is fine"

• Cosine of the two averaged vectors: 1.0000. The same tokens occur the same number of times, so the unordered mean is mathematically the same vector
• In floating point the order of addition produces small differences: np.allclose is True, the bit-level comparison is False (largest difference \(1.2 \times 10^{-7}\))

Averaging does not keep the word order that determines which part failed: chapter 2’s bag-of-words problem, now in 100 dimensions. Word order needs a model that reads the sequence (Chapters 6 and 7).

Ask the room for the cosine before revealing (one click).

Coverage in numbers: 23,395 of 27,681 types.

The chapter’s code averages the GloVe vectors of the split sentences with np.mean and prints the cosine: 1.0000 and “identical: True”. “Identical” there is the result of np.allclose; bit-level equality is not guaranteed in every implementation.

Word order belongs to Chapter 6 (Language Models) and Chapter 7 (Transformers and Attention).

In this space, cosine reflects relatedness more than similarity

pair	cosine
good – bad	0.770
hot – cold	0.725
increase – decrease	0.881
cat – dog	0.880
cat – car	0.311
teacher – student	0.808

• The antonym pair gets a high cosine too — the antonym problem; the gap between 0.881 and 0.880 is not a meaningful ordering
• The model sees a word’s surroundings: hot and cold appear next to the same words
• In this space cosine mostly reflects relatedness

Do not read the raw cosine of static vectors directly as a measure of sentiment or polarity.

The road map of the geometry part: distributional relatedness, the distribution of directions and the effect of frequency; then the arbitrariness of the coordinate system and how consistent different training runs are.

The practical cost of the antonym problem: a system searching for “hot” may return documents containing “cold” among its most similar results. Task-specific labels, for contradiction or synonymy for example, can help a model learn the distinction and generalize to pairs not seen directly in training; success depends on the coverage of the data and on the training objective.

Steck, Ekanadham and Kallus (2024): in certain linear matrix-factorization models the cosines can change while the training predictions are preserved. Cosine is not a scale of meaning independent of the learned representation. The distribution over random pairs gives a useful basis for comparison; assessing similarity for a task additionally requires labelled examples. Do not treat 0.8 in two models as equivalent.

Evaluation data measure different relations as well: WordSim-353 (Finkelstein et al. 2002) contains judgements that also reflect broad semantic relatedness; SimLex-999 (Hill et al. 2015), developed later, aims to separate similarity from relatedness.

The space is not isotropic

4,000 unit vectors from the top 50,000 GloVe words

statistic	value
mean cosine, random pairs	0.029
standard deviation	0.119
pairs with cosine > 0	56.9%
norm of the mean vector	0.171
mean cosine after centring	−0.000

• Isotropic = the same in every direction. Anisotropic = not; a shift toward a common direction is one form, not a requirement
• Mean cosine and mean-vector norm are one piece of evidence: \(\bar{c} = (n\lVert\bar{v}\rVert^2 - 1)/(n-1) \approx 0.029\)
• Random pairs show how unusual a score is; a fixed 0.3 does not by itself mean a semantic relation
• Centring removes the mean direction, not all anisotropy. All-but-the-top (Mu, Bhat and Viswanath 2018): measure on the task

Explain the word first. Picture every vector as an arrow leaving a common centre: in an isotropic space the arrows spread evenly in all directions. A shift toward a common direction is not required for anisotropy: the vectors can concentrate in the two opposite directions of one axis and still have a zero mean.

The null: for independent unit vectors drawn uniformly in d dimensions the angles concentrate near 90°, with mean cosine 0 and variance 1/d. The 56.9% proves little on its own: the 8 million pairs come from 4,000 vectors, so they are not independent observations.

The values in this sample are consistent with a shared mean direction; they do not by themselves show that the whole space is squeezed into a narrow cone. That the mean cosine after centring should be exactly zero is not a general identity. The results belong to the sample drawn from the first 50,000 words. Centring does not make the covariance spherical either.

Ethayarajh (2019) observes strong anisotropy in certain layers of the contextual models he examines; do not read that as a fixed ordering for every method of contextual representation.

The histogram in the book: in this sample, centring brings the mean cosine close to zero.

Frequency leaks in; your domain lives in the tail

Frequent words have longer vectors on average: mean norm 6.19 for the top 100 words, 3.62 for ranks 100,000–400,000.

term	rank	nearest neighbours
water	430	natural 0.70, dry 0.68, salt 0.68
turbine	15,747	turbines 0.80, compressor 0.78, generator 0.72
cavitation	109,044	aliasing 0.56, barotrauma 0.54, attenuation 0.54
impeller	124,835	armature 0.62, evaporator 0.61, nozzle 0.60
eutectic	159,022	azeotrope 0.61, azeotropic 0.55, tiwai 0.54

Evaluate domain terms separately. Neighbour lists are a good first check; choose the method by comparing pretrained, domain-adapted and subword representations with task metrics.

Norm and direction. The table shows that the norm can be related to frequency; it does not show that cosine is the best measure for every task. Cosine leaves the norm out and compares directions; Euclidean distance is affected by both. For unit-norm vectors ‖a − b‖² = 2 − 2cos(a, b), so the two measures give the same ranking. Whether keeping the norm or removing it helps is assessed on the task.

Mean norm by frequency rank: 0–100: 6.19; 100–1,000: 5.85; 1,000–10,000: 5.45; 10,000–100,000: 4.99; 100,000–400,000: 3.62. Spearman(log rank, norm) over the top 100k words: −0.313. That is a trend across bands.

Raw dot products and Euclidean distance can be affected by differences in norm; a model that takes unnormalized vectors can make use of that information, and how far it does so depends on its objective and parameters.

Neighbour lists: the turbine neighbours look related to equipment and energy systems. For the cavitation and eutectic lists, which relations are useful for the task has to be judged with domain knowledge; being related within the same physical process is not the same as being synonymous. No general quality ranking can be drawn from these short lists.

Arbitrary coordinates, variable runs

Same corpus, same settings, two seeds (text8, 6,794,527 tokens)

measurement	value
same word, raw coordinates: mean cosine	−0.016
after orthogonal (Procrustes) alignment	+0.951
top-10 neighbour overlap, 1,000 most frequent	62.2%
top-10 neighbour overlap, 1,000 rarest	46.6%
largest cosine change under an orthogonal map	5.5 × 10⁻⁷

• Multiply both tables by the same orthogonal \(R\): \((WR)(UR)^\top = WU^\top\) — every coordinate changes, no dot product does
• A coordinate’s meaning in one run may not survive a change of coordinates
• +0.951: the coordinates explain much of the difference, not all — the neighbour overlaps show local differences remain

Same corpus + different seed → geometrically different, functionally similar spaces. Do not argue from one neighbour list of one run.

The numbers are the chapter’s measurement: word2vec trained twice on the same text8 slice with the same settings, changing only the seed (gensim 4.4, vocabulary 18,677). workers=1 reduces the variability that comes from thread ordering; exact reproduction also needs the data order, PYTHONHASHSEED, the library versions and the environment to be fixed.

The transformation: multiply every vector by the same random orthogonal matrix; all of king’s coordinates change and the king–queen cosine stays at 0.6826. Orthogonal Procrustes alignment may include a reflection; do not call it “just a rotation”.

When comparing different periods or corpora, align on shared anchor words that are as stable as possible, and take care not to absorb the change under study into the alignment.

Stability: even for the most frequent words about four of ten neighbours differ between the two runs; for the rarest, more than half. If a neighbourhood matters, train with a few seeds and look at which neighbours come back every time.

Bias is in the data, and the geometry records it

Projection onto woman − man

word	projection
homemaker	+0.363
nurse	+0.329
receptionist	+0.281
teacher	+0.152
doctor	+0.109
scientist	−0.046
programmer	−0.083
engineer	−0.123

The sign shows the direction of the projection on the chosen axis; it does not measure all of social bias.

1. Usually not a training error — associations in the data carry over into the geometry; what you measure depends on the corpus, the model and the axis
2. It can carry over into decisions — depending on the downstream objective, inputs and decision rule; evaluate the application’s outputs too
3. One intervention does not remove every indicator — gender-related information can remain (Gonen & Goldberg 2019)

Bolukbasi et al. 2016 · Caliskan et al. 2017 · Gonen & Goldberg 2019

The chapter lists the words in a different order (doctor, nurse, engineer, teacher, programmer, homemaker, scientist, receptionist); the slide sorts them by projection. Positive means the woman side.

Bolukbasi et al. (2016) estimate a direction, or a subspace, from several defining word pairs and not from a single pair; the single-pair calculation here is a teaching simplification. Their method includes neutralize and equalize steps for selected words. Caliskan, Bryson and Narayanan (2017), with WEAT, find relations similar to various human association patterns in the word sets they examine. Reaction times in people and relations between vectors are different measurements; do not read the effect sizes as the same quantity.

What Gonen and Goldberg show: a fall in the chosen metric does not show that all the related structure has been erased.

Ask the room the chapter’s intuition check: is a model that shows no bias on this test unbiased? No. The test measures one direction defined by one word pair. Passing a single fairness metric is evidence only about the kind of bias that metric can detect.

How do you know an embedding is any good?

• Intrinsic: does the space agree with human judgements — similarity ratings, analogies? Fast; needs no real task
• Extrinsic: does it improve the thing you care about? Slow, but the evaluation that measures the contribution directly
• The same GloVe vectors: WordSim-353 Spearman 0.533; SimLex-999 0.296. The coverage of the sets and their kinds of judgement differ too; the gap cannot be explained by antonymy alone
• Intrinsic scores are useful for comparing properties and eliminating candidates; for choosing what to deploy, performance and cost on the target task decide (Schnabel et al. 2015)

Google analogies: 0.633 overall

category	accuracy
capital-common-countries	0.939
capital-world	0.889
nationality-adjective	0.879
family	0.816
city-in-state	0.308
opposite	0.201
currency	0.152

Intrinsic and extrinsic evaluation measure different aspects of a representation.

The Google analogy set: 19,544 questions in 14 categories. The per-category breakdown shows differences the overall score hides. Those differences can be affected by corpus coverage, the type of relation and the scoring method; they do not by themselves prove which cause dominates. Report the overall accuracy together with the category scores, the coverage and simple baselines. Even the low categories are far above random choice from a set of 300,000 candidates.

The other categories: gram1-adjective-to-adverb 0.244; gram3-comparative 0.791; gram4-superlative 0.543; gram5-present-participle 0.695; gram7-past-tense 0.554; gram8-plural 0.720; gram9-plural-verbs 0.584. The “gram6-”/“gram2-” prefixes are dropped from the category names on the slide.

A gensim detail: the default is restrict_vocab = 300,000. In the word-pair evaluation, pairs containing a word outside that limit are skipped; no search for a candidate answer takes place. In the analogies both the questions covered and the candidate vocabulary are limited.

The same dataset, different evaluated examples

	GloVe · 6 billion tokens	from scratch · 6.8 million tokens
WordSim-353	0.533 · 0.0% skipped	0.627 · 11.9% skipped
the 311 common pairs	0.549	0.627
Analogies, questions scored	0.633 · 19,486	0.200 · 9,282
Analogies, all 19,544 questions	0.631	0.095

• WordSim: the small model is ahead on the common pairs too — here coverage does not close the gap. Analogies: counting skipped questions as wrong (dummy4unknown=True) halves the small model’s score
• All 58 questions GloVe skips contain denar: in the vocabulary (rank 394,434), outside the default limit of 300,000

Give the coverage and the number of examples scored together with the score. The direction of the effect of skipped examples depends on which examples are skipped.

The same dataset name does not mean the same evaluated examples.

WordSim-353: the small model’s vocabulary is 18,677 words; 11.9% of the pairs are skipped. On the 311 pairs (88.1%) both models cover, GloVe scores 0.549 and the NumPy model 0.627. On the 42 pairs the small model skips, GloVe’s correlation is 0.473; those pairs are somewhat harder than average for GloVe, but the gap does not close. This result on a single relatedness set does not show the small model is better in general; the analogy numbers say the opposite.

Analogies: the default dummy4unknown=False skips questions containing an out-of-coverage word. dummy4unknown=True counts skipped questions as wrong; it does not give the model vectors for unknown words. GloVe gets 12,334 of the 19,544 questions right, the NumPy model 1,852. The candidate vocabulary, case sensitivity and the input filter should be reported the same way.

The rule: do not compare scores computed on different subsets of examples as if they had the same coverage.

Choosing between them

method	reach for it when	watch for
PPMI + SVD	you want to inspect the steps of the computation, or set up a count-based baseline on a modest vocabulary	sparse count matrix, low-rank factors and solver memory; no row for out-of-vocabulary forms
word2vec	you want to learn word vectors from a domain corpus — measuring against fastText	performance depends on sampling, settings and coverage; the final vectors alone are not enough to continue training
GloVe	you want a quick start with released vectors that cover the language and domain you need	training needs the co-occurrence data prepared; coverage of a released vocabulary varies with language and domain
fastText	the OOV rate, inflectional variety or spelling variability is high	the subword table needs extra memory; similar spelling can create unwanted closeness (kitap/kitle 0.667)

Evaluate the coverage gain on task performance and hard negatives (the measurement in Chapter 4). For documents: on the single split of this experiment none of the dense methods beat the TF-IDF row; that finding does not cover every classification or retrieval task.

Four methods, all producing a dense vector per word, all comparable by cosine. The differences matter only in specific circumstances.

Counting and predicting as engineering. Memory: large corpora need a sparse co-occurrence matrix, the low-rank vectors and the solver’s working memory; in a streaming SGNS implementation the two tables take O(Vd) space, and the sampler and buffers need memory too. The chapter’s NumPy example stores every pair, so its memory also depends on corpus size. Reproducibility: for SVD the solver, the environment and (if randomized) the seed are fixed; the signs of singular vectors and the choice of basis in equal-singular-value subspaces are not unique. For SGNS the seed, data order, thread count and environment are controlled. New data: counts can be updated incrementally; SGNS training can continue if the two tables, the vocabulary and the training state have been kept.

Documents: averaged word vectors need no extra training of a document representation; they lose word order and syntactic relations and do not reliably keep the scope of negation. doc2vec gave higher accuracy than averaged GloVe in this experiment (0.839 against 0.820); the uncertainty of the difference has to be measured with a paired evaluation. Chapter 4 (A Representation Bake-Off) runs the comparison with a different language, domain and amount of data, along a learning curve.

Applications in production systems

• Airbnb search ranking — listing embeddings from skip-gram on click sessions: session = sentence, listing = word, the booked listing as global context (Grbovic & Cheng 2018)
• YouTube recommendations — the embeddings of watched videos and of search tokens are averaged separately and passed, with other user information, to a deep candidate-generation network; the network’s output is the user representation (Covington et al. 2016)
• Item2vec — skip-gram-like objectives can be applied to item co-occurrences such as purchase baskets; not every *2vec uses the same objective (Barkan & Koenigstein 2016)
• fastText for classification — the vectors of word and word n-gram features with a linear output layer on top (Joulin et al. 2017); not the same unit as character n-grams
• Retrieval with vectors — some of the systems in Chapters 19 and 20 represent texts as vectors; RAG can also be built on lexical or hybrid retrieval

An embedding layer maps discrete identities to learnable continuous vectors. What is differentiated is not the identity itself but the vector parameters that stand for it.

word2vec’s legacy is not the “word”: it is the idea of learning an embedding for any discrete entity from co-occurrence.

Airbnb: deployed to production search and reported as a substantial booking gain. fastText for classification: a baseline worth testing for its balance of speed and accuracy; training time and performance depend on the data and the settings.

In a standard table every identity has its own row; the shared training objective makes those rows be learned in a related space. In structures such as fastText there is real parameter sharing as well, through the subword pieces. The resulting vectors can be used in downstream models with a suitable input dimension and structure.

What static embeddings cannot do

limitation	evidence from this chapter	common way to address or mitigate it
One static vector per form	python: neighbours from different usages	representations computed from context (Ch. 7)
Unordered combination	same tokens, different order → same mean	sequence models (Ch. 6)
Vocabulary coverage	KeyError; OOV 7.3% on the Turkish sample	subword representations
Similarity and oppositeness mixed	increase/decrease 0.881	evaluation with task-suited labels
Uneven directions	mean cosine 0.029 (top-50,000 sample)	testing centring on the task
Rare / out-of-domain words	neighbours of some domain terms	domain evaluation, adaptation
Social associations	homemaker +0.363	several metrics; testing interventions

A static embedding is a type-level lookup table: word form → one vector. A contextual model computes the token’s representation from the surroundings it can access.

A static table maps a word form to the same vector; different usages can influence that vector, but no sense is selected anew in each sentence. In a contextual model the surroundings depend on the attention pattern: bidirectional models use both sides, causal models only the preceding tokens they are allowed to see.

The second row is a different problem: word order is about how individual vectors are combined into a sentence or document vector, not about the vectors themselves.

ELMo (Peters et al. 2018) was an important turning point in the spread of this approach, using the internal states of a bidirectional language model as word representations that change with context. The idea of contextual representation existed earlier; ELMo is an effective example of producing general-purpose, transferable representations from it.

Intuition check 1 of 6

Your embedding has 300 dimensions. Why 300 and not 30 or 3,000?

• There is no universal optimum at 300
• With fewer dimensions the capacity of the representation can be limited; with more, memory and computation cost rise, and how well detail learned from little data generalizes is assessed separately
• A low dimension does not force unrelated words onto the same point; a high dimension does not necessarily produce noise
• GloVe 50 → 100 dimensions: WordSim 0.503 → 0.533, SimLex 0.263 → 0.296, analogy 0.464 → 0.633
• On the real task, compare the dimension options for performance, memory and latency

Give students a minute before revealing (one click).

The dimension sweeps in the GloVe paper (Pennington et al. 2014) show that, in particular experiments, the gain can slow down as the dimension grows. That can make a few hundred dimensions a good starting range; it is not evidence for a universal default.

Intuition check 2 of 6

Why negative sampling instead of a softmax over the vocabulary?

• Scoring cost: about \(O(Vd)\) per pair for the full softmax, \(O((K+1)d)\) with \(K\) negatives. The sampler has a cost too: the binary search used here takes \(O(\log V)\) per sample
• Not merely a faster computation: the training objective changes
• With \(q = P(c)\) and free dot products the target is shifted PMI (Levy and Goldberg 2014)
• Hierarchical softmax: a normalized probability through decisions along a tree path; about \(O(d \log V)\) for a balanced tree

Hierarchical softmax is not standard in modern large language models; you can meet it as an option in classic word2vec training.

Intuition check 3 of 6

You have 50,000 support tickets in Turkish. word2vec or fastText?

• Measure first: the number of tokens and types in the tickets and the OOV rate on a train/test split
• The inflectional variety of Turkish makes fastText a strong candidate; but the 7.3% OOV rate on Wikipedia is not a lower bound for support tickets — the vocabulary of a narrow, repetitive domain can be smaller, too
• Compare the two under the same preprocessing and evaluation conditions
• Weigh the coverage gain together with unwanted spelling-based closeness (kitap/kitle)

The good answer to this question is a measurement plan, not the name of a method.

Intuition check 4 of 6

Cosine similarity between two documents’ averaged vectors is 0.95. Are they about the same thing?

• That does not follow from 0.95 alone. Averaging loses order; opposite relations can be built from the same words
• We measured it: over the chapter’s 2,248 Newsgroups documents, random pairs have mean cosine 0.944 and median 0.961; 65.6% of the pairs are above 0.95
• Even among pairs from different newsgroups, 63.8%
• Evaluate the decision threshold with pairs labelled as similar, unrelated and hard negatives; a percentile is not by itself a probability of semantic equivalence

The shared component does not have to grow under averaging: the independent, varying contributions shrink and the relative effect of the common direction can rise. In this corpus 0.95 is below what a typical random pair scores.

Intuition check 5 of 6

Should you fine-tune pretrained embeddings on your own corpus, or train from scratch?

• Look at two axes: the size of your domain corpus, and how well the pretrained vocabulary covers your terms
• Small corpus, high coverage: use the pretrained vectors. Small corpus, low coverage: a subword model or a contextual encoder
• Medium–large corpus: test continuing from pretrained; with low coverage, consider domain-adaptive training or training from scratch
• Separate two operations: giving released vectors to a downstream model as its initialization, and continuing the original embedding training. The second may require the output parameters and the training state, not only the KeyedVectors

Domain shift and coverage matter alongside the amount of data. Examining the neighbours of domain terms is a first check; the choice has to be confirmed on the target task.

Intuition check 6 of 6

“Embeddings replaced TF-IDF.” True or false?

• It depends on the task. On the single split here TF-IDF had the highest accuracy (0.853); its gap to averaged GloVe is hard to explain by chance in the paired test
• Lexical retrieval (BM25, for example) can be useful for exact term matching, dense retrieval for relations between different wordings
• Hybrid retrieval combines the two kinds of information; its performance has to be evaluated
• Embedding vectors do not remove every use of TF-IDF or of one-hot encoding: selecting an embedding row can be written as a matrix product with a one-hot vector

The gap to the doc2vec rows could not be measured: the predictions cannot be reproduced exactly.

Interview-style questions for discussion

Three of the chapter’s twelve questions: one on the objective, one on the geometry, one on system design.

1. Explain word2vec to someone who knows regression but not neural networks.
2. Two unrelated words have cosine 0.3. Is that high or low?
3. You have 100,000 items and click logs. Design an item embedding.

General preparation resources: Huyen 2021 and the AIMLInterviews repository

On sourcing: these questions were written for working through the concepts and the practical decisions of the chapter; no claim is made that they were asked at particular companies. For general preparation, see Chip Huyen’s Introduction to Machine Learning Interviews and the AIMLInterviews repository on GitHub.

The chapter’s other questions, as homework: skip-gram versus CBOW and when to prefer each; why the 0.75 exponent; how subsampling can affect speed and the quality of the representation; what to watch for in the king − man + woman demo; why semantic search returns antonyms and what to do; document vectors for 50 million documents under 100 ms; embeddings trained on your own corpus coming out worse than pretrained; detecting that a production embedding has gone stale; the connection between PPMI+SVD, word2vec and GloVe.

Suggested format: pairs, one question per pair, two minutes; then the next slide.

What a strong answer names

1. Skip-gram with negative sampling — observed pairs positive, pairs from the noise sampler negative. The score is the dot product of two vectors, the probability \(\sigma(v \cdot u)\), the derivative of one term \(\sigma(v \cdot u) - y\). The form of the loss is shared with logistic regression; both tables are learned, so the problem is not convex. Then the \(\mathrm{PMI} - \log K\) connection, with its conditions
2. A reference distribution — in the sample from the first 50,000 GloVe words the mean is 0.029 and the standard deviation 0.119; 0.3 is about 2.3 standard deviations up. High relative to the sample; no normal distribution, p-value or guaranteed semantic relation is implied. Look at the percentiles of your own data and at labelled pairs
3. Skip-gram on sessions — session = sequence of items. Airbnb: the booked listing as an additional positive context; within-market negatives added to the general ones (Grbovic & Cheng 2018). Session boundaries, the definition of a negative and the time split have to be explicit. Cold start: content features such as title and category

The longer answers from the chapter.

1. The actual low-rank model and PPMI–SVD use different losses and different transformations of the target; they need not give the same result.
2. Centring removes the shared mean (Mu, Bhat and Viswanath 2018); it does not remove all anisotropy.
3. It should not be assumed that splitting arbitrary item IDs into character pieces carries meaning.

Exercises

1. Compare the small example for windows 1 and 4. Which neighbourhoods change? Report it if the expected tendency does not show
2. Limits of analogies: three that give the expected answer, three that do not; normalization and filter fixed; compare with the nearest-neighbour baseline
3. Audit your own domain: twenty terms; criterion fixed in advance; Spearman between grades and frequency rank
4. Vocabulary coverage: OOV learning curves in Turkish and English with fixed test sets

5. The limit of averaging: two sentences with the same tokens in a different order; which relation is lost?
6. TF-IDF and dense representations: choose settings on validation data, test only at the end; discuss it if there is no gain
7. Implement CBOW yourself: extend the NumPy example; same data and budget, compared by frequency group
8. Explore an association direction: fix pairs and terms in advance; compare the structure after removing the direction

The chapter’s eight exercises, abbreviated. Details worth reading out:

3. Grade the lists preferably without seeing the frequency rank; count out-of-vocabulary terms separately. A twenty-term result is exploratory.
4. Choose two corpora as similar as possible in topic, period and preprocessing. Do not derive an exact token requirement from type/token and hapax ratios alone.
5. Options: doc2vec settings, different dimensions, weighted representations such as SIF (Arora et al. 2017), or combinations.
6. Write down the difference you expect before the experiment. CS224N assignments change from year to year; if you use one, give the year and the direct link.
7. Do not produce the group labels by looking at the outcome of the operation. Present your small experiment as an exploration inspired by Gonen and Goldberg’s study, not as a replication of it.

The expected observations are checkpoints, not an answer key: it is possible, for example, that no candidate beats TF-IDF; the ordering of CBOW and skip-gram depends on the data and the training budget.

Where to go deeper

resource	why
Mikolov et al. 2013a, 2013b	skip-gram and CBOW; then negative sampling and subsampling — read the second alongside the chapter’s code
Rong 2014 · Goldberg & Levy 2014	the gradients derived step by step; the reasoning behind the negative-sampling objective
Levy & Goldberg 2014 · Levy et al. 2015	the SGNS–PMI relation and the effect of hyperparameters
Bolukbasi et al. 2016, then Gonen & Goldberg 2019	read together: the proposed operations, then an examination of what remains
Jurafsky & Martin, Chapter 5	word representations in the August 19, 2026 draft; Appendix J for PPMI; free
Stanford CS224N	notes of the word-vectors lecture; the assignments change from year to year
Elena Voita, NLP Course For You	skip-gram and negative sampling, with visuals
fastText pretrained vectors	released vectors for 157 languages including Turkish

From the chapter’s list.

Papers, in reading order, all free: Mikolov et al., “Efficient Estimation of Word Representations in Vector Space” and then “Distributed Representations of Words and Phrases”; Xin Rong, “word2vec Parameter Learning Explained” — derives the gradients of CBOW, skip-gram and the related output objectives step by step — together with Goldberg and Levy’s “word2vec Explained”; Levy and Goldberg, “Neural Word Embedding as Implicit Matrix Factorization”, and Levy, Goldberg and Dagan, “Improving Distributional Similarity with Lessons Learned”; GloVe (Pennington et al. 2014) and “Enriching Word Vectors with Subword Information” (Bojanowski et al. 2017); Bolukbasi et al. and then “Lipstick on a Pig”.

CS224N: on the Winter 2026 page the first assignment is an introduction to word vectors and the second covers neural network foundations, tensor derivatives and dependency parsing.

To watch: CS224N Lecture 1, word2vec derived at the board; Google’s “Visualizing High-Dimensional Space” video.

Hands on: gensim and its word2vec tutorial; Jay Alammar’s “The Illustrated Word2vec” (visual explanations) and Sebastian Ruder’s “On Word Embeddings” (historical and methodological development); TensorFlow Projector — look up your own domain terms.