Text as Data: Tokens, Normalization, Zipf

Chapter 2 · Applied Natural Language Processing

Dr. Batuhan Bardak

After this lecture you can

• turn raw text into model-ready units and defend each preprocessing choice
• describe the distribution of word frequencies with Zipf’s law, and the growth of the number of distinct words with Heaps’ law
• derive a TF-IDF vector by hand and get the same numbers as scikit-learn
• explain with examples what a bag-of-words representation keeps and what it loses, and justify which tasks it suits
• explain how Turkish lowercasing and agglutinative structure affect normalization decisions and vocabulary coverage

The chapter’s five learning objectives, condensed. In full, the second also asks what these patterns mean for building a vocabulary and representing rare words, and the fourth asks you to justify the choice in an interview.

What we will look at: first how text is cut into tokens, and what normalization decisions merge and what they delete. Then the distribution of word frequencies, with Zipf’s law, and the growth of the number of distinct words as a corpus grows, with Heaps’ law. Then four short sentences become document vectors in six steps, with every intermediate result shown, and we compute the similarity between them. Finally: what information this representation loses, and why the next chapter moves to other representations.

Every number and every code output was produced by running the code shown, with scikit-learn 1.9. The LogisticRegression experiments use the default lbfgs solver, which draws no random numbers, so the random_state=0 in the code does not affect the result. Different library versions or computing environments can still produce small numerical differences.

The chapter’s collapsible “Deep dive” boxes can be skipped on a first reading.

The problem, in one sentence

The models in this chapter work on numerical inputs. To give them text, we have to turn strings of characters into a numerical representation.

• Two basic decisions: which units to cut the text into (tokenization) and the value of each unit in the document vector (weighting)
• This chapter builds sparse representations: most components of a document vector are zero. Chapter 3 moves to dense vectors
• Simple models on sparse representations can be a strong baseline where labelled examples are few, where long documents are classified by topic, or where literal word matching matters

Preprocessing decides what information reaches the model, so it is part of the modelling process.

We study this approach through bag-of-words and TF-IDF. These methods are also a useful starting point for evaluating more complex text models.

Sparse and dense. Only a small part of a vocabulary built from a large corpus usually occurs in any one document, so most components of a document vector are zero. Dense vectors describe each word with far fewer coordinates, most of them non-zero; that is the next chapter.

Why a baseline: such a model helps us weigh what a more complex method gains against what it costs.

In building these representations we decide which units to cut the text into, which differences to keep and which units to count. Large language models involve similar design decisions; but when you use a pretrained model you inherit a large part of them from the tokenizer that ships with it. This chapter examines the options and the libraries’ assumptions, to show how each decision affects the representation.

Treat the choices in this chapter as modelling decisions and record your reasons for them. If you present your work in a paper, name the preprocessing steps that affect the results in the methods section.

Split on whitespace — and see what it leaves

text = "The pump P-101 didn't fail; it's the seal (again!)."
print(text.split())

['The', 'pump', 'P-101', "didn't", 'fail;', "it's", 'the', 'seal', '(again!).']

• The and the both occur, and count as two different tokens; fail; and it's stay units of their own
• Three decisions: map The and the to the same form? Keep the semicolon in fail; or remove it? Keep it's whole, or map it to it is?
• A word-level tokenizer can use separate rules for punctuation, apostrophes, URLs, numbers and abbreviations

Normalization decisions are separate operations from deciding where tokens begin and end. Whether a normalization erases a useful distinction has to be judged for the task.

Tokenization: cutting text into the tokens a model will process as separate units. The tool that does it is a tokenizer. A token is an item the model processes as a separate unit; depending on the tokenizer it is not always a whole word, and not always a piece of text that can be read on its own.

The choice of units affects the size of the vocabulary, how new words are represented, and how many tokens a text is cut into. In services billed per token, the number of tokens also affects the cost.

A simple place to start is to split at whitespace characters such as spaces, tabs and line ends; str.split() with no arguments does this. It does nothing separate about punctuation, upper and lower case, or suffixes.

If another text processed the same way contained fail or it is, those would be counted as different tokens from fail; and it's. The three decisions affect case, punctuation and grammatical information, respectively. In deciding, weigh which distinctions carry meaning and which only create an unnecessary number of different entries for the same information.

Normalization can shrink the vocabulary by merging the counts of some forms.

The normalization ladder

Operation	Possible benefit	Possible loss or cost
Lowercasing	Pump = pump: two columns merge into one and their counts add up	US and us become the same form; str.lower() does not apply the Turkish mappings I–ı and İ–i
Removing punctuation	by removing selected marks, can match fail; with fail	deleting indiscriminately can change identifiers such as P-101, decimals and URLs
Stop word removal	leaves the listed words out of the count: 43.9% of all tokens on this chapter’s corpus	if not is on the list, negation can be lost
Stemming	can merge word forms into one output: Porter turns replaced into replac	can wrongly merge different words: university and universe both become univers
Lemmatization	aims at the dictionary form: replace for replaced used as a verb	can need a dictionary, rules, part-of-speech or context; can err on ambiguous forms

This table is for a pipeline you build yourself. If the text is going into a pre-trained model, stay with the tokenizer and the preprocessing assumptions that model was trained under.

Normalization, in this chapter: the general term for operations that reduce orthographic or morphological differences judged not to matter for the task. Unicode normalization is a separate operation: it brings character sequences that look the same into one standard form.

These operations can delete useful information too. Which distinction an operation removes can be read from its definition; what that does to the task has to be evaluated.

The chapter’s table has a fourth column, a recommendation for each row. Lowercasing: decide by language and task; with a pretrained model follow the model’s own preprocessing rules. Punctuation: instead of deleting in bulk, decide by task which marks to keep, split off or remove. Stop words: measure the effect, especially with sparse count-based methods, and check that the list does not delete words that matter for the task. Stemming: try it in lexical search and sparse models, and decide by looking at the outputs and at task performance. Lemmatization: before adding it as an extra step for a pretrained transformer, check what input the model expects.

The 43.9% is measured with the English stop word list used in this chapter, on the 20 Newsgroups corpus and the tokenization rule shown later.

Adding these operations on your own to the input of a pre-trained model moves your input away from the form the model expects.

Reading the ladder

• The table is not a required order of application; stemming and lemmatization are in most cases alternatives to each other
• Each operation rests on the assumption that the distinction it removes is not needed for the task
• Lowercasing can help in some topic classification tasks, and removes a cue that comes from capital letters in named entity recognition (NER)
• Take the model’s requirements and the properties of the data into account

The recommendations are a starting point: compare the suitable options on validation data, and keep the test set apart for the final evaluation.

The example sentence called for three decisions; the ladder generalises them. Which distinction an operation removes can be read from its definition. What that does to the task has to be measured.

Lowercasing is the worked case on the slide: the same operation helps one task and removes a cue another task uses. Ask the room for the assumption behind each of the other rows of the ladder.

Lowercasing Turkish: the default is not Turkish

"İSTANBUL".lower()              # 'i̇stanbul': starts with U+0069 + U+0307; 9 code points in all
"İSTANBUL".lower() == "istanbul"  # False
"IĞDIR".lower()                 # 'iğdir' — should be 'ığdır'

• Python’s str.lower() applies Unicode’s default lowercase mapping; it takes no locale parameter for Turkish. Turkish has two separate mappings: I → ı and İ → i
• The first result starts with U+0069 + U+0307, not with a plain i
• No error is raised, but spellings of the same word may fail to match: the default CountVectorizer produces the token stanbul from the input İSTANBUL

Once the preprocessing is fixed, the vocabulary and the models that depend on it may have to be rebuilt.

Start with the first row of the ladder: lowercasing needs care with the letter mappings of the language.

What is Unicode? Computers store text by encoding it as numbers. Unicode is the standard that gives the characters of many writing systems shared numbers, called code points: A is 65, ç is 231, İ is 304. A sign that looks like one letter on screen is sometimes made of more than one code point, which is what happens to the lowercased İ here.

These conversions raise no error, but they can produce unexpected results downstream: different spellings of the same word may fail to match, or the tokenizer may split the word in an unexpected way.

Chapter 4 shows the problem on a corpus, and the Turkish NLP chapter covers a conversion that suits Turkish.

Stemming and lemmatization do different things

	What it does	Examples
Stemming (Porter)	rules aimed at inflectional or derivational suffixes; no guarantee that the result is a word found in a dictionary	replaced → replac; university and universe → univers
Lemmatization	aims at the base form in a dictionary: the infinitive of a verb, the uninflected form of a noun	replaced (as a verb) → replace; left can need context
Snowball Turkish stemmer	the stemmer used in this book	kitaplarımızdan → kitap; kullanıyorum → kullanıyor; kullandık → kulla

• With a pretrained transformer, use the model’s tokenizer and the preprocessing it expects; do not add stemming or lemmatization on your own
• Stemming can help in lexical search or sparse classification, but check which forms your tool actually merges: Snowball does not merge kullanıyorum and kullandık

Porter 1980

Both are used to gather different forms of a word under one form, but they do not do the same thing.

Lemmatization: the form you would look the word up under. Some methods use tables and rules alone; choosing the right result for an ambiguous form can need context. In English, left can be the past tense of leave in one sentence and the word for a direction in another.

kitaplarımızdan means “from our books”, kullanıyorum “I am using” and kullandık “we used”. The Snowball version used is snowballstemmer 3.1.1.

One measured gain. The Turkish NLP chapter runs a BM25 experiment with queries built from the target documents, most of whose terms are in a different inflected form from the one the document uses. There Snowball stemming raises the share of queries whose target document comes first (recall@1) from 28.0% to 95.3%. The result shows what stemming can gain in that experimental design; the same gain should not be expected on real user queries.

Stop words: 0.4% of the vocabulary, 43.9% of tokens

• scikit-learn’s English stop word list has 318 entries; in this chapter’s 20 Newsgroups measurement, 311 of them occur in the corpus under the tokenization rule used
• They are about 0.4% of the vocabulary (78,649 distinct token types) but 43.9% of all tokens: nearly half of the tokens are left out of the count
• That share does not mean the text shrinks by the same share in bytes, or the sparse matrix in memory
• If a word that matters for the task, such as not, is removed, the information needed to tell meanings apart can be lost

Weigh the information lost when you remove stop words. Whether that is a saving or a mistake depends on the task.

Stop word: a word so common that some pipelines throw it away before counting anything — the, of, is.

Vocabulary: the finite set of token ids a tokenizer can emit.

The memory saved also depends on how many documents the removed words occur in, so the 43.9% cannot be read as a saving of the same size.

We come back to this list twice: in the negation example, where it deletes the one word that tells two sentences apart, and in the classification experiment, where removing stop words raises accuracy.

The limit of the word level

• For the rest of this chapter a token is a normalized word form; that unit is enough for bag-of-words and TF-IDF
• Once the vocabulary is fixed, new text can contain word forms that are not in it: a word-level representation has no column for those forms and leaves them out of the counts
• In an agglutinative language such as Turkish, where one stem has many inflected forms, that happens more often
• Today’s language models get past this limit with units smaller than a word: the vocabulary holds pieces of words learned from data (subwords), and a new word is written with those pieces

Subword methods are covered in chapter 8, together with the pretrained models that use them. For now we continue with words.

We build the vocabulary from the training text and count how often each word occurs in each document. That choice has a limit, and it is worth knowing from the start.

The next section measures how the vocabulary keeps growing as the data grows, with Heaps’ law.

Chapter 8 is also where the book measures why the same text is cut into more tokens in Turkish. None of that is needed for this lecture: from here on a token is a word.

Zipf and Heaps: two separate questions

1. How are frequencies distributed between the commonest and the rarest words in a corpus? Zipf’s law
2. As you add more text, how does the number of distinct words grow? Heaps’ law

• Zipf’s law describes how frequencies are distributed within a given corpus; Heaps’ law describes how the number of distinct word forms changes as the corpus grows
• The two can be connected under specific assumptions

Both are approximate, observed regularities; neither dictates how tokenization has to be done.

Lü, Zhang & Zhou 2010

So far we have seen how to cut text and what each choice costs. This section looks at two empirical relationships used to describe how word frequencies are distributed and how the number of distinct word forms grows as data is added.

They are easy to conflate, which is why the chapter states them as two separate questions. We look at the frequency distribution first and vocabulary growth second.

20 Newsgroups: 2.29 million tokens, 78,649 types

tokens: 2288401  distinct types: 78649

top     10 types -> 20.99% of all tokens
top    100 types -> 46.99% of all tokens
top   1000 types -> 71.04% of all tokens
top  10000 types -> 92.50% of all tokens

types appearing exactly once: 38571  (49.0% of the vocabulary,
                                      1.69% of all tokens)

• A type is each distinct string our splitting produces; a token is each occurrence: pump a hundred times is one type and a hundred tokens
• The ten most frequent types make up about a fifth of the tokens, while about 49% of the types are seen exactly once

The data: the 11,314-document training split of the 20 Newsgroups collection (Lang 1995). After applying the options that strip headers, signatures and quoted text, we lowercase the text and count every string matched by the regular expression [a-z']+ as a token. That yields 2,288,401 tokens. These units do not always coincide with linguistic words, and both counts depend on the normalization and splitting rules.

The single-occurrence types are called hapax legomena. They are nearly half of the observed types and only 1.7% of the tokens in this corpus. Whether all of them belong in a model’s vocabulary depends on the task and on how the vocabulary is built.

This result shows that most observed types are rare. How many new types more data would bring is a separate question, taken up with Heaps’ law.

Zipf’s law: an approximate power relationship

\[f(r) \propto r^{-\alpha}\]

• \(r\) is the type’s position in the frequency ranking, \(f(r)\) its number of occurrences; on log-log axes the relationship looks approximately like a straight line
• Take the 106,425 occurrences of the as the scale and \(\alpha = 1\): about 10,600 expected at rank ten. The observed value is 26,985, roughly two and a half times that
• The Zipf–Mandelbrot form, \(f(r) \propto (r + b)^{-\alpha}\), gives the head of the distribution more flexibility

rank	occurrences
1	106,425
2	62,043
10	26,985
97	3,027
1,113	221
10,352	13
77,121	1

Take the straight line as a reference that summarizes the shape of the distribution, not as a model estimated from the data.

Zipf’s law (Zipf 1949) is the observation that, when types are ranked by frequency, frequency follows an approximate power relationship with rank.

“Approximately” matters. The comparison at rank ten shows that the first ranks do not follow the 1/r curve we chose. We do not estimate the parameters here or compare how well the two forms fit.

The table holds seven of the ranks the chapter’s interactive plot draws, out of 78,649 types. On linear axes the differences between rare types are squeezed into the bottom of the plot; logarithmic axes make them easier to see. The distribution does not change, only its appearance.

Deep dive, for those who ask about estimating the exponent. Fitting a straight line by least squares to a log-log rank–frequency plot can describe the shape of a distribution, but the slope alone is neither a reliable estimate of a power-law exponent nor evidence that a power law fits. There are three separate questions: estimating the coefficient, computing its standard error, and whether a power law is an appropriate model at all. The approach of Clauset and colleagues (2009) estimates a power-law model by maximum likelihood together with a lower cutoff estimated from the data, assesses the fit with the Kolmogorov–Smirnov distance rather than R², and compares the power law with alternative distributions. The exponent that method estimates belongs to the distribution of occurrence counts; it is not the same parameter as the rank–frequency exponent above. The chapter does no such estimation, and no result depends on the exact value of the exponent.

Three points for vocabulary design

1. A fixed word vocabulary may not cover new input completely. Text whose topic and source are not restricted keeps bringing new names, part numbers, spellings and compounds
2. Reliable statistics are hard to learn for rare types, yet a rare term can still matter. On this corpus, with CountVectorizer’s defaults, min_df=2 removes 62,208 of 101,631 columns (61%); the removed columns carry only 3.4% of token occurrences. The English stop word list removes 309 columns, but those carry 43% of the occurrences
3. How frequent a term is does not by itself determine how useful it is for a task. Terms found in many documents may do less to tell documents apart; but occurring in few documents is no guarantee of meaning or usefulness either

The numbers belong to this corpus and this threshold; how the filter affects accuracy has to be measured separately.

Three practical points the observations bring out for vocabulary design and feature selection.

Point 2: a single observation is weak ground for generalizing, but a rare product code or proper name can be decisive for a task. scikit-learn’s min_df parameter drops terms that occur in few documents: an integer is a number of documents, a float a proportion of documents. The 309 columns are two short of the 311 on the stop word slide because the default setting does not produce the one-letter tokens a and i.

Point 3: IDF, later in the lecture, takes that spread across documents into account. The 'ax example two slides ahead shows the other half: rare across documents and still not useful.

Heaps’ law: the vocabulary keeps growing

|V| ~ 16.5 * N^0.580
new types in the first 100,000 tokens: 13497
new types in tokens 2,180,001-2,280,000: 1814

\[|V(N)| \approx K N^{\beta}\]

• \(K\) and \(\beta\) depend on the corpus and on the preprocessing: here \(K \approx 16.5\) and \(\beta \approx 0.58\); the values reported for Reuters-RCV1 are \(K \approx 44\) and \(\beta \approx 0.49\)
• The first 100,000 tokens bring 13,497 types; the last 100,000 tokens we measured bring only 1,814 new ones. New types keep arriving, but ever more rarely.

Heaps’ law is an empirical regularity that models that growth approximately over a given range, not a proven theorem.

Heaps 1978 · Manning, Raghavan & Schütze 2008

The second question: as the number of tokens processed, N, grows, how many distinct types do we meet? We measure the cumulative number of types along the corpus every 10,000 tokens, then fit a straight line to that curve in log-log space.

Read the fit carefully. The line is a descriptive summary of cumulative growth in the present document order. Because we measure ever-longer prefixes of the same token sequence, the points are not independent samples. The coefficients belong to this order and this range; they should not be read as a confidence interval or as a prediction validated on new data.

With K > 0 and 0 < β < 1 the continuous model grows, but ever more slowly. Because β > 0, its value grows without bound as N does. That is a property of the model; it does not mean that finite data has proved the language will produce new types forever. The measured type count is also a step-wise integer sequence, and the power curve is an approximate summary of it.

The effect of the min_df filter does not follow from Heaps’ law; it follows from how types are spread over documents, that is, from document frequencies. min_df does not change the source text; it only drops features from the vectorizer’s vocabulary.

Always look at your top-50 list

In this count the second most frequent token is 'ax, with 62,043 occurrences, ahead of to, a and of. It is not a word of natural language.

• All of its occurrences sit in 13 of the 11,314 documents; twelve of those are parts 3–14 of a fourteen-part posting of a file called roman.bmp, with lines of the form >'AX>'AX>'AX
• One document alone, part 12, accounts for 11,147 of the 62,043 occurrences
• Second by total count, yet found in only 0.1% of the documents: frequency in the corpus as a whole and spread across documents are different things
• TF-IDF does not recognize non-language content by itself: a noise string that occurs in few documents can receive a high IDF

Looking at the list of most frequent tokens is a simple way to catch data and preprocessing problems before any weighting.

The bodies of those twelve postings hold the file’s content encoded as text (uuencode), and runs of identical bytes produce the repeated lines. The thirteenth document is a separate posting with another encoded file and contributes a single occurrence.

Where the apostrophe comes from: it is already in the raw data. Our regular expression [a-z']+ keeps that character inside the token, and lowercasing turns AX into ax.

TF-IDF uses both pieces of information, frequency in a document and spread across documents. That is not the same as recognizing non-language content.

Three counts to keep apart

1. the number of times a term occurs in one particular document
2. its total number of occurrences in the whole corpus
3. the number of documents that contain it

• For 'ax the corpus total is high and the number of documents containing it is low
• The form of the token depends on a preprocessing decision: scikit-learn’s default token_pattern produces ax from the same documents, and ax still ranks second after the, with 62,387 occurrences

TF-IDF combines the term’s frequency in each document (TF) with an IDF weight computed from how many documents contain the term.

A bridge slide: these three counts are the vocabulary of the rest of the lecture.

scikit-learn’s default token_pattern does not include the apostrophe. The source content stays the same, but the form of the token depends on a preprocessing decision.

Ask the room which of the three counts each of these is: term frequency, the frequency Zipf’s law ranks, and document frequency. The answers are the first, the second and the third.

From string to vector: six steps

corpus = [
    "the pump failed",           # d1
    "the pump was replaced",     # d2
    "the valve failed",          # d3
    "the seal leak was repaired", # d4
]

1. Index: failed→0, pump→2, the→6
2. One-hot: pump = [0 0 1 0 0 0 0 0 0]
3. Sum = counts: [1 0 1 0 0 0 1 0 0]

4. TF-IDF weights: [1.51 0 1.51 … 1.00 …]
5. L2 normalize: [0.64 0 0.64 … 0.42 …]
6. Compare: cos(d1, d3) = 0.528

Every number shown is the one the code actually produces for d1 = "the pump failed".

This is the core of the chapter. We take four short maintenance logs all the way from strings to vectors you can compute distances on, skipping no step and showing every intermediate number.

The whole chapter in one picture. Steps 1–3 build the vocabulary and the term counts and throw away word order; steps 4–5 weight those counts by document frequencies and scale them by the vector norm; step 6 compares the vectors.

The vocabulary index is alphabetical, so failed is column 0, pump is column 2 and the is column 6.

In this pipeline the search, clustering and classification steps use the properties of the text these vectors represent.

Step 1 · the vocabulary is just an index

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer()
X = cv.fit_transform(corpus)
print(cv.get_feature_names_out())

['failed' 'leak' 'pump' 'repaired' 'replaced' 'seal' 'the' 'valve' 'was']

• The column number is the term’s address in the vocabulary; by itself it says nothing about closeness in meaning
• Nine terms, so every vector from here on lives in \(\mathbb{R}^9\)
• The order in which the columns are numbered does not change any result, as long as training and use keep the same mapping

After preprocessing and tokenization we sort the distinct terms and map each one to a column number.

With its default settings CountVectorizer lowercases the text and counts runs of two or more letters or digits as tokens.

In word-level or n-gram systems the vocabulary can reach tens or even hundreds of thousands of entries. That size matters especially for sparse storage and for learning reliable patterns from rarely seen terms.

Step 2 · one-hot: a word as a vector

index of 'pump': 2
one-hot: [0 0 1 0 0 0 0 0 0]

pump  : [0 0 1 0 0 0 0 0 0]
valve : [0 0 0 0 0 0 0 1 0]
        ------------------- multiply position by position
        [0 0 0 0 0 0 0 0 0]  -> sum = 0

• The one-hot vectors of different words are orthogonal: under this encoding pump is exactly as similar to valve as it is to repaired
• The vector says which column the word is kept in, and nothing about what it is like

One-hot is an intermediate structure here, not the final way to represent word meaning. Learned dense vectors — embeddings — are chapter 3.

The numerical difference between two column numbers says nothing about how close two words are in meaning: pump at 2 is no closer to repaired at 3 than to valve at 7. So how do we turn a word into a vector a model can compute with? The simplest possible encoding: a vector of zeros with a single 1 at the word’s address.

Take the dot product of any two distinct one-hot vectors and you get zero: the single 1 in one vector always lands on a 0 in the other, so every product in the sum is zero. The vector of a word matches itself exactly (dot product 1); closeness in meaning between different words is not part of this encoding.

One-hot is used here as a way to see how a document count vector is built, which is exactly the next step.

Step 3 · a document is the sum of its one-hots

d1 tokens: ['the', 'pump', 'failed']
d1 vector: [1 0 1 0 0 0 1 0 0]

• Add up the one-hot vectors of every token in a document: a count vector. That is bag-of-words: the multiset of tokens, order discarded
• Do it for all four documents: the document-term matrix
• Standard layout: rows are documents, columns are terms — here 4 documents × 9 terms. The block is only shown transposed

          d1  d2  d3  d4
failed     1   0   1   0
leak       0   0   0   1
pump       1   1   0   0
repaired   0   0   0   1
replaced   0   1   0   0
seal       0   0   0   1
the        1   1   1   1
valve      0   0   1   0
was        0   1   0   1

One word, one vector. But we set out to compare documents, and a document is many words at once. The simplest possible answer works: add them up.

The document-term matrix is the central object of classical NLP.

A statistician will recognize the structure: a design matrix with many correlated, mostly-zero predictors. Each document is an observation and each term’s count is a feature — a variable that describes an observation; here each feature says how many times a particular term occurs in the document. If you come from statistics, this is a predictor variable under a different name.

Regularization, feature selection and generalization need attention here as well, and the assumptions of whichever method we use have to be checked separately.

Step 4 · does every word deserve the same weight?

• the appears in all four documents: it contributes to every comparison while distinguishing nothing. valve appears once and is the only thing that makes d3 distinctive. Raw counts give them the same treatment

To give a term a high weight we want two things:

1. It is frequent in this document. The factor that measures this is TF
2. It is not ordinary across all documents. The factor that measures this is IDF

• Term frequency is the raw count we already have: scikit-learn does not divide it by document length

TF-IDF is the product of the answers. “TF-IDF” on its own is not a specification: say which variant produced a number you report.

We now have term counts. The next step is to decide how much weight each term should get when we compare documents. The formulas that follow are one particular way of turning the two questions into numbers.

On dividing by document length: had we divided the counts by document length first, the same IDF weights and the L2 normalization in step 5 would still give the same vector, because a common positive factor applied to every component of a row cancels in the final division. That does not mean every effect of document length disappears.

Practice note: TF-IDF is not one formula. Many textbooks define term frequency as the count divided by document length, or as the logarithmic 1 + log of the count, and the resulting numbers differ. The classical shorthand for naming a variant is SMART notation, a three-letter code for (tf) × (idf) × (normalization). In that notation scikit-learn’s default is summarized roughly as ntc: natural (raw) tf, idf and cosine normalization. The code does not specify the details of IDF smoothing, though. Its counterpart with logarithmic tf is ltc, a common choice in classical IR. Setting sublinear_tf=True switches the TF component to the logarithmic weight; in the experiment at the end of the lecture that option’s accuracy is slightly higher than the basic TF-IDF configuration’s (0.843 against 0.831).

Why invert df — and why with a logarithm

• Document frequency, \(\mathrm{df}(t)\), is how many documents contain the word at all. We want the weight to fall as df rises, so we invert it
• Raw inversion, \(1/\mathrm{df}\), has far too wide a range: in a collection of a thousand documents, a term in one document gets 1 and a term in every document gets 0.001
• A logarithm compresses that range: \(\log(n/\mathrm{df})\) stays between \(\ln 1000 \approx 6.9\) and 0, and doubling the document frequency costs the same amount (\(\ln 2 \approx 0.69\)) wherever it happens

\[\mathrm{idf}(t) = \ln\!\left(\frac{n}{\mathrm{df}(t)}\right)\]

A second justification comes from information theory: in a model where documents are drawn with equal probability, \(\log(n/\mathrm{df})\) is the self-information of the event that the drawn document contains the term.

Spärck Jones 1972 · Robertson 2004

A word that occurs in every document does not distinguish documents by its mere presence; how often it is used in each can still differ.

A thousandfold gap can let rare terms contribute excessively to the weight in some documents. Those terms include distinctive technical names as well as typos.

The chapter’s intuition check on the logarithm, in three parts.

It compresses the range. In a collection of 10,000 documents, 1/df gives a term found in one document ten thousand times the weight of a term found in every document. log(n/df) stays between 9.2 and 0 and treats equal ratios equally: raising df from 1 to 10 costs the same weight (ln 10 ≈ 2.3) as raising it from 1,000 to 10,000.

Its relation to self-information. When a document is drawn from the collection with equal probability, the event “this document contains the term t” has probability df(t)/n, and unsmoothed IDF equals the self-information of that event. This reading does not guarantee that the term matters semantically or is useful for every task. The historical order is that Spärck Jones proposed IDF on empirical grounds in 1972, and the theoretical readings were developed later; Robertson’s paper is the careful treatment.

Its relation to Zipf. A logarithm turns ratios into differences and makes wide ranges of values easier to compare. But the total frequency of use in Zipf’s law and the document frequency in IDF are different quantities.

scikit-learn’s smoothed IDF, by hand

\[\mathrm{idf}(t) = \ln\!\left(\frac{1 + n}{1 + \mathrm{df}(t)}\right) + 1\]

• The \(1+\) terms are smoothing: the formula behaves as if the collection held one extra document containing every term once
• The trailing \(+1\) makes the IDF of a term that occurs in every document 1 rather than 0: the term is not zeroed out

\[\mathrm{idf}(\texttt{pump}) = \ln\!\left(\frac{1+4}{1+2}\right) + 1 = 0.511 + 1 = 1.511 \qquad \mathrm{idf}(\texttt{the}) = \ln\!\left(\frac{1+4}{1+4}\right) + 1 = 1\]

This is scikit-learn’s exact formula; other libraries differ in the details, which is a frequent source of “why don’t my numbers match”.

Libraries adjust the basic form a little. In a vocabulary extracted from the training corpus every term occurs in at least one document; a vocabulary fixed in advance, however, can contain terms never seen in training, with df = 0. The smoothed form handles a zero document frequency safely and keeps terms that occur in every document.

Read the formula one symbol at a time: t is a term, n the number of documents in the collection, and df(t) counts how many of them contain t at least once. The extra-document reading explains why the denominator always stays positive. For a fixed n the IDF is finite; as n grows, that upper bound grows too. The add-one resembles the Laplace smoothing of the language models chapter; the denominators of the two formulas are not the same.

Work the two by hand. Our corpus has n = 4 documents. pump appears in d1 and d2, so df = 2: ln(1.667) + 1 = 0.511 + 1 = 1.511. the appears in all four, so df = 4: ln(1) + 1 = 1.

The second calculation sums up the design. The presence of a word that occurs in every document carries no information about which document you are holding, so the term is left on the floor IDF value of 1. The term is not deleted: pump gets a factor of 1.51 and the a factor of 1.

The librarian’s version: ask for “the book about pumps” and the word the narrows the shelf down not at all, while pumps narrows it to a metre of shelving.

IDF on four documents, and on a real corpus

          df     idf
failed     2  1.5108
leak       1  1.9163
pump       2  1.5108
repaired   1  1.9163
replaced   1  1.9163
seal       1  1.9163
the        4  1.0000
valve      1  1.9163
was        2  1.5108

the            df=  9477  df/N= 0.838  idf= 1.18
is             df=  7034  df/N= 0.622  idf= 1.48
car            df=   394  df/N= 0.035  idf= 4.35
encryption     df=   169  df/N= 0.015  idf= 5.20
gravitational  df=     8  df/N= 0.001  idf= 8.14

• Left: the is in all four documents and lands exactly on the floor of 1.0; every word unique to one document gets 1.9163
• Right: on the real corpus the spread is much larger
• Computed by hand and checked against the library: the largest absolute difference is 0.0

The weight is a monotone decreasing function of how many documents a word appears in. It shows how widespread a term is across documents, not whether it is a content word.

The hand computation is three lines: n = len(corpus), then df = (X.toarray() > 0).sum(axis=0), then idf_manual = np.log((1 + n) / (1 + df)) + 1. The library’s idf_ attribute holds the same nine values.

The ordering is the point of the left block.

The right block is the 20 Newsgroups training split again, 11,314 documents: the is in 9,477 of them and gravitational in 8.

The chapter’s interactive curve, in words: under this formula a term that occurs in every document has an IDF of 1, the floor. The same increase in the share of documents changes the weight more at low shares; at higher shares the IDF still keeps falling.

Step 5 · multiply, then normalize

counts        : [1 0 1 0 0 0 1 0 0]
times idf     : [1.5108 0.     1.5108 0.     0.     0.     1.     0.     0.    ]
l2 norm       : 2.3591
normalized    : [0.6404 0.     0.6404 0.     0.     0.     0.4239 0.     0.    ]
sklearn row d1: [0.6404 0.     0.6404 0.     0.     0.     0.4239 0.     0.    ]

\[\lVert v \rVert_2 = \sqrt{1.5108^2 + 1.5108^2 + 1.0000^2} \approx 2.3591 \qquad \frac{1.5108}{2.3591} = 0.6404 \qquad \frac{1.0000}{2.3591} = 0.4239\]

• d1 is the pump failed: three positions are non-zero, each with a count of 1, each multiplied by its IDF
• The hand computation and TfidfVectorizer agree to the four decimal places displayed

That final division by \(\lVert v \rVert_2\) is L2 normalization.

Two ingredients are on the table: a count that says how often a word appears here, and a weight that says how unusual it is everywhere. Combining them is the last piece of arithmetic in this chapter: a multiplication followed by a division.

TF-IDF is the product, computed cell by cell. Multiply each count by its IDF and you get 1.5108 for failed, 1.5108 for pump and 1.0000 for the. The length of that vector is the ordinary Pythagorean one, extended to nine dimensions: square, add, take the root. The squares are 2.28259, 2.28259 and 1.00000; their sum is 5.56518. Intermediate values are rounded for display; carried at full precision the norm is 2.35906509…, which is what the library uses.

On this example, comparing the two results without rounding also gives a largest absolute difference of 0.0. In general, floating-point results should be compared with a tolerance, for instance with np.allclose.

The six zero entries do not change the sum. To benefit from that on large vocabularies we keep the matrix in sparse form; the toarray() conversion used here to display intermediate values is not something to apply to large matrices.

What L2 normalization changes

Why should two documents that use similar terms in similar proportions — one 20 words long, the other 2,000 — look far apart only because one is longer?

• L2 normalization brings the norm of a non-zero vector to 1 while keeping the ratios between its components: every non-zero document vector moves onto the unit sphere
• Without it, every comparison based on a dot product or Euclidean distance — including a linear classifier’s score — becomes partly a comparison of vector sizes
• A document left with no active term gives the zero vector, and cosine with it is \(0/0\), undefined. scikit-learn returns a cosine of 0, which is a convention: count these rows and handle them separately
• Raw TF, smoothed IDF and L2 is scikit-learn’s default; other tools may choose a different formula and normalization

Only direction, the weighted mix of terms, is left in the comparison. Its effect on classification accuracy is something we measure on validation data.

Salton & Buckley 1988

In longer documents some term counts and the vector norm can grow, but not every term’s count necessarily does. After normalization the comparison turns on the vector’s direction rather than its absolute size. Cosine divides by the norms itself, so it is unaffected.

Think of it as putting every document on a sphere of the same radius. A 50-word report and a 5,000-word report both end up at unit length — the squares of the components sum to one, not the components themselves — so the question becomes which direction each one points, not how far it reaches.

The zero vector. The cause can be empty text, only out-of-vocabulary terms, or every term being filtered out. Do not read the 0 the library returns for such rows as a measured semantic dissimilarity between two valid documents.

The particular combination built here — term frequency, times inverse document frequency, then length-normalized — is not the only one anyone tried. Salton and Buckley compared different term-weighting schemes empirically.

Step 6 · cosine compares direction, not length

\[\cos(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert} = \sum_{i} a_i b_i \quad \text{(when both are already unit length)}\]

\[\cos(\text{d1}, \text{d3}) = \underbrace{0.6404 \times 0.5729}_{\texttt{failed}} + \underbrace{0.4239 \times 0.3792}_{\texttt{the}} = 0.3669 + 0.1607 = 0.528\]

• For the non-zero vectors without negative components used here, 1 means identical direction and 0 means no shared term with positive weight
• failed supplies 70% of the similarity and the supplies 30%, even though each word occurs exactly once in each document; under raw counts they would have contributed equally

The weighting is what decides which shared word counts as evidence.

Every document is now a point in nine-dimensional space. That lets us ask the question the chapter has been building towards: how similar are the weighted term mixes of these two documents?

Two normalized vectors are compared by their dot product, which is the cosine of the angle between them. A short note and a long report can get a high cosine score when their weighted term proportions are similar. The score alone does not prove that they share a topic or a meaning.

The reading “1 is identical direction, 0 is no shared term” does not carry over directly to representations that can have negative components.

Compute one cell by hand. d1 is the pump failed and d3 is the valve failed; they share failed and the and disagree everywhere else. Only positions where both vectors are non-zero can contribute, so the whole nine-dimensional sum has exactly two live terms.

The same four documents, two weightings

       d1     d2     d3     d4          <- TF-IDF
d1  1.000  0.458  0.528  0.112
d2  0.458  1.000  0.125  0.286
d3  0.528  0.125  1.000  0.100
d4  0.112  0.286  0.100  1.000

       d1     d2     d3     d4          <- raw counts
d1  1.000  0.577  0.667  0.258
d2  0.577  1.000  0.289  0.447
d3  0.667  0.289  1.000  0.258
d4  0.258  0.447  0.258  1.000

• Under raw counts, d1 and d4 sit at a cosine of 0.26, but they share only the word the; by reducing the relative contribution of the shared the, TF-IDF brings that score down to 0.11

A cosine is the cosine of the angle between the two vectors. It is not a calibrated percentage of semantic similarity, which is what reporting it as “26% similar” implies.

The two panels come from cosine_similarity applied to the TF-IDF rows of step 5 and to the raw counts of step 1. Row and column labels were added to the output for readability.

In this example every score between different documents drops, and the proportional drop is largest for the pairs that share only the. Each document’s similarity with itself stays at 1.

d1 and d3 share failed and the; d1 and d2 share pump and the. Ask the room which pairs share only the, and check their answers against the smallest numbers in the top panel.

Why cosine and not Euclidean distance?

Pair, as raw count vectors over two words	Euclidean distance on raw counts	After L2 normalization
A (2, 1) and B (20, 10)	20.12 — far away, but B is exactly 10 × A	the same point
A (2, 1) and C (1, 4)	3.16 — nearest, but C has different term proportions	an angle of 49.4°: \(\cos = 0.651\), distance 0.836

• On raw counts, Euclidean distance puts A more than six times closer to C than to B
• After L2 normalization all vectors lie on the unit sphere, and there the two measures are monotonically related: \(\lVert a - b \rVert^2 = 2 - 2\cos(a,b)\)
• For L2-normalized non-zero vectors the two give the same ranking in an exact nearest-neighbour search; numerical thresholds have to be converted

Cosine looks only at direction: it compares the proportions of the terms, not their total amount. This is why classical text pipelines normalize before they compare.

The chapter’s figure, as a table. Every term count of B is exactly ten times A’s, so A and B lie on the same ray out of the origin. After L2 normalization every vector has length 1, A and B become the same point, and the only thing left to measure is the angle between directions.

Two reports of the same failure can produce vectors of different magnitude but similar direction if their term proportions are similar. The extra detail in a longer report can change the direction too, however.

Check the identity against the table: cos = 0.651 gives a squared distance of 2 − 2(0.651) = 0.698, and the distance is its square root, 0.836. Before normalization that pair was 3.16 apart.

Where 2 − 2cos comes from. Expand the squared distance: (a − b)·(a − b) = a·a − 2(a·b) + b·b. Every normalized vector has length exactly 1, so a·a = 1 and b·b = 1, and their dot product is the cosine. Distance is then a strictly decreasing function of similarity: as the cosine climbs from 0 to 1, the squared distance falls from 2 to 0.

The limits of the equivalence: it does not mean that every clustering method or approximate search implementation will return the same result, and without normalization the ranking equivalence no longer holds. The identity turns up again in chapter 3 whenever someone swaps a cosine index for an L2 index and expects the same neighbours.

What bag-of-words cannot do · word order

pair = ["the pump failed and the valve is fine",
        "the valve failed and the pump is fine"]

vocabulary: ['and','failed','fine','is','pump','the','valve']
row 1: [1 1 1 1 1 2 1]
row 2: [1 1 1 1 1 2 1]
vectors identical: True
cosine similarity: 1.0

• It cannot represent word order. Not “represents it poorly” — cannot represent it at all
• Two reports naming opposite pieces of failed equipment are the same point in space

Information discarded at the representation stage cannot be recovered by a stronger classifier.

Now look at what this representation loses while the vector is still being built. These losses explain why the order- and context-based representations of the later chapters are needed.

No classifier, however good, can separate the two reports, because the information was destroyed before the classifier saw it.

n-grams buy back a little

bigram vocabulary: ['and the','failed and','is fine','pump failed',
                    'pump is','the pump','the valve','valve failed','valve is']
cosine similarity: 0.714
vocabulary size: unigrams 7 -> bigrams 9

• Using pairs of adjacent words instead of single words, the two reports are distinguishable
• The cost: in the experiment below, adding bigrams took the feature count from 32,318 to 249,522, a 7.7× increase, and accuracy fell
• For jobs like language identification, spam filtering or short-text classification, n-grams can be enough on their own

The feature space grows quickly as the context gets longer; long-range, context-sensitive relations need a more general representation.

n-grams buy back some local order information. That gain has a cost: on real corpora an n-gram vocabulary can become very large. How large depends on the corpus, on the n-gram range and on the filters applied.

The more general representation is what the transformers chapter is about.

What bag-of-words cannot do · negation and meaning

"no leak was found"  vs  "a leak was found"
vocabulary (the default token_pattern drops the one-letter "a"):
    ['found', 'leak', 'no', 'was']
raw counts  ->  cosine 0.866
TF-IDF      ->  cosine 0.777

• not is a feature like any other; what a bag of words cannot recover is what was negated
• If your stop word list contains no — as scikit-learn’s built-in English list does — the two vectors become identical (cosine 1)
• It does not encode synonymy or polysemy directly: pump failed and pump broke share nothing beyond pump

In tasks where negation changes the decision, the stop word list needs to be checked separately.

It cannot represent the scope of negation. A classifier can learn that the presence of not shifts the odds. Two sentences with opposite operational meaning sit at cosines of 0.78 to 0.87 depending on the weighting. If no is kept, the vectors differ and a classifier can use that difference; which expression the negation attaches to, however, is not encoded directly by single-word counts. With no deleted by the stop word list, a model that uses only these vectors cannot tell the two sentences apart.

Synonymy and polysemy. In a lowercased bag of words, Apple the company and apple the fruit land in the same column; keep the case and they are two columns, which is a direct consequence of the preprocessing decision on the normalization ladder. In this representation different terms sit in separate columns, and the semantic relation between them is not encoded directly, as we saw with one-hot vectors in step 2. The dense representations of chapter 3 help to learn those relations. Separating the senses a word takes in different uses additionally requires taking context into account.

Does any of this still matter?

Four newsgroups, 2,327 training documents, 1,550 test documents, logistic regression; no hyperparameter search was done.

raw counts               accuracy 0.767   features   32318
counts + stop words      accuracy 0.795   features   32014
tf-idf                   accuracy 0.831   features   32318
tf-idf + stop words      accuracy 0.853   features   32014
tf-idf + bigrams         accuracy 0.819   features  249522
tf-idf + sublinear tf    accuracy 0.843   features   32318

This table is a demonstration of a mechanism and a baseline comparison. Do not read small differences as universal superiority.

Yes, and you should be able to say why with numbers rather than opinion.

The four categories are sci.med, sci.space, rec.autos and talk.politics.guns, with headers, footers and quotes removed. Each row is a pipeline of a vectorizer and LogisticRegression(max_iter=2000, random_state=0); max_iter=2000 was set for convergence. The six vectorizers: CountVectorizer(), the same with stop_words="english", TfidfVectorizer(), the same with stop_words="english", with ngram_range=(1, 2), and with sublinear_tf=True.

The next two slides read the six lines and then say, in numbers, why small differences should not be over-read.

Four conclusions from six lines

1. TF-IDF is worth 6.4 accuracy points over raw counts, without collecting more training documents and without a hyperparameter search
2. Stop word removal helps here (+2.2 points); in sentiment analysis or negation detection the same step can hurt if the list removes words that matter for the task
3. Accuracy fell when bigrams were added, while the feature count grew 7.7-fold. More features do not guarantee higher accuracy
4. sublinear_tf points the same way (0.843 vs 0.831): it replaces the raw count with \(1 + \log(\text{tf})\), the range-compression argument we made for IDF, applied to the term-frequency side

scikit-learn’s own documentation calls stop_words='english' problematic. If stop words matter to your result, write the list yourself and print it.

Conclusion 1: computing IDF and normalizing still take extra computation.

Conclusion 2: on this dataset and this topic-classification task, the function words the list removes do not appear to carry a strong signal for separating the classes. The gain here does not by itself show that every removed word was useless. On the list itself: it is not a general-purpose list, its selection criteria are undocumented, and a few of its entries can never be produced by the default token_pattern in the first place.

Conclusion 3: the added sparse n-grams can raise the risk of overfitting; pinning down the cause of the difference in this experiment would take a separate comparison of regularization and feature selection.

How uncertain are these numbers?

• On 1,550 test documents, a single accuracy near 0.83 carries a 95% interval of roughly ±1.9 points (binomial approximation)
• The six models see the identical test documents: the setting for a paired test such as McNemar’s

Comparison	Only the first correct	Only the second correct	p
TF-IDF and raw counts	119	20	2.4 × 10⁻¹⁸
TF-IDF + stop words and TF-IDF	86	52	0.0048
TF-IDF + bigrams and TF-IDF	17	35	0.018
TF-IDF + sublinear tf and TF-IDF	29	11	0.0064

With four comparisons, a Bonferroni correction puts the threshold at 0.05/4 = 0.0125. The TF-IDF, stop word and sublinear_tf differences fall below it; the bigram difference does not.

McNemar 1947

A caution before you quote any of the six numbers: they are point estimates with no intervals attached. The binomial approximation treats documents as independent. If two accuracies are treated as independent estimates, the interval of their difference is roughly ±2.6 points. Those intervals give a rough idea of the scale of the uncertainty; but the six models are measured on the same test documents, so deciding about the differences under an independence assumption is not appropriate.

McNemar’s test compares the number of documents one model classifies correctly and the other incorrectly; documents that both models get right, or both get wrong, do not affect the result. The p-values come from the exact McNemar test, which is based on the binomial distribution.

These p-values measure only the uncertainty from sampling the test documents. They rest on a single train–test split and a single training run, and do not generalize to other datasets or tasks. Because six configurations are compared, selection of results and multiple comparisons also have to be taken into account.

Two of the four lessons reverse on another task

• Every number here comes from English topic classification on long documents
• On Turkish product reviews — short documents, sentiment rather than topic — the first table in chapter 4 points the opposite way on both counts: at a fixed label budget bigrams give the largest gain, and plain counting beats TF-IDF
• But the second result does not come from IDF: that chapter’s ablation shows L2 normalization costs the points, and adding IDF on top wins them back
• The result hinged on an untuned C and reverses under an equal tuning budget

The transferable lesson is the method, not the ranking.

One more warning about generalising from six lines.

The obvious explanation for “plain counting beats TF-IDF” would be that IDF down-weights the frequent sentiment words. Chapter 4’s ablation refutes it.

The point to carry: give both sides of a comparison the same tuning budget before you credit the representation with the difference.

Readable, and small

rec.autos            car, cars, engine, ford, oil, dealer, drive, price
sci.med              msg, doctor, disease, medical, pain, patients, treatment
sci.space            space, nasa, orbit, moon, launch, earth, spacecraft
talk.politics.guns   gun, guns, weapons, fbi, law, government, firearms

shape: (2327, 32318)
stored non-zeros: 234281
density: 0.312%
dense float64 would need :    601.6 MB
sparse csr actually needs:      2.8 MB
ratio: 213x

• The terms with the largest coefficients for each class; a large coefficient is not by itself a causal explanation
• The matrix is 99.7% zeros: 2.8 MB instead of 601.6 MB. That is the storage cost of the sparse matrix alone

The other reason to care is that the model is readable. Ask a TF-IDF-plus-logistic-regression model what it learned and it will tell you, in words. In a linear bag-of-words model the coefficients can be tied to the terms or n-grams of the vocabulary; in a multi-class model we inspect them class by class. The contribution to the score of a particular document depends on the product of the coefficient and the feature value.

Dense representations rarely offer a term–coefficient mapping that clear. Attribution, SHAP-style explanations and probing exist for dense models too; none of them reads as directly as this table. When you have to explain a model to a safety board, a regulator or a sceptical plant manager, that readability makes it easier to examine errors and to discuss the model’s decisions. The trade-off against accuracy is set by the requirements of the application.

The computational cost is low as well. This example needs no GPU. Computing the classification scores comes down to sparse dot products; end-to-end inference also includes preprocessing and vectorization. The vocabulary, the IDF values, the model coefficients and working memory come on top of the 2.8 MB. Training time depends on hardware and solver settings: on the CPU of the laptop used for the chapter’s measurements, each of the six configurations finished in under 2 seconds, including vectorization, training and prediction on the test set.

Bag-of-words, honestly

Why bag-of-words survives	Why it is not enough
A small sparse matrix. In this example the matrix for 2,327 documents takes 2.8 MB. No GPU needed	No word order. pump failed, valve fine and valve failed, pump fine are the same vector
Legible. The model names its own features — gun, orbit, doctor	No synonymy. car and automobile are as unrelated as car and banana
Strong baseline. 0.853 here, and it beat averaged embeddings in chapter 3 — then beat them again on Turkish data in chapter 4	No polysemy. One column for bank, whatever it meant
No training data beyond your own. Nothing pretrained, nothing downloaded, no vendor	Out-of-vocabulary terms can occur in new documents
Reproducible with a fixed pipeline. The same input gives the same output	Negation has no scope. What a bag cannot recover is what was negated

The two columns show the strengths and the limits of one and the same representation. Which side weighs more depends on what the task requires.

The whole chapter’s evidence in one place. The right reading is not “old technology, still surprisingly good.”

Small: total model memory is larger than the matrix. Legible: that makes it easier to inspect how terms contribute to a score. Reproducible: with preprocessing, vocabulary, IDF and model parameters fixed. Out-of-vocabulary: new documents can contain terms that are not in the fixed vocabulary built in training. Negation: not is a feature like any other and can carry signal, and removing stop words deletes it outright.

When exact term matching, speed and traceability matter, bag-of-words is a strong starting option. Make the final decision by weighing accuracy, latency, memory and the kinds of error together.

When the task needs order or context information, the representation has to be extended beyond single-word counts. N-grams can help with local relations, and character features with new forms. For broader semantic relations we look at dense and contextual representations, starting with chapter 3.

Where this is actually deployed

Where	The idea from this chapter, in use
Ticket triage at Uber	COTA, described in 2018: clean, tokenize, remove stop words, lemmatize, TF-IDF, reduce with LSA; a cosine similarity from those vectors feeds a random-forest matching model
Search ranking	Lucene, and therefore Elasticsearch and OpenSearch, scores documents with BM25 by default
Google’s text classification guide	reports TF-IDF n-grams as the most accurate of the encodings it compares
Hybrid retrieval	dense embedding search combined with sparse BM25 search: the sparse half catches the exact matches
Spam and abuse filtering	sparse features as a first-line filter; feature hashing maps terms to a fixed number of columns
Near-duplicate detection	simhash fingerprints; MinHash over sets of shingles, fixed-length windows of words

For tasks it suits, such as text classification and lexical retrieval, include a TF-IDF baseline in the comparison.

Some of these rest on dated system reports published by companies, others on method papers; read the architectures described as limited to the period those sources cover. Three new terms, each an applied form of an idea from this chapter: BM25, the TF-IDF family’s weighting, developed for search; feature hashing, a way to get a fixed-size sparse representation without holding a vocabulary in memory; shingling, an n-gram representation that keeps part of the word order.

Uber (Uber Engineering 2018). LSA is latent semantic analysis, a truncated SVD of the TF-IDF matrix. The cosine similarity goes to the matching model together with ticket and trip features, and candidate solutions are ranked by that model’s score. According to the write-up the system can reduce ticket resolution time by over 10%, and the ranker is 25% better in relative accuracy than a multi-class classifier fed the LSA topic vectors directly.

Search (Robertson & Zaragoza 2009). BM25 is a common scoring method in many Lucene-based text search systems; it cannot be assumed that every search box uses it.

Google’s guide (Google Developers 2022). In its experiments on English text, accuracy levels off at around 20,000 selected features for many datasets. That is not a fixed threshold for every language.

Hybrid retrieval: the exact matches are part numbers, error codes and surnames, which embeddings can miss. The retrieval chapter builds one; the RAG chapter wires it to a language model.

Spam filtering (Weinberger et al. 2009). Feature hashing stores no explicit vocabulary; in exchange, different terms can fall into the same column, and tracing a coefficient back to a single term becomes harder.

Near-duplicates. Google’s web-crawl deduplication work, published in 2007 (Manku et al.), computes simhash fingerprints from weighted document features; MinHash is Broder 1997. The three-word shingles of the pump failed last night are the pump failed, pump failed last and failed last night: the same structure as the n-grams used above to recover some word order.

The practical rule: reach for TF-IDF when your labels are few, your documents are long, your vocabulary is technical and exact-match matters, or when you need a baseline quickly. If your transformer cannot beat it, you have learned something important about your data, and you have learned it cheaply.

BM25 in one formula

\[\mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{idf}(t) \cdot \frac{c(t,d)\,(k_1 + 1)}{c(t,d) + k_1\!\left(1 - b + b\,\frac{|d|}{\mathrm{avgdl}}\right)}\]

• The \(\mathrm{idf}\) factor is the same idea as ours (BM25 uses a probabilistic variant)
• The fraction replaces raw term frequency with a saturating one: with the other variables fixed, it approaches the limit \(k_1 + 1\) as \(c(t,d)\) grows
• \(b \in [0,1]\) interpolates between ignoring document length (\(b = 0\)) and dividing the term frequency by the document’s length relative to the corpus average (\(b = 1\))
• In Lucene 9.12.1 the defaults of the BM25Similarity class are \(k_1 = 1.2\) and \(b = 0.75\)

The same concern L2 normalization addresses — long documents should not win on length alone — handled by a different formula and with a dial instead of a switch; the two are not the same operation.

Robertson & Zaragoza 2009

BM25 comes from the TF-IDF family and adds document-length normalization and a saturating term frequency.

In this simplified formula the sum runs over the distinct terms of the query; repeated query terms are not weighted separately. c(t,d) is the count of term t in the document, |d| the number of tokens in the analysed document, and avgdl the average document length in the same collection.

Saturating: additional repetitions contribute less and less. sublinear_tf also slows the effect of repetitions, but 1 + log(tf) grows without bound and is not the same function as BM25.

Intuition check 1 of 6

Your TF-IDF matrix has 200,000 columns and 5,000 rows. Should you worry?

• Not about memory: in a sparse matrix the real memory cost depends mostly on the number of non-zeros, not on rows × columns
• The real worry is statistical: 5,000 observations against 200,000 features. That is a p ≫ n problem
• The usual answers are strong regularization (L2 or L1) and cutting the feature count with min_df
• On the four-category corpus, min_df=5 takes the vocabulary from 32,318 down to 6,281 — 81% of the columns gone. The deleted columns carry 11.6% of the token occurrences

The chapter’s six intuition checks: let students answer before revealing (one click).

A p ≫ n problem is a situation in which regularization and the design of validation become important.

Why does min_df remove so many columns? The Zipf section showed it: most of the vocabulary occurs in few documents, which can make the effect of those terms hard to learn reliably. Some rare terms are nevertheless useful for telling particular classes apart.

min_df=5 does not only drop terms seen once in total; it drops every term that occurs in fewer than five different documents. Such a term may be repeated many times within one document, which is why the share of columns removed and the share of token occurrences removed are reported separately. Whether the trade is worth it is an empirical question rather than a rule, and exercise 3 asks students to measure it.

Intuition check 2 of 6

Should IDF be computed on the training set or on all your data?

• In a supervised classification evaluation the vocabulary and the IDF are learned from the training portion only and applied unchanged to validation and test data
• IDF is a parameter estimated from data, so computing it over train + test leaks information about the test distribution into your features
• In scikit-learn: fit_transform on train and transform on test. Calling fit_transform twice is a real and common bug
• In production the IDF vector is a model artifact you version and ship, not something you recompute per batch

In cross-validation that learning is repeated on each training fold.

In search systems, computing IDF from the indexed document collection is a separate evaluation setting.

Why not recompute per batch: otherwise your features drift silently as traffic changes.

Intuition check 3 of 6

A word appears in every document. What is its TF-IDF weight?

• With scikit-learn’s smoothed formula the IDF factor is \(\ln\frac{1+n}{1+n} + 1 = 1\) when \(\mathrm{df} = n\); before normalization the TF-IDF value is the term’s TF value
• With the textbook unsmoothed formula \(\log(n/\mathrm{df})\) it would be exactly 0 and the term would drop out of the vector
• The IDF is 1 under smooth_idf=False too: scikit-learn still keeps the outer \(+1\) under that setting, and the result is not the textbook formula

L2 divides the whole row by one common norm; the step that creates the difference in weight relative to rarer terms is the multiplication by IDF.

smooth_idf=True is the default. With smooth_idf=False the formula is ln(n/df) + 1.

Intuition check 4 of 6

Why do we normalize document vectors but not feature columns?

• The two normalizations answer different questions: L2 row normalization brings non-zero document vectors to unit norm, so comparisons rest on the proportions of the weighted terms
• Subtracting non-zero column means and storing the result explicitly turns the existing zeros into non-zero values: this 2.8 MB matrix would need about 601.6 MB as a dense float64 array
• Not every column scaling behaves this way: StandardScaler(with_mean=False) can keep the sparse structure
• Sparsity is a constraint that determines which operations are practical

Row normalization: comparisons then rest on the proportions of the weighted terms rather than on the absolute size of the vector.

In the example about 99.7% of all cells of the matrix are zero; that share need not be the same in every column.

Scaling without centring is the case that keeps the sparse structure.

Intuition check 5 of 6

Your search engine returns nothing for the part number “P-101”. Why?

default    : ['101', '102', 'pump', 'tripped']
custom     : ['p-101', 'p-102', 'pump', 'tripped']

• scikit-learn’s default token_pattern requires two or more word characters and does not cross the hyphen: P-101 silently became 101
• Getting no result at all can also come from a mismatch between the query and index analysers, or from other filters
• The habit that prevents it: after fitting any vectorizer, print get_feature_names_out() and read it

Start by comparing the tokens produced by the query analyser and the document analyser. The input was "P-101 / P-102 pump tripped"; the default pattern is (?u)\b\w\w+\b and the custom one (?u)\b\w[\w-]*\w\b.

P-102 also became a bare number. The two are still different tokens, but the equipment prefix is gone: 101 is now the same token as the 101 in a page number, a quantity or a pressure reading. If the same transformation is applied on the query side and the document side, a match through 101 is still possible.

In technical domains tokenization errors of this kind lose important identifiers and can seriously affect model performance. They are also easy to miss: nothing raises an error, and accuracy just stays lower than it should.

Actually read the output.

Intuition check 6 of 6

When is a bag-of-words model the right choice in 2026?

A bag-of-words baseline is especially attractive when several of these hold together:

• you have very few labelled examples per class
• your vocabulary is domain-specific and exact matching matters: part numbers, gene names, legal citations
• you must explain every prediction to someone with authority over you
• you have no GPU or a hard latency budget
• your documents are long and topical rather than short and nuanced

A practical starting heuristic, not a rule.

In suitable classification and retrieval tasks, use it also as a strong classical baseline for measuring what more complex methods gain. When a team reports “our transformer gets 0.91” without reporting that TF-IDF gets 0.89, we simply do not know how much of a gain 0.91 is. The chapter on evaluating LLM systems returns to this.

Interview questions for discussion

Five of the chapter’s interview questions. The chapter groups them by what they test.

1. Explain TF-IDF to someone who has not seen it. Then explain why the IDF term uses a logarithm
2. When would raw term counts outperform TF-IDF?
3. You add bigrams and accuracy drops. What happened?
4. Why does removing stop words sometimes hurt a sentiment model?
5. Stemming versus lemmatization — and would you use either today?

The rule to state out loud: preprocessing must be chosen against the task, not applied because it is the default.

Suggested format: pairs, one question per pair, two minutes; then compare with the chapter’s answers.

On attribution. Not every company attribution on pages headed “NLP questions asked at Google and Amazon” rests on a verifiable source. The chapter’s question groups were prepared by drawing on the topics covered in two free sources — Chip Huyen’s Introduction to Machine Learning Interviews and the AIMLInterviews repository — and the examples and answers were adapted to the scope of this chapter.

Question 1. Give the two-factor decomposition first; that weight is not the same thing as importance for a task. Then the information-theoretic reading and range compression. Do not reach for “1 versus 2 documents matters more than 100 versus 200”; the log counts both doublings as equal.

Question 2. Raw counts can be enough when document frequency adds nothing for telling terms apart; whether they are actually better still has to be measured. Tests like chi-square or the log-likelihood ratio expect counts. And give both sides the same tuning budget before you credit the representation.

Question 3. One possible cause is that the added sparse n-grams increase overfitting; that cannot be concluded from the accuracy difference alone. In this chapter’s experiment the feature count went from 32,318 to 249,522 and accuracy from 0.831 to 0.819; in the paired test that difference did not clear the threshold corrected for four comparisons. Things to try: raise min_df, strengthen regularization, or select among bigrams by pointwise mutual information.

Question 4. If the list removes not, no, never, very, only or but, distinctions useful for sentiment can be lost; scikit-learn’s English list contains all six.

Question 5. With a pretrained transformer you keep the tokenization and normalization pipeline the model expects. For sparse retrieval or classification, measure the benefit on the task; they can help in morphologically rich languages where you have a good analyser (Oflazer 1994).

Interview question 1 of 2

Your document-term matrix is 10⁶ × 10⁵. How do you store and work with it?

• A million documents against a hundred-thousand-word vocabulary is \(10^{11}\) cells: stored as 8-byte float64 values, the data area alone is 800 GB
• Text matrices typically run 0.1–1% non-zero. Compressed sparse row (CSR) holds three lists: the non-zero values, the column each value belongs to, and where each row starts
• At 0.1% density that is \(10^8\) values, at 12 bytes per entry roughly 1.2 GB: a saving of roughly 670×
• Sparsity is not a storage trick, it is a constraint on which operations you are allowed to perform

The 12 bytes are a float64 value and a 32-bit column index; the row pointers and object overhead are extra. Under the same assumption, values and column indices take about 12 GB at 1% density. Where 64-bit indices are needed the cost is higher.

What separates a good answer is the last line. Operations that destroy sparsity: subtracting non-zero column means and storing the centred matrix explicitly; a textbook PCA mean-centres first as well. Operations that preserve it: row-wise L2 normalization; a sparse matrix–vector product can be computed without densifying the input, but the resulting vector can be dense. Operations whose input stays sparse and whose output need not: an all-pairs similarity matrix. On a corpus where every document contains one common word it is completely dense — n² floats, which at n = 50,000 is 20 GB. Compute it in blocks, or keep only each row’s nearest neighbours. For fewer dimensions: truncated SVD, which unlike PCA skips the centering step and runs directly on the sparse matrix; applied to a term-count or TF-IDF matrix it is called latent semantic analysis (LSA), the historical bridge to chapter 3.

Interview question 2 of 2

Your text classifier performs well offline and badly in production. Walk me through your debugging.

1. Train/serve skew: is the identical preprocessing pipeline running in both places — including the fitted TfidfVectorizer, its vocabulary and its IDF vector?
2. Vocabulary drift: new product names and new error codes map to no column at all, so the model never sees them
3. Distribution shift in the label mix
4. A leak in the offline evaluation, such as fitting the vectorizer before the train/test split
5. Only then, the model

The expected structure is a checklist, in cost order.

“Serving” is the stage after training, where the model answers real requests. Once fitted, the vectorizer is no longer just code: it carries two things learned from the training data, the vocabulary that maps each word to a column and the per-term IDF vector. Production has to use the same preprocessing rules, vocabulary and IDF values as training. Refitting the vectorizer on different data can change the column mapping or the weights; the problem then arises in the feature pipeline, before the classifier.

The fitted vectorizer is a component that has to be stored with the model: saved, versioned and shipped alongside the model weights. A good answer is expected to name that nuance.

Two more questions from the chapter for those who want them: design a system to detect near-duplicate documents in a corpus of 100 million (the reflex being tested: never enumerate the pairs), and build a search engine for internal technical documentation (start with BM25, check in the token output that the part numbers survive intact, then add dense retrieval and combine the two).

Where to go deeper

Your goal	Starting sources
I have thirty minutes	the 3Blue1Brown video, the scikit-learn text-features page
I want the TF-IDF and IR side	Manning et al.’s IR book (the idf and BM25 pages), the scikit-learn docs
I want the academic reference	Jurafsky & Martin, the original papers (Zipf, Heaps)

• The list in the chapter is long, and you do not need all of it: decide what you want first
• All the links are on the chapter page

The video is Large Language Models explained briefly, by 3Blue1Brown: seven minutes, a short, visual introduction to how text is turned into a numerical representation.

Introduction to Information Retrieval, by Manning, Raghavan and Schütze, is free in full; the single most useful page for this chapter is the one on inverse document frequency.

Speech and Language Processing, by Jurafsky and Martin, is the standard reference, free and continuously revised. Chapter numbers change between its releases, so check the table of contents of the draft a source refers to.

If you come from statistics and think in data frames rather than matrices, the TF-IDF chapter of Text Mining with R, by Silge and Robinson, may click faster. The scikit-learn text feature extraction page documents the exact smoothed TF-IDF variant used here, which differs from the textbook formula.

The sources on tokenizer tools are at the end of chapter 8.

What to remember

1. Tokenization is a design decision about which units the model is shown
2. Normalization rules can merge some distinctions of spelling or encoding; which to keep depends on the task and on the input the model in use expects
3. A fixed word-level vocabulary cannot cover every word form in new text; today’s language models get past that limit with units smaller than a word, learned from data (chapter 8)
4. Zipf describes the approximate empirical relation between word frequencies and their ranks, Heaps the one between the amount of text processed and the number of distinct terms observed
5. Bag-of-words represents a document by which terms occur and how often, and loses word order
6. Other things being equal, TF-IDF gives more weight to a term that is frequent in the document and found in few documents of the corpus
7. Cosine similarity compares the direction of vectors rather than document length; L2 normalization is what makes that geometry convenient
8. Bag-of-words is a fast, sparse, readable baseline that cannot directly represent order, the scope of negation, or semantic similarity

The chapter’s eight points. If students can say them in their own words, they have the core of the chapter.

On point 3: a form that is not in the vocabulary stays out of the counts. On point 4: how well the two laws fit depends on the corpus, the tokenization and the range measured.

Lab and exercises

• The chapter has six exercises, each labelled Basic, Applied or Advanced
• Exercise 1: fit a CountVectorizer on ten documents from your own domain, print get_feature_names_out(), and find at least two tokens that are wrong
• Exercise 2: compute the TF-IDF vector for d3 with pen and paper, then verify against TfidfVectorizer
• Exercise 3: reproduce the classification table, then add one row: TF-IDF with min_df=5

The chapter: Text as Data. The lab follows the chapter notebook, en/chapters/02-text-as-data/02-text-as-data.ipynb in the notebooks repository.

Basic consolidates a concept, Applied makes you measure something on your own data, and Advanced goes past the chapter.

Exercise 1 in full: a destroyed identifier, a merged pair, a lost unit. If the problem comes from token boundaries, fix it with token_pattern; where needed, also adjust lowercasing, preprocessing or a custom tokenizer. Report which example each change fixes and which new errors it could introduce.

Exercise 3 in full: compare the two configurations on a validation portion split off from the training data, by accuracy, feature count, memory and processing time, and use the final test set only after the choice is made.

The other three. Exercise 4: construct two documents from your own field with opposite operational meaning and cosine similarity above 0.9 under TF-IDF. Exercise 5: plot rank against frequency on a log-log scale for a corpus you own, and check whether any of the top five tokens is an artifact like 'ax. Exercise 6: work through the five-document TF-IDF ranking problem in §8.2.1 of Chip Huyen’s book with pen and paper, including its follow-up.

The chapter closes the exercises with a box of expected observations, not solutions.