4 A Representation Bake-Off
Learning objectives. After this chapter you can:
- build a pipeline from raw text to an evaluated prediction for a real, imbalanced dataset, using the methods of the previous two chapters (Chapter 2, Chapter 3);
- explain why accuracy alone is not enough on this imbalanced task, compute macro-F1 from its per-class arithmetic, and discuss when a different business goal calls for a different metric;
- form a starting expectation for count and embedding representations on a task by weighing the amount of labelled and unlabelled data, the structure of the task, the preprocessing and the tuning budget together — and then test that expectation with an experiment;
- measure the out-of-vocabulary rate on Turkish text, and evaluate separately which new word forms fastText can produce a vector for, how good those vectors are, and what they do to the task score;
- show with an example the word-order information that a unigram bag and an average of word vectors both lose, and explain which distinctions longer n-grams and order-sensitive models can bring back.
What we will look at. One dataset of Turkish reviews, with count and embedding representations compared under one shared evaluation protocol. A bake-off, in the engineering sense, is a controlled comparison of this kind: several candidate methods, one task, one protocol, and the same measuring stick for all of them. We first settle the metric that suits the class imbalance and the preprocessing, then look at each method’s score, coverage, cost and errors. Finally we show how the result changes as the amount of labelled data changes, and which word-order information both representation families are blind to.
4.1 Comparing two representation families on one task
Chapter 2 taught you to count words. Chapter 3 taught you to place them as coordinates. Each chapter ended with a claim. The two claims do not logically contradict each other — a sparse classifier can be strong while a word-level vocabulary generalises badly across morphology at the same time — but which one dominates on a particular dataset is something you only learn by measuring.
baseline — the reference result a new method is judged against, on the same data and the same metric. A plain majority-class rule can be a baseline, and so can a strong TF-IDF system. Reading a score also takes the class distribution and the cost of each kind of error. ↩︎
agglutinative — describes a language in which morphemes mostly attach to the word as separable affixes. In Turkish, inflectional and derivational suffixes can give one root or stem many surface forms; how many depends on usage and on the corpus. ↩︎
Chapter 2 showed that sparse count vectors can be a strong, cheap, interpretable baseline. Chapter 3 argued that giving every word form its own independent feature can leave a coverage problem on the inflected forms of an agglutinative language like Turkish. Its measurement was this: build the vocabulary on 90% of a Turkish Wikipedia sample and test on the remaining 10%, and 7.33% of test tokens were never seen in training — and 28.99% of distinct word types. Those rates belong to that sample and that definition of the vocabulary; they are not constants of every Turkish corpus.
Both claims can hold at once. This chapter compares the two representation families on one Turkish review task and one test set. All seven rows use logistic regression; but the vectorisers do not share the same preprocessing, vocabulary threshold, feature scale or extra training cost. So the results compare these concrete setups of the methods, and we do not read the gap between them as the effect of the representation family alone.
Chapter 3’s own comparison ran on 20 Newsgroups: English, balanced, four classes, 2,327 training documents. The Turkish product reviews here differ in data, task and training protocol all at once. Where the conclusions differ, we cannot say how much each factor contributes without comparisons that change them one at a time.
The setup is deliberately plain. Four things are the same in every row: the train/test split, the LogisticRegression class, the class weighting and the random seed. What changes from row to row is the vectoriser’s vocabulary threshold, the tokenisation path, the scale of the features, and the separate training of the dense representations on unlabelled text. So a difference in score may be decided by the representation together with these accompanying design choices.
Holding a hyperparameter fixed is not the same as being fair to every representation. Every row of the main comparison uses C = 1.0. C is logistic regression’s regularisation setting; in scikit-learn it is the inverse of the penalty strength, so a smaller C shrinks the coefficients harder. How hard a given C actually shrinks depends on the scale of the features. TfidfVectorizer L2-normalises its rows by default, so in the TF-IDF rows every document arrives with norm 1.0; the averaged word2vec and fastText vectors are not normalised and their norms vary. The same C need not mean the same regularisation on those two scales. In the main comparison C was not searched for any row (one later ablation tunes it for two sparse rows only). A fairer design gives each representation a comparable validation budget for its C and its normalisation option; it states which data the vocabulary, the IDF weights and the embeddings were fitted on, and it keeps the test labels out of the selection. This version does not do that, which is a real limitation and is flagged again where it matters.
The experimental design. Keeping the test set and the metric the same across the models being compared makes the difference easier to read. When the classifier, the tuning budget or the preprocessing changes along with the representation, the gap you see is their combined effect.
4.2 The problem
We set the comparison up as a binary sentiment task on Turkish product reviews, with imbalanced classes and inconsistent spelling.
Turkish product reviews, labelled positive or negative. The dataset published by Fatih Barmanbay holds 235,165 reviews and is distributed as a single parquet file (Barmanbay 2020). Its card says the reviews were collected online; it does not document which site they came from, when, or how the labels were produced. The texts are short and informal: misspellings, missing diacritics, run-together words. They are not raw either: all of the distributed text has already been lowercased (measured below).
We chose this dataset for three reasons:
- It is Turkish, so chapter 3’s morphology argument (Chapter 3) can be tested on real user text.
- Its classes are imbalanced — the positive class is 14.8 times larger — so the choice of evaluation metric has to be argued.
- It is large enough to subsample, so we can compare how the methods behave as the amount of labelled training data changes. That last comparison is a learning curve valid for this data and these settings.
import hashlib
import pandas as pd
from sklearn.model_selection import train_test_split
URL = ("https://huggingface.co/datasets/fthbrmnby/turkish_product_reviews/"
"resolve/main/data/train-00000-of-00001.parquet")
df = pd.read_parquet(URL).rename(
columns={"sentence": "text", "sentiment": "label"})
print(f"missing text or label {int(df.isna().any(axis=1).sum()):>7,}")
df["text"] = df["text"].astype(str).str.strip()
labels_per_text = df.groupby("text")["label"].nunique()
print(f"empty texts {int(df['text'].eq('').sum()):>7,}")
print(f"texts with both labels {int(labels_per_text.gt(1).sum()):>7,}")
df = df.drop_duplicates(subset="text")### checks before de-duplication
missing text or label 0
empty texts 21
texts with both labels 77### the raw data
rows as downloaded 235,165
after de-duplication 233,508 (1,657 exact duplicates removed)
label 1 (positive) 218,747 93.7%
label 0 (negative) 14,761 6.3%The first three lines are the checks made before de-duplication. Nothing is missing. 21 rows have an empty text; de-duplication collapses them into one row. 77 texts carry both labels (297 rows in all): the same short review is labelled positive in one place and negative in another. drop_duplicates keeps the first row in which each text appears; in this file every positive review comes before every negative one, so all 77 texts stayed with their positive label. This chapter left that choice as it was; 17 of the 77 texts fall inside the 50,000-review subsample below. On your own data, choose the policy for conflicting labels — keep the first, drop them all, or decide by hand — explicitly, and report it.
The class counts can be checked against the source. The dataset card reports 220,284 positive and 14,881 negative reviews. After de-duplicating on the text the counts are 218,747 and 14,761; the differences are 1,537 and 120, and \(1{,}537 + 120 = 1{,}657\) — the number of duplicates removed. That agreement shows the counts are consistent; it does not show which records de-duplication kept. The check above does.
Provenance, and what is not documented. The dataset was compiled by Fatih Barmanbay (Barmanbay 2020). The licence is stated differently in two places: the licensing section of the card and the licence file of the original GitHub repository say CC BY-SA 4.0, while the licence field at the top of the Hugging Face page says “unknown” (checked on 10 October 2026). CC BY-SA is share-alike, a stricter licence than this book’s own CC BY 4.0, so check the original repository’s licence and the card together before you redistribute the data or anything derived from it. Three things the card does not say: which site was scraped, when, and how the positive/negative label was derived — a star-rating threshold, manual annotation, something else. That last gap matters, because Chapter 5 is built entirely on the label quality of this corpus and has to work without knowing how the labels were made.
The same text sitting on both sides of the train/test split can, especially with short reviews, let a memorised answer score as a generalisation. That is why this experiment de-duplicates on the text before splitting; the 1,657 duplicates are only 0.7% of the file. The step is necessary but not sufficient on its own: templated or near-duplicate texts can survive it, and a text that carries two different labels needs its own decision. Report the de-duplication policy you chose together with the dataset.
We then take a stratified 50,000-review subsample of the 233,508 de-duplicated reviews, preserving the class ratio, and split it 80/20, again stratified:
sub, _ = train_test_split(df, train_size=50_000,
stratify=df["label"], random_state=42)
train, test = train_test_split(sub, test_size=0.2,
stratify=sub["label"], random_state=42)
def fingerprint(frame):
rows = ",".join(map(str, sorted(frame.index)))
return hashlib.sha256(rows.encode()).hexdigest()[:12]
for name, part in [("train", train), ("test", test)]:
pos = int(part["label"].sum())
print(f"{name:<6} {len(part):>7,} {pos:>7,} positive "
f"{len(part) - pos:>6,} negative rows {fingerprint(part)}")### the split (stratified, seed 42)
train 40,000 37,471 positive 2,529 negative rows 8205c2be5ebf
test 10,000 9,368 positive 632 negative rows 634cdb3bb002The rows column is a digest computed from the positions of the selected rows in the raw file. If your own run prints the same 12 characters, you selected the same rows. The class totals do not show that; many other splits give the same totals.
Two warnings about this split.
Reproducing it. The first version of this chapter recorded the split only as “stratified with seed 42”. That does not name a procedure; it names a family of procedures, and they disagree: it does not say which call did the sampling, in which order, or in how many stages. The two calls above were recovered later; the story is in Chapter 12. The numbers 50,000, 40,000 and 10,000, and the 101,659 features from the bigram vectoriser, are strong diagnostic signs — a plausible alternative construction yields 100,350 features instead, as you will see there — but they are not proof that you selected the same rows. The proof is a matching row digest, or the exact sampling code. Chapter 11, drawing from the same source with the same seed, ended up with a different 50,000 rows.
The test set and selection. Every number below is read off the same 10,000 test reviews: seven representations, nine training sizes, a hybrid, three diagnostics. Read these results as exploratory: you cannot pick a method or a crossover point from these rows and then report an unbiased final score on the same rows. In real work, split train again, select on that validation slice, and use the test set only for the final evaluation. Chapter 12 does precisely that, and says why it had to.
Subsampling to 50,000 is a convenience, not a necessity: it shortens the experiment. Keeping the same 10,000 test reviews and training on all of the other 223,508 reviews leaves the ranking unchanged, and every row gains (macro-F1, single run): bag of words 73.51 → 74.50; TF-IDF 71.95 → 73.00; bigram TF-IDF 76.21 → 76.69; character TF-IDF 70.39 → 71.12; averaged word2vec 61.39 → 62.89; averaged fastText 61.36 → 62.84; doc2vec 55.13 → 61.99.
4.3 The class error the first score hides
Before comparing the methods, we use the simplest possible rule — always pick the majority class — to show which errors accuracy can hide on an imbalanced test set.
Predict “positive” for every review. Never look at the text at all.
### the majority-class baseline
accuracy 93.68%A rule that only ever predicts “positive” reaches 93.68% accuracy without using the review text at all, because 93.68% of the test set is positive. That score does not say whether the rule can find the negative reviews. The confusion matrix does:
### majority baseline, in full
predicted neg predicted pos total
actual neg 0 632 632
actual pos 0 9,368 9,368
precision (neg) undefined — it never predicts negative
recall (neg) 0.0% — it finds none of the 632
F1 (neg) 0.00 — 2TP / (2TP + FP + FN) = 0 / 632
F1 (pos) 96.74
macro-F1 48.37Macro-F1 is the equally weighted average of the per-class F1 scores: each class gets the same weight regardless of its size. The majority rule scores F1 = 0 on the negative class, so its macro-F1 stays at 48.37. That makes visible what the 93.68% accuracy hides. The decision metric for a real system still has to be chosen separately, from the costs of its errors.
So on this dataset accuracy cannot be the decision metric on its own. What replaces it, and why does that replacement not have the same problem?
For one class, with \(TP\) true positives, \(FP\) false positives and \(FN\) false negatives:
\[ \text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}, \qquad F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} = \frac{2\,TP}{2\,TP + FP + FN} \]
\(F_1\) is the harmonic mean of precision and recall. When either of them drops to zero, so does \(F_1\): with precision 1.00 and recall 0.00 the arithmetic mean is 0.50 and \(F_1\) is 0.00. That property keeps the misses in one class from being hidden behind a single high precision.
Mind the zero denominators. If a class is never predicted (\(TP + FP = 0\)) its precision is undefined; \(F_1\) can still be computed from the second form, which is written in counts. \(F_1\) is undefined only when \(TP = FP = FN = 0\), that is, when the class has neither true nor predicted examples; in that case scikit-learn reads what to return from its zero_division argument.
Macro-F1 then averages across classes:
\[ \text{macro-}F_1 = \frac{1}{K}\sum_{k=1}^{K} F_1^{(k)} \]
For the majority baseline the negative class has \(TP = 0\), \(FP = 0\) and \(FN = 632\), so \(F_1^{(\text{neg})} = 0/632 = 0\), and
\[ F_1^{(\text{pos})} = \frac{2 \cdot 0.9368 \cdot 1.0}{0.9368 + 1.0} = 0.9674, \]
so macro-F1 \(= (0 + 0.9674)/2 = 0.4837\).
Which F1 your library reports. “F1” alone is ambiguous, and in scikit-learn the average argument decides it. The default for two classes, average="binary", reports the F1 of class 1 only — here the majority positive class, so the majority rule scores 96.74. average="micro" pools all the counts before dividing; for single-label classification with every class included, that equals accuracy (93.68). average="weighted" weights each class’s F1 by its number of examples, which gives 90.62 on this class distribution. average="macro", which weights the classes equally, gives 48.37. Always write the argument out explicitly.
Accuracy weights examples equally; macro-F1 weights classes equally. On a test set where only 6.32% of the reviews are negative, calling everything “positive” gives 93.68% accuracy and catches no complaint at all. A system that triages product reviews has to set the cost of a missed complaint and of a false alarm separately. Macro-F1 is used here as a comparison metric that weights the two classes equally; it is not the only possible product metric.
Every model below uses class_weight="balanced". It sets the class weights from the inverse of the class counts in the training set. With the 37,471 positive and 2,529 negative training examples here, the class coefficient of a negative example in the loss is about 14.8 times that of a positive one. That does not mean the gradients of individual examples carry the same ratio; a gradient also depends on the features and on the predicted probability. This chapter did not measure the unweighted setup; exercise 2 asks you to.
Do not conflate the two: choosing macro-F1 as the headline metric and reweighting the training loss by class frequency are separate design decisions. One sets what the evaluation cares about, the other what training cares about, and picking one does not automatically bring the other. Balanced class weights are a strong default, but compare unweighted, weighted and threshold-tuned variants on a validation set.
In this chapter macro-F1 is the main summary score for comparing the methods, and accuracy is reported alongside it for context. Watch how the two columns diverge in the tables. Macro-F1 is not a universal answer: in a real system that processes reviews, recall on the negative class, precision at a given false-alarm budget, or expected business cost can be more direct decision metrics.
4.4 Tokenising Turkish, and the trap inside that
Before the count and embedding models, we define the preprocessing decisions both of them will use. We look at the Turkish-specific lowercasing that chapter 2 mentioned (Chapter 2) on this data, and then show which characters the tokeniser keeps and which it throws away. The details of the conversion are in the Turkish NLP chapter (Chapter 16).
"IYI".lower() # → 'iyi'
"İYİ".lower() # → 'i̇yi̇' (an i with an extra combining dot)In Turkish, dotted i/İ and dotless ı/I are different letters that distinguish different words. Lowercasing correctly spelled Turkish text needs the mapping I → ı and İ → i; that is one of the language-specific tailorings the Unicode standard defines and the default does not apply. Python’s str.lower() implements Unicode’s default case mapping — the language-independent one — and does not change with the locale: it produces i for I, and i plus a combining dot (U+0307) for İ. In text typed on an ASCII keyboard, this mapping alone cannot tell which letter the writer meant; we come back to that limit below.
TR_LOWER = str.maketrans({"I": "ı", "İ": "i"})
def tr_lower(s):
return s.translate(TR_LOWER).lower()
tr_lower("IYI") # → 'ıyı'
tr_lower("İYİ") # → 'iyi'### the same string, two lowercasing rules
str.lower() tr_lower()
"IYI" iyi ıyı
"İYİ" i̇yi̇ iyiThe default operation cannot turn correctly spelled IYI and İYİ into their Turkish lowercase forms: it makes IYI into iyi, and it leaves a combining dot in İYİ. Both run without an error, and both can break tokenisation and vocabulary matching. In this experiment’s main score, however, no difference was measured between the two lowercasing routines, because the distributed data is already lowercase (shown below the round 1 table); combining-dot damage that already exists needs a separate repair step. Case conversion is language-dependent, and a language-specific lowercasing policy is part of the preprocessing and tokeniser contract. (The operation here is lowercasing; Unicode casefolding is a related but distinct operation intended for caseless matching, and it has its own Turkish tailoring.) One limit up front: tr_lower assumes correct spelling. In user text typed on an English keyboard, IYI may well mean “İYİ”, and tr_lower then produces ıyı, a new error. What ASCII-typed text needs is the restoration of Turkish characters (deasciification; see Section 16.2).
Case mappings that depend on the locale or on the context matter in other languages too: in Greek, a sigma at the end of a word lowercases to ς, not σ. German ß belongs to a different operation: "ß".lower() leaves it unchanged, and it is casefolding that equates ß with ss ("ß".casefold() → ss). That is why the “just lowercase it” step in an English tutorial is not portable advice; but do not treat these examples as the same problem as Turkish I/İ either.
After Turkish lowercasing, the simple token pattern we use extracts unbroken runs of a chosen set of lowercase letters:
import re
TOKEN_RE = re.compile(r"[a-zçğıöşü]+")
def tokenize(s):
return TOKEN_RE.findall(tr_lower(s))The pattern keeps only the listed letters: punctuation, digits and emoji all disappear, and circumflexed letters (â, î, û) and combining marks split a token. hâlâ or kâr, for example, does not survive as one token. In product reviews !!!, 5/5 or an emoji can carry sentiment; this experiment does not model them. The effect of these simplifications on the classification score was not measured; on your own corpus, compare alternative patterns.
With this tokeniser the training corpus looks like this:
### the training corpus, 40,000 reviews
tokens 903,190
distinct word forms 74,317
hapax (seen exactly once) 43,579 58.6% of all forms
mean review length 22.6 tokens
median 19
95th percentile 56
longest 296Of the 74,317 distinct forms, 43,579 occur exactly once; the other 30,738 are seen at least twice. After min_df=2 the vectoriser has 30,195 columns. We counted the 543 forms in between one by one: every one of them occurs two or more times in the corpus but only inside one document. min_df counts document frequency, not corpus frequency.
58.6% of the vocabulary appears exactly once. In chapter 3’s Turkish Wikipedia sample the rate was 50.0% (Chapter 3). The two corpora differ in topic, document length and preprocessing, so the gap cannot be put down to typos alone; the spelling variation of user-generated text is one possible factor.
The variety of forms becomes visible if you group word forms by their first five characters:
### distinct word forms sharing a five-character prefix
kulla 895 forms kullanışlı, kullanıyorum, kullandım, kullanımı, kullanım, ...
bekle 285 forms beklediğimden, beklediğim, beklemiyordum, beklemeyin, ...
çalış 247 forms çalışıyor, çalışma, çalışan, çalışması, çalışırken, ...
çıkar 209 forms çıkarıyor, çıkar, çıkartıyor, çıkarmıyor, çıkardı, ...
düşün 195 forms düşünüyorum, düşünmeden, düşünenlere, düşünen, ...bag-of-words — count which words a document contains and forget the order they came in. “Dog bites man” and “man bites dog” become the same document. ↩︎
The five-character prefix kulla appears in 895 distinct forms in this corpus. The group is not 895 inflections of one stem: it holds inflections, derivations and spelling variants together. In a bag-of-words model every form that enters the training vocabulary becomes its own column (with min_df=2 the forms seen in only one review never become columns at all — they are simply dropped). The coefficient learned for kullanışlı does not transfer by itself to the kullanıyorum column; other features in the same document, and regularisation, still affect how the model generalises.
A five-character prefix is a crude look at the variety of Turkish word forms; it is not stem or morpheme analysis. The kulla group puts kullanışlı (an adjective), kullanım (a noun) and kullanıcı (a noun) side by side, and the same prefix also collects typos. So this table does not measure how many columns a real stem family is spread over. Finding that out takes morphological analysis and a separate look at the spelling variants; the analysis is classically done with finite-state transducers (Oflazer 1994). For Turkish morphological analysers and how to use them, see Section 16.3 and Section F.2.
The 895-form prefix group is a sign of how far the number of word-level features can grow. Below we look separately at how this variety relates to coverage and to the classification score.
4.5 Round 1 — the chapter-2 methods
We have chosen the evaluation metric and defined Turkish lowercasing explicitly. The combining-dot errors already present in the data are still there; we come back to them below the table. Now we compare the four count representations of chapter 2 (Chapter 2) with the same classifier.
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
PAT = r"[a-zçğıöşü]+"
# lowercase=False + preprocessor=tr_lower: otherwise scikit-learn calls
# Python's str.lower() and commits the bug we warned about above
LOC = dict(preprocessor=tr_lower, lowercase=False)
schemes = {
"bag of words": CountVectorizer(min_df=2, token_pattern=PAT, **LOC),
"tf-idf": TfidfVectorizer(min_df=2, sublinear_tf=True, token_pattern=PAT, **LOC),
"tf-idf 1-2 gram": TfidfVectorizer(min_df=2, sublinear_tf=True,
ngram_range=(1, 2), token_pattern=PAT, **LOC),
# a separate feature-extraction path: token_pattern is not used, and min_df differs
"tf-idf char_wb 3-5": TfidfVectorizer(min_df=3, sublinear_tf=True,
analyzer="char_wb", ngram_range=(3, 5), **LOC),
}
clf = LogisticRegression(max_iter=2000, class_weight="balanced", random_state=42)### round 1: counting
accuracy macro-F1 F1(neg) dims fit
majority baseline 93.68 48.37 0.00 — —
bag of words 92.14 73.51 51.30 30,195 0.7 s
tf-idf 90.07 71.95 49.41 30,195 0.5 s
tf-idf 1-2 gram 92.57 76.21 56.47 101,659 1.0 s
tf-idf char_wb 3-5 88.63 70.39 47.14 94,939 3.3 sThe rows of the table share the same train/test split and the same classifier; but they differ from one another in three respects.
The character row does not go through the same tokeniser. In scikit-learn, token_pattern is used only when analyzer="word"; char_wb ignores it entirely and runs its own character stream. So the character row differs from the word rows not only in feature granularity but in the feature-extraction path itself:
### the same string through the two analyzers
input: "5/5 harika!! ürün çok i̇yi 👍"
analyzer="word", token_pattern=[a-zçğıöşü]+
['harika', 'ürün', 'çok', 'i', 'yi']
— 5/5, !!, the emoji are gone; the combining dot split "i̇yi" in two
analyzer="char_wb", ngram_range=(3, 5) (first few of the 3-grams)
[' 5/', '5/5', '/5 ', ' ha', 'har', 'ari', 'rik', 'ika', 'ka!', 'a!!', '!! ', ...]
— digits, the slash, the exclamation marks and the spaces all survive as featuresSo the score difference of the character row may reflect this preprocessing choice along with the character features: char_wb does not apply the word token pattern, and it treats digits, punctuation and spaces differently. Keep that in mind when you read the conclusions this chapter draws from character n-grams later on.
The character scheme uses min_df=3 where the three word schemes use min_df=2. That is a defensible choice — character n-gram counts explode — but it is a choice, it was not swept, and this chapter did not measure what the other setting would have done.
The fit column here and the build the representation column in the cost table near the end of the chapter do not time the same step. Re-measured five times each, vectorising the training text took 0.82–0.90 s for the bigram scheme and 2.68–2.79 s for the character scheme; fitting the classifier took 0.09–0.10 s and 0.38–0.39 s respectively. The gaps between the two tables (1.0 s against 0.9 s; 3.3 s against 2.9 s) are consistent with the classifier’s fitting time: fit covers vectorising plus fitting the classifier, the cost table covers vectorising only. The timings in the tables come from single runs, and the hybrid’s cost arithmetic later uses the second set; read all of them as approximate values for this machine.
The lowercase=False line above was added later, because the first version committed exactly the bug this chapter warns about two sections earlier. Both versions were then measured, and round 1’s table came out byte-identical.
The reason is the corpus itself: there is not a single uppercase character among the 37,076,865 characters of the 233,508 de-duplicated reviews (counted with str.isupper()). That observation is consistent with the distributed text having been lowercased beforehand; the dataset card does not document with which tool, or when. So the two paths coincide here.
That is not the same as the corpus being clean. U+0307 (COMBINING DOT ABOVE) sits in 4,194 documents of the 50,000-review working subsample — 8.4% of it, and every time it follows a letter i. Unicode’s default lowercasing produces exactly that sequence for İ. The earlier processing steps of the data are not documented, so we cannot settle where the mark came from; the observation is consistent with such a conversion. Our own [a-zçğıöşü]+ pattern does not treat a combining mark as a word character, so in those documents tavsiye splits into tavsi + ye. We measured the effect on the vocabulary: deleting the mark and re-vectorising the training text takes the 30,195-column vocabulary down to 29,823. 595 columns exist only because of this splitting (alişveri, alabi and the like); after the repair, 223 forms that had never appeared in one piece enter the vocabulary. Chapter 11 rediscovers the same damage in its own sample as a clustering result and measures it line by line. tr_lower does not delete a combining dot that is already in the text — the repair is to delete the mark from the text you were given, not to lowercase it differently. Exercise 7 asks you to measure what the repair does to the scores.
Why fix it anyway? Because the reader copying this code will not have that corpus. A small illustration of the difference:
'İYİ'.lower() -> 'i̇yi̇' -> under [a-zçğıöşü]+ ['i', 'yi'] split in two
'IYI'.lower() -> 'iyi' -> ['iyi'] wrongly mergedThe same warning applies to analyzer="char_wb". Character n-grams are sold as language-independent, but the lowercasing step in front of them is still locale-dependent.
Three things in that table are worth stopping for.
F1 score / macro-F1 — F1 is the harmonic mean of precision and recall for one class. Macro-F1 averages the F1 of every class with equal weight, so a rare class counts as much as a common one. In this experiment it is the metric that makes the majority rule’s F1 = 0 on the negative class visible. ↩︎
Accuracy and macro-F1 part ways over the majority rule. All four count methods have lower accuracy than the majority rule’s 93.68%, while scoring 22 to 28 macro-F1 points above it. (Among the four methods the two metrics give the same order: bag-of-words, for example, has both higher accuracy and higher macro-F1 than TF-IDF.) The models evaluated here give up some of the accuracy that ignoring the minority provides, and gain negative-class recall in return. If you chose a method by accuracy alone in this setup, the majority rule would look like a strong candidate. That is not a law: other thresholds and training settings can raise both accuracy and performance on the negative class; we did not test those options here.
In this setup raw counts are ahead of TF-IDF. The first explanation that comes to mind is IDF: reviews average 22.6 tokens, most words appear once in a document, so IDF may be down-weighting the frequent sentiment words — güzel, iyi, tavsiye. That is a hypothesis, and a testable one, because three things change at once between CountVectorizer and TfidfVectorizer: IDF weighting, sublinear term frequency (sublinear_tf=True, which replaces a count \(c\) by \(1 + \log c\)), and the L2 row normalisation TfidfVectorizer applies by default. Turn each on and off separately — same vocabulary, same classifier, same C = 1.0:
### one change at a time, unigram word features, min_df=2
idf log-TF norm accuracy macro-F1 F1(neg)
raw counts (= "bag of words") no no none 92.14 73.51 51.30
counts + L2 no no L2 88.07 69.57 45.85
log-TF + L2 no yes L2 88.37 70.00 46.53
counts + idf + L2 yes no L2 90.20 72.16 49.74
log-TF + idf, no norm yes yes none 92.67 71.83 47.61
log-TF + idf + L2 (= "tf-idf") yes yes L2 90.07 71.95 49.41The result does not support the first explanation. With C = 1.0 fixed, adding L2 row normalisation to raw counts takes macro-F1 from 73.51 down to 69.57 (3.94 points); IDF, added on top of it, brings the score up to 72.16. So the original gap does not come from IDF alone, and “TF-IDF fell behind here because of IDF” is not what the experiment shows. Nor does it mean normalisation is harmful in itself: its effect has to be judged together with C.
A likely explanation is that we did not tune C. L2-normalised rows have norm 1 while raw count rows grow with document length, so a fixed C = 1.0 applies a different effective regularisation to the two representations — the same caveat this chapter flagged up front for the dense family, now inside the sparse one. Give both the same small tuning budget: split the 40,000 training documents into 32,000 for fitting and 8,000 for validation, choose C on the validation slice, then refit with the chosen C on all 40,000 documents and measure on the test set:
### C chosen on a validation split carved out of train (32,000 fit / 8,000 val)
best C on validation test macro-F1
raw counts 1.0 73.51
tf-idf 10.0 74.71The selection step is spelled out because it changes the result: the test scores in the table belong to models refitted with the chosen C on all 40,000 documents (the 32,000-document models score 73.65 and 73.83 on the same test set). This test set was already used earlier in the chapter, so these rows are not an independent test result opened for the first time.
The ordering flips: with C chosen on validation, TF-IDF scores 74.71 macro-F1 and raw counts 73.51. We measured how solid that 1.2-point gap is on this test set by pairing the two models’ predictions on the same 10,000 test reviews. Over 10,000 paired bootstrap resamples the 95% interval of the macro-F1 difference runs from 0.42 to 1.99 points; there are 151 reviews only TF-IDF gets right and 80 only raw counts get right (exact McNemar test on the error rates, \(p < 0.001\); round 2 explains how the two tests differ). So the gap is larger than the sampling noise of this test set. The interval belongs to one training split and one test set; it does not say what would happen on another training sample. What the experiment shows is this: the “raw counts ahead of TF-IDF” result in the round 1 table did not survive an equal tuning budget; it is not a fixed property of the representations, it depended on a shared, untuned C. The table is not wrong — that is genuinely what was measured — but the lesson to take from it is not “TF-IDF falls behind on short text”: it is that a representation comparison run without an equal tuning budget may not be measuring what you wanted it to measure.
bigram — a pair of words counted as a single unit, so that not good can be told apart from good. ↩︎
In this setup word unigram + bigram TF-IDF scores 4.26 macro-F1 points above unigram-only TF-IDF, and 2.70 above raw counts. (Not the largest gaps in the chapter — the two families are 14.8 points apart, and embeddings lead counts by 14.6 points at 250 labels. This is the largest difference measured within the count family once the label budget is fixed.) Bigrams keep local sequences such as iyi değil (“not good”) and güzel değil (“not nice”) as separate features, which can help capture negation built with a separate word. The same bigrams also pick up multi-word sentiment phrases, product names and collocations. Part of Turkish negation, though, is a suffix: beğendim (“I liked it”) and beğenmedim (“I did not like it”) are separate features at the unigram level too. In pairs like tavsiye etmem (“I do not recommend”) or memnun kalmadım (“I was not satisfied”) the negation is again a suffix: etmem and kalmadım are strong negative unigrams in the table below, and the bigram adds the tavsiye or memnun context to them. This chapter did not measure how much of the 4.26 points comes from which kind of negation; saying that takes a targeted error analysis or an ablation.
What the model learned, in words
In this sparse linear model every vocabulary feature has one coefficient. Sorting the coefficients together with their feature names helps you inspect which features the model has associated with each class on this data and with this preprocessing; it does not give the effect of a word in every context.
### tf-idf + logistic regression, largest coefficients
towards POSITIVE towards NEGATIVE
gayet +8.04 (quite) etmiyorum -8.16 (I do not [recommend])
ederim +6.94 ([I] do) değil -6.95 (not)
mükemmel +6.13 (perfect) kötü -6.63 (bad)
teşekkürler +6.09 (thanks) etmem -6.35 (I would not)
harika +5.93 (wonderful) beğenmedim -6.21 (I did not like it)
memnunum +5.84 (I am pleased) iade -5.85 (return / refund)
güzel +5.48 (nice) fakat -5.52 (however)
süper +4.98 (super) kalitesiz -5.47 (low quality)
kaliteli +4.32 (good quality) kalmadım -5.33 (I was not [satisfied])Two features in the list look as if they are about the domain rather than about language: iade (“refund”) and fakat (“however”) are among the features this model associates with the negative class. iade may refer to a return that happened; it may also only mention the possibility of one. fakat may be joining a positive judgement to a negative one. The coefficients show a conditional association with the training labels; which reading holds has to be checked by reading example reviews. Nobody wrote these associations down by hand; they were learned from 40,000 labelled reviews.
Do not read the list as more than it is. The coefficients are not universal sentiment scores for the words; they are global associations learned on this training distribution, together with every other feature. On a single document, the contribution of feature \(j\) to the linear decision score is \(x_j \beta_j\); the intercept and the contributions of the other features are added to the total score, and the probability is computed by applying the sigmoid to that total. The global coefficient list tells you what the model learned; it is not a complete explanation of one particular prediction. Those are separate questions.
Keep this column in mind when the dense methods arrive, because they do not give you the same direct word-to-coefficient table for free.
4.6 Round 2 — the chapter-3 methods
Now the same task with the three models of chapter 3 (Chapter 3). The word2vec, fastText and doc2vec models were trained from scratch on the text of the 40,000 training reviews, without using the labels; no pretrained vectors and no test reviews went into that training. Later, in the learning curve, the classifier sees fewer labels while this fixed representation, learned from the text of all 40,000 reviews, stays in use.
from gensim.models import Word2Vec, FastText
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
w2v = Word2Vec(train_tok, vector_size=100, window=5, min_count=2,
sg=1, negative=5, epochs=10, workers=1, seed=42)
ft = FastText(train_tok, vector_size=100, window=5, min_count=2,
sg=1, negative=5, epochs=10, workers=1, seed=42,
min_n=3, max_n=6, bucket=500_000)
d2v = Doc2Vec([TaggedDocument(d, [i]) for i, d in enumerate(train_tok)],
vector_size=100, min_count=2, epochs=20, dm=0,
negative=5, workers=1, seed=42)The code above shows only the training settings of the three models; the steps from there to a document matrix are these. For word2vec, a document’s vector is the mean of the vectors of its tokens that are in the vocabulary; a document with no token in the vocabulary gets the zero vector. For fastText every token of the document goes into the mean, because the model builds a vector for out-of-vocabulary forms too, from subword vectors. doc2vec learns the vectors of the training documents directly; the vectors of the test documents are inferred with infer_vector at its default settings. All three matrices go to the same logistic regression as in round 1.
### round 2: coordinates
accuracy macro-F1 F1(neg) dims embed train
averaged word2vec 79.92 61.39 34.64 100 14.7 s
averaged fastText 79.64 61.36 34.79 100 31.6 s
doc2vec (PV-DBOW) 71.96 55.13 27.66 100 8.5 sUnlike the sparse rows, these rows cannot be reproduced to the last digit. Re-running the same code with the same seed (42) while this chapter was being revised gave 61.37 macro-F1 for word2vec, 61.29 for fastText and 55.44 for doc2vec. Later runs of the chapter’s notebook gave the same 61.37 and 61.29 every time, while doc2vec moved again (55.57, 55.58, 55.54). The reason is in gensim: infer_vector starts each test document from a vector seeded with Python’s string hash, and that hash changes from process to process unless PYTHONHASHSEED is fixed. Changing only the embedding seed (five seeds) kept word2vec between 61.57 and 62.06, and fastText between 60.86 and 61.68. Read the decimal differences in the dense rows together with that spread.
Set against round 1:
### both rounds, sorted by macro-F1
tf-idf 1-2 gram 92.57 76.21 56.47 101,659
bag of words 92.14 73.51 51.30 30,195
tf-idf 90.07 71.95 49.41 30,195
tf-idf char_wb 3-5 88.63 70.39 47.14 94,939
averaged word2vec 79.92 61.39 34.64 100
averaged fastText 79.64 61.36 34.79 100
doc2vec (PV-DBOW) 71.96 55.13 27.66 100
majority baseline 93.68 48.37 0.00 —Do not read that ordering as a general ranking. The row names are short; in full, the three dense rows are averaged static vectors trained from scratch on this corpus’s 0.9 million tokens, not pretrained ones. The scale of the dense vectors was not equalised and C was tuned for no row at all — recall what the C experiment in the previous section did on the sparse side.
Every row comes from one split and one run. To measure the sampling uncertainty of the test set, we paired the two models’ predictions on the same 10,000 reviews and resampled the test rows 10,000 times (a paired bootstrap; both models’ macro-F1 is recomputed on each resample):
### paired bootstrap on the 10,000 test reviews: macro-F1 difference, 95% interval
tf-idf 1-2 gram - bag of words +2.69 [ +1.60, +3.82]
bag of words - tf-idf +1.56 [ +0.53, +2.55]
tf-idf - tf-idf char_wb 3-5 +1.57 [ +0.59, +2.54]
tf-idf char_wb 3-5 - averaged word2vec +9.01 [ +7.97, +10.01]
averaged word2vec - averaged fastText +0.08 [ -0.60, +0.77]
averaged fastText - doc2vec (PV-DBOW) +5.85 [ +5.02, +6.69]
tf-idf 1-2 gram - averaged word2vec +14.83 [+13.65, +16.01]The dense rows are taken from the re-run mentioned above. The gap between the families, and the adjacent gaps inside the sparse family, are larger than the sampling noise of this test set; the gap between word2vec and fastText is not. These intervals cover test sampling only: they do not say what happens when the training split changes, and for the dense rows the spread over training seeds comes on top. The wobble of the learning curve between 250 and 40,000 labels is not an uncertainty interval of this kind; those points differ both in training size and in which documents were sampled.
Error rates need a separate test. McNemar’s test compares the two methods’ error rates, so it is about the accuracy column, not the macro-F1 one the ranking is sorted on. Later in the chapter, McNemar’s test on the right/wrong outcomes of plain word TF-IDF and averaged fastText on the same test documents gives \(\chi^2 = 684.2\); that supports the difference in error rate between those two setups. A macro-F1 difference is not a paired count of errors; it needs a paired bootstrap, as above, or a randomisation test. Dror et al. (2018) lays the protocol out in detail, and exercise 8 asks you to reproduce these intervals in your own run and extend them over the training split.
recall — of the reviews that really were negative, the share the model found. Low recall goes with a lot of misses. ↩︎
precision — of the reviews the model called negative, the share that really were. Low precision goes with a lot of false alarms. ↩︎
In this setup the best sparse representation is well ahead of the from-scratch averaged embedding setup. The gap between the best sparse method and the best dense one is 14.8 macro-F1 points; on the negative class it is 21.83 points of F1 — and the shape of that gap matters. (A coincidence to disarm before it confuses you: a precision of 21.8% turns up a few lines below. The two numbers are unrelated — this one is a difference in F1 points, that one is a precision.) Averaged word2vec actually finds more negatives (84.2% recall against bigram TF-IDF’s 76.3%); but only 21.8% of the reviews it flags as negative really are negative (precision), where the figure for bigram TF-IDF is 44.8%. These are aggregate rates over the test set; do not read them as the probability that one new prediction is right.
On chapter 3’s 20 Newsgroups task (Chapter 3) the sparse method in the comparison was also ahead of the averaged static embedding. Seeing a similar ordering on two different datasets is a useful observation; but the language, the class structure, the amount of training data and the preprocessing all differ, so no general rule about document classification follows from these two results. The rest of the chapter looks at how the ordering is affected, in this Turkish review setup, when the number of labels and the kinds of error change.
State the limit plainly too. All three round-2 models were trained from scratch on the 903,190 tokens of the training reviews of this corpus only; work such as Bojanowski et al. (2017) uses corpora at Wikipedia scale. So the measurement above is precisely “static embeddings trained from scratch on 0.9M tokens fell behind, in this setup, counts extracted from the same 40,000 documents” — not “static embeddings fall behind counts”. Vectors pretrained on large Turkish corpora may give a different result; this chapter did not measure the direction or the size of the difference. Exercise 13 is where you measure it yourself.
What does not depend on corpus size is the averaging defect. No amount of training data makes kalite iyi fiyat kötü and kalite kötü fiyat iyi stop having cosine 1.0000, because that failure is in the pooling step, not in the vectors. That is the transferable result of this chapter; the ranking above is the local one.
Where the dense methods clearly differ is the number of dimensions: 61.39 macro-F1 from 100 numbers per document, against 76.21 from 101,659 columns.
Do not read the number of dimensions as memory. In this experiment the dense document matrix has 100 columns and the word TF-IDF matrix 30,195; even so, the measured memory of the training matrices is 16.0 MB and 9.7 MB respectively. A sparse matrix stores only the non-zero values and their indices: a row with 19.8 non-zeros on average holds 19.8 values, not 30,195. Fewer coordinates can be an advantage in some dense-vector structures — a nearest-neighbour index, approximate-nearest-neighbour (ANN) search, the first layer of a network; the actual memory and latency of search have to be measured with the chosen index, the number of documents and the sparsity.
The ratio depends on which sparse model you compare against. A 100-dimensional dense document vector has about 1,017 times fewer coordinates than bigram TF-IDF’s 101,659 columns, and about 302 times fewer than plain word TF-IDF’s 30,195 (302 is the figure in the summary table later in this chapter). These ratios are not, by themselves, ratios of memory, latency or product architecture. In your target use, measure the costs of sparse retrieval, dense retrieval and classification separately.
Note the direction of the dense models’ errors: in this run averaged word2vec finds 84.2% of the negative reviews — more than any sparse model — but the precision of its negative predictions is 21.8%. That pattern of false alarms may have as much to do with class_weight="balanced", the decision threshold and the choice of C as with how well the representation separates the classes. Telling the causes apart takes threshold and class-weight ablations.
4.7 The out-of-vocabulary test
We test chapter 3’s subword claim (Chapter 3) in two stages: can fastText produce a vector for a form that is not in the learned word vocabulary, and is the vector it produces useful on this classification task? In word2vec, looking up an out-of-vocabulary form directly raises an error; the document average simply skips such tokens. Here is the measurement on the test set.
### coverage of the 10,000 test reviews
test tokens 229,055
outside the word2vec vocabulary 16,660 7.27% of tokens
distinct test word forms 32,573
outside the vocabulary 15,098 46.4% of forms
test reviews with no usable token 7The first rate is computed over tokens, the second over distinct forms. The seven reviews in the last line have no token in the vocabulary; in the word2vec average they get the zero vector. All seven are labelled positive, and the classifier calls the zero vector negative (intercept −0.83), so all seven are misclassified.
Chapter 3’s 7.33% token rate was about forms never seen in training. The 7.27% here is the rate of test tokens not in the vocabulary of a word2vec trained with min_count=2; the two definitions differ. Counted with the same definition, the table splits like this: 5.10% of test tokens (11,681 tokens) never occur in training; 33.5% of distinct test forms (10,921 forms) were never seen, and another 4,177 forms were seen exactly once in training and cut by min_count=2. So the like-for-like counterparts of the 7.33% and 28.99% in chapter 3’s Wikipedia sample are 5.10% and 33.5% here. The corpus, the tokenisation and the sample size all differ, so the numbers can be read side by side but do not confirm each other.
46.4% of the distinct word forms in the test set are not in the vocabulary word2vec learned. Accessing those forms directly with w2v.wv[word] raises a KeyError; the document average skips them, so the consequence shows up not as an error but as lost information and the seven empty documents above. The rate varies from corpus to corpus and domain to domain.
For every test form this experiment’s tokeniser produces, gensim’s fastText can build a vector: the vector of a form that is not in the vocabulary is assembled from the vectors in the buckets its character n-grams fall into. That takes vocabulary-lookup coverage to 100%; it does not guarantee the quality of the vector: an n-gram never seen in training falls either into a bucket that was never updated (still at its random initialisation) or into a hash bucket shared with other n-grams (bucket=500_000). We measured it: on average 84.5% of the 3-to-6-character n-grams of the 15,098 out-of-vocabulary test forms also occur in at least one vocabulary word; for 24.7% of the forms every n-gram is familiar, and for 4.8% fewer than half are. Here are the neighbours of the vectors for three forms that never occur in training:
### nearest neighbours for unseen word forms
kargolanmamış ("not having been shipped")
word2vec KeyError
fastText kargolanmış, kargolandı, kargolanma, kargolanıp, kargolu
tavsiyelerim ("my recommendations")
word2vec KeyError
fastText tavsiyeleri, tavsiyeler, tavsiyeederim, tavsiyedir
hızlıcana (colloquial "nice and fast")
word2vec KeyError
fastText hızlıca, hızlıda, hızlısı, hızlıydı, hızlıOn these three out-of-vocabulary queries fastText returns a vector and brings back neighbours related in spelling (Bojanowski et al. 2017). That the first neighbour of kargolanmamış (“not yet shipped”) is kargolanmış (“shipped”), which carries no negative suffix, is an example of how similarity of form can overshadow the negation. It does not follow from three examples that every unseen form lands in a meaningful stem family.
Now compare the two models on words that are in both vocabularies:
### nearest neighbours for a common word
kargo ("shipping")
word2vec teslimat, gönderim, gönderi, kargoda, kargoyla, kargolama
(delivery, dispatch, shipment — different words, same topic)
fastText kargocu, kargolu, kargosu, kargonuz, kargom, kargomun
(courier, with-shipping, its-shipping, your-shipping, my-shipping)
kalitesiz ("low quality")
word2vec dandik, kullanışsız, dayanıksız, uyduruk, kırılacak
(shoddy, unusable, flimsy, fake, about-to-break)
fastText kalitesizmiş, kalitesizliği, kalitel, kalitesi, kalitesindeki
(…all forms of the same stem, including "kalitesi" = quality)On these two queries word2vec’s neighbours are related words from other stems, and fastText’s are forms close in spelling. Asked what is like kalitesiz (“low quality”), word2vec answers with five other ways to say a product is bad. fastText’s list contains kalitesi (“its quality”) — which is not a synonym: it is a noun form without the negation that -siz carries, and the first eight letters of kalitesiz are kalitesi itself.
Two words are an illustration, not a measurement, so we measured the spelling side on a larger sample. From the 2,806 vocabulary words with at least six letters and at least 20 occurrences in training we drew 500 at random and looked at each model’s top-5 neighbours. The share of neighbours that share the query’s first five characters is 6.5% for word2vec and 60.7% for fastText (8.6% and 67.4% for the first four characters). So fastText returning neighbours close in spelling is a general tendency of this model. The measurement counts orthographic overlap only; it does not say how appropriate the neighbours are in meaning. That takes human judgement or a task-based measurement.
This is chapter 3’s stated weakness (Chapter 3) — assumes spelling similarity implies meaning similarity — becoming visible. That the subword parameters are spent on modelling form, at the expense of information about meaning, is a plausible explanation; the measurements in this chapter do not prove it.
In this run fastText’s macro-F1 is 61.36 and word2vec’s 61.39: a difference of −0.03 points. Coverage went to 100%; macro-F1 did not change in any distinguishable way: the paired bootstrap interval from round 2 (−0.60 to +0.77) contains zero, and the training seed alone moves each model by about 0.5–0.8 points. The two models differ in more than coverage, so this result cannot be put down to out-of-vocabulary coverage alone.
fastText could build a vector for every one of these test forms; in this run macro-F1 did not change in any distinguishable way. Producing a vector does not guarantee that the new form is represented with the right meaning. Measure intrinsic properties, coverage and task performance separately; when the score does not move, look at representation quality, pooling and the classifier settings together.
intrinsic / extrinsic evaluation — intrinsic measurements look at properties of the representation in its own space, such as similarity or analogy; vocabulary coverage is a separate measure of usability. Extrinsic (task-based) evaluation measures success on the target job, such as classification. The two answer different questions; the choice of model is decided by the target task and the conditions of use. ↩︎
One more diagnostic: being in the vocabulary does not by itself show that a representation is reliable. Here are the two models’ neighbours for kalitesizmiş, which occurs only twice in training:
### kalitesizmiş — in the word2vec vocabulary, seen twice in training
word2vec ayakkabıların, kaymak, yapışır, temizlenmemiş, zorlamadım
(of the shoes, cream, it sticks, uncleaned, I did not force)
fastText kalitesiz, kalitesizliği, kalitesinide, kalitesiIn this example word2vec’s neighbours reflect neither the sentiment nor the form of the word; fastText brings back forms close in spelling. Past the vocabulary boundary a direct lookup raises a clear error; just inside it, a rarely seen form can return a poorly estimated vector without any error. So the 46.4% out-of-vocabulary rate may not show the whole of the representation problem. A general conclusion about the quality of rare forms takes a wider evaluation by frequency band.
4.8 Error examples for understanding the score gap
The table shows that the count methods score higher in this setup; it does not show why. Measuring how much each cause contributes takes separate experiments. Here we look at the predictions the two methods disagree on, at negation examples, and at the limits of the representations.
Take two representations: plain word TF-IDF (the 90.07-accuracy row) and averaged fastText (79.64). They give the same label on 84.1% of the test documents. On the 1,587 documents where they differ, TF-IDF is right 1,315 times and fastText 272. (With two classes, whenever the two disagree exactly one of them is right, so 1,315 − 272 = 1,043 documents is exactly their accuracy gap: 10.43 points.)
Those two counts are enough for a significance test. The intuition: if the two methods were equally accurate, their disagreements would split roughly half and half, and McNemar’s test on paired predictions (McNemar 1947) asks how far from half-and-half the split is. It uses exactly these off-diagonal counts: \(\chi^2 = (|1315 - 272| - 1)^2 / (1315 + 272) = 684.2\) on one degree of freedom, far past any conventional threshold. That is strong evidence that the two setups make errors at different rates on the same test set, which is a statement about accuracy. The uncertainty of the macro-F1 gap does not come out of this test; we measured it with a paired bootstrap over the test rows: the gap is 10.66 points, with a 95% interval from 9.52 to 11.77 (the fastText side comes from the re-run mentioned in round 2). Both results hold only for the two setups examined and this test data.
The interesting question is what the 272 look like.
### averaged fastText right, tf-idf wrong (gold: positive)
"en guvenli alısveriş sitesi" diacritics partly dropped
"çok hizli ve temi̇z tesli̇mat" broken diacritics mid-word
"çok sessiz çalışıyor makina ." "makina" — colloquial for "makine"
### tf-idf right, averaged fastText wrong
"sıkıcı edebi yonu cok zayif" gold negative — fastText says positive
"koku ne çok kötü ne çok iyi" gold negative — "neither… nor…"
"cok guzel! kedi bayildi buna" gold positive — fastText says negativeThese six selected examples show the kinds of text on which the two models can part ways: missing diacritics, colloquial spelling, and scope relations such as ne … ne … (“neither … nor …”). Which feature decided each prediction does not follow from the list; that takes looking at decision contributions, making controlled changes to the text, or labelling and analysing all of the disagreements. Texts with missing diacritics appear in both directions, so no single explanation is enough.
We counted two properties over all the disagreements (with the fastText model of the re-run, where the counts are 1,323 and 272). The share of tokens outside the TF-IDF vocabulary is 9.49% in the reviews only fastText gets right and 9.48% in the reviews only TF-IDF gets right; in the reviews both get right it is 6.51%. So out-of-vocabulary forms are more frequent among the disagreements, but they do not separate which model turns out right. What does differ is the class: 31% of the reviews only fastText gets right are negative (6.3% in the whole test set), while 98% of those only TF-IDF gets right are positive. That is consistent with the averaged-embedding setup working at higher recall and lower precision.
Negation examples let us test the representations directly as well.
### cosine between a review and its negation
averaged tf-idf
fastText
ürünü çok beğendim / ürünü hiç beğenmedim 0.776 0.193
kesinlikle tavsiye ederim / … tavsiye etmem 0.919 0.474
kargo çok hızlı geldi / kargo çok yavaş geldi 0.943 0.572
fiyatına göre gayet kaliteli / … hiç kaliteli değil 0.892 0.690
bu üründen memnun kaldım / … memnun kalmadım 0.962 0.582On these five opposed review pairs the cosine of averaged fastText runs from 0.78 to 0.96, and that of word TF-IDF from 0.19 to 0.69. (Averaged word2vec gives 0.72–0.96 on the same pairs; the high values are not specific to fastText’s subwords.) Averaging can reduce the effect of the one word that changed, through the contribution of the words the two sentences share. But the two representations use different weights and different vectors, so the whole gap in the table cannot be put down to the averaging operation alone. Nor does a high cosine mean the classifier will give the two reviews the same label; we come back to that below.
That explanation was qualitative. It is worth doing properly, because the result is a formula. It does not hand you a fixed length threshold — that depends on the norms and directions of the vectors — but it does show how the dilution scales as the shared content grows.
Take a five-word review and its negation, differing in exactly one word:
\[ \mathbf{d}_+ = \frac{1}{5}\left(\mathbf{v}_1 + \mathbf{v}_2 + \mathbf{v}_3 + \mathbf{v}_4 + \mathbf{v}_{\text{liked}}\right), \qquad \mathbf{d}_- = \frac{1}{5}\left(\mathbf{v}_1 + \mathbf{v}_2 + \mathbf{v}_3 + \mathbf{v}_4 + \mathbf{v}_{\text{disliked}}\right) \]
The difference is
\[ \mathbf{d}_+ - \mathbf{d}_- = \frac{1}{5}\left(\mathbf{v}_{\text{liked}} - \mathbf{v}_{\text{disliked}}\right), \]
Be careful reading that \(1/5\): it scales the shared words too, and cosine is invariant to a common factor, so the factor itself is not the problem. What matters is the ratio — one differing vector against four shared ones. Lengthen the review to twenty words and there is still one differing vector, but now nineteen shared ones instead of four, and the difference vector is scaled by \(1/20\). That the cosine of the two means approaches 1 is not guaranteed, though: it depends on the sum of the shared word vectors dominating the words that differ. If the shared contributions cancel or point in opposing directions, the result need not follow however long the review gets. The five examples above show that a high cosine can happen.
A subword model adds one more factor: beğendim and beğenmedim share character n-grams such as beğ, eğe, ğen, which gives their vectors common contributions. The two words also have separate whole-word components and n-grams that differ, so shared spelling alone does not settle the distance between them. Measured on this chapter’s models, the cosine between beğendim and beğenmedim is 0.74 in fastText and 0.53 in word2vec; for kaldım and kalmadım it is 0.69 and 0.56. For ederim / etmem, which do not share a stem, the two models are in the same place: 0.56 and 0.57.
In word unigram TF-IDF, beğendim and beğenmedim are two distinct columns; a document containing one has a zero in the other’s column. That helps keep the two forms from being confused. TF-IDF is not immune to dilution either: with L2 normalisation, two \(n\)-term documents differing in exactly one term (each term used once, all weights equal) sit at cosine \((n-1)/n\), which also goes to 1 as \(n\) grows. The 0.19–0.69 values on the real reviews, though, depend on the IDF weights of the terms and on the token sequences; those numbers cannot be put down to the separate columns alone.
The analogy. Averaging word vectors is taking the mean colour of a painting. Two paintings that differ only in that one is a sunrise and the other a sunset have nearly the same mean colour, and the mean colour is the only thing your classifier gets to see. A bag of words is instead a list of which pigments appear — rougher about relationships, but it keeps the pigment that changed as a separate entry.
A high cosine does not mean there is no separation along the decision direction. The classifier on averaged fastText assigns “ürünü çok beğendim” probability 0.82 of being positive and its negation 0.02 — at cosine 0.776. In the re-run, the probability of the negated version stays below 0.5 for all five pairs (between 0.02 and 0.33). Logistic regression can find a direction in which the small difference between the two averages is discriminative. Averaging can weaken some signals; these examples do not measure its share of the total 14.82-point gap.
4.9 The crossover: when embeddings do win
Everything so far has compared methods with 40,000 labelled reviews. In practice there are often far fewer labels, so we now look at how the ordering changes as the number of labels changes.
In this learning curve both representation families use the text of the 40,000 training reviews: TF-IDF learns its vocabulary and IDF values from it, and the embedding models learn their co-occurrence and subword parameters from it, all without looking at the labels. What varies is the number of labels given to the classifier: from 100 to 40,000. So the question is: with access to unlabelled text held fixed, how do additional labels change the decision boundary? What TF-IDF takes from the unlabelled text is limited to document frequencies; the embeddings also take co-occurrence relations between words.
For each n, a stratified subsample of the training set is drawn (train_test_split(train_size=n, stratify=…, random_state=42)) and the classifier is trained with the labels of those rows only.
### macro-F1 by labelled training size (same test set of 10,000)
n negatives tf-idf tf-idf char word2vec fastText
100 6 48.37 48.37 53.51 53.87
250 16 48.53 48.84 61.59 63.10
500 32 49.28 50.60 61.02 61.16
1,000 63 55.40 58.55 59.52 59.93
2,500 158 66.61 68.36 60.91 61.19
5,000 316 71.16 70.47 61.52 60.80
10,000 632 72.55 71.29 61.80 61.56
20,000 1,264 72.42 71.26 60.94 60.96
40,000 2,529 71.95 70.39 61.39 61.36Every row of the table comes from a single subsample. The word2vec column moves between 59.52 and 61.80 from 250 to 40,000 labels. The table does not say how much of that wobble comes from the number of labels, from which documents were selected, or from the training of the vectors; nor can it be used as a significance threshold for pairs of models. The dense columns are also sensitive, at small n, to the training of the vectors: with vectors retrained by the same code (the re-run in round 2), word2vec gives 58.23 at 100 labels and 65.21 at 250 — in place of the table’s 53.51 and 61.59. The sparse columns come out identical in the re-run.
To measure the uncertainty we drew ten separate stratified label samples at each training size (representations fixed; the dense columns use the vectors of the re-run):
### ten label samples per size: mean macro-F1 [min, max]; representations fixed
n tf-idf tf-idf 1-2 gram word2vec fastText tf-idf > fastText
100 48.49 [48.35, 48.85] 48.45 [48.37, 48.69] 59.68 [54.63, 61.68] 58.66 [53.80, 61.44] 0 / 10
250 48.98 [48.35, 49.63] 48.70 [48.37, 49.17] 61.69 [58.58, 65.07] 60.96 [56.55, 64.83] 0 / 10
500 50.65 [49.46, 51.94] 49.17 [48.51, 49.95] 61.85 [59.63, 64.18] 61.96 [60.47, 63.84] 0 / 10
1,000 56.57 [55.14, 59.80] 51.68 [50.39, 53.36] 62.15 [61.61, 63.32] 61.96 [61.01, 63.15] 0 / 10
1,250 59.59 [57.39, 62.41] 53.43 [51.60, 56.44] 62.01 [60.81, 62.91] 62.00 [60.83, 63.29] 2 / 10
1,500 61.87 [60.50, 65.01] 55.63 [53.54, 58.41] 62.06 [60.69, 63.43] 61.90 [60.36, 63.52] 4 / 10
1,750 63.95 [62.34, 65.35] 57.81 [56.37, 60.89] 61.89 [60.30, 63.11] 61.64 [60.22, 62.92] 10 / 10
2,000 65.54 [63.74, 67.20] 59.69 [57.89, 61.64] 61.70 [60.38, 62.67] 61.44 [60.35, 62.82] 10 / 10
2,500 67.52 [66.29, 69.69] 63.42 [62.18, 64.71] 61.68 [60.65, 62.70] 61.55 [60.61, 62.53] 10 / 10
5,000 70.97 [69.72, 72.27] 71.49 [70.06, 72.88] 61.48 [60.00, 62.64] 61.18 [59.98, 62.40] 10 / 10At the same size, changing only which documents are labelled moves the scores by a few points: at 1,000 labels TF-IDF ranges from 55.14 to 59.80 and fastText from 61.01 to 63.15. Read any ordering that rests on a single subsample together with that spread.
The chart shows word TF-IDF, averaged fastText and (as the unlabelled dashed line) averaged word2vec; character TF-IDF is left off for readability, and the results for bigram TF-IDF are in the ten-sample table above. Nine training sizes, the same test set throughout, a single subsample at each size. At 250 labels fastText is 14.57 points above word TF-IDF (word2vec 13.06); at 40,000 labels it is 10.59 points below. In this setup the number of labels is related to the ordering; the same crossover should not be expected for other methods and settings.
Do not read the crossover range as a transferable number. In the single sequence of subsamples of the first table, word TF-IDF is below the embedding methods at 1,000 labels and above them at 2,500; the place where the lines cross (about 1,500) was a value read between two measurements, not a measured point. Measured over ten label samples, TF-IDF beats fastText in none of them at 1,000 labels, in two of ten at 1,250, in four at 1,500 and in all ten at 1,750: in this setup the ordering changes between 1,250 and 1,750 labels. For word bigram TF-IDF, the chapter’s highest-scoring model, the range lies further right: with few labels it stays below unigram TF-IDF (it has 3.4 times as many columns to estimate from a handful of negatives), and it beats fastText in two of ten samples at 2,000 labels and in nine at 2,500. All of these numbers were read off the same test set. To choose a method, split train again and decide on the validation slice; use the test set only to evaluate the method chosen beforehand.
What transfers is not the threshold but the shape: in a low-label regime a representation learned from unlabelled text can have a head start, and as labels accumulate the sparse linear model closes it. Plot your own curve for your own task.
Read the left edge of that chart carefully. With 250 labelled reviews — of which 16 are negative — word TF-IDF scores 48.53 macro-F1 and the majority rule 48.37. We looked at this model’s predictions: it calls exactly one of the 10,000 test reviews negative (and that review really is negative). So in this run it behaves, in effect, like the majority rule; the paired bootstrap interval of the 0.16-point difference runs from 0.00 to 0.51. Averaged fastText scores 63.10 with the same 250 labels. Over ten label samples, the gap between fastText and TF-IDF at 250 labels averages 12.0 points (between 8.2 and 16.1).
One explanation consistent with this pattern: with few labels, the coefficients of the rare features of a word-based linear model may not be estimated reliably — 16 negative examples are 16 sparse rows drawn from a 30,195-column space. The embedding model instead offers a shared structure learned beforehand from the unlabelled text of 40,000 reviews: word2vec, for example, places kalitesiz, dandik and dayanıksız close to one another (the neighbour lists above), and the classifier has far fewer coefficients to estimate in 100 dimensions. Testing this explanation means looking at the coefficients and errors of the low-label condition and at stronger regularisation options; this chapter did not do that.
Self-supervised training extracts information from the patterns in which words occur together, without looking at labels. Those patterns can be useful to a classifier with few labels; that does not mean the model has learned all of Turkish grammar, or sentiment polarity. Which examples the learned information actually fixes on this task has to be examined separately. This is the small-scale version of the general principle behind the pretrained models in this book.
And then the curve flattens. Averaged fastText scores 61.16 macro-F1 with 500 labels and 61.36 with 40,000; in the ten-sample measurement, too, the means between 500 and 5,000 labels stay in the 61.2–62.0 band. That closeness suggests more labels bring limited gain with the current pooling and settings. To say it is a fixed ceiling of the representation, you would have to compare normalisation, C, the decision threshold, different seeds and alternatives that keep word order. Word TF-IDF, in the same table, keeps rising until about 10,000 labels and then drops slightly in this single run.
The two curves have different shapes: in this setup the averaging-based static dense representation gains almost nothing after a few hundred labels, while the sparse linear model keeps gaining as labels arrive. Keeping the sentence that narrow matters — what was measured is not “dense representations” but this setup: averaged 100-dimensional static vectors trained from scratch. What generalises is the question itself: on this task, am I limited by labels, by the representation, or by the settings?
4.10 What it costs
Choosing a method means weighing training and inference cost along with the score. This experiment measured three things: the time to build the representation, the time to vectorise the 10,000 test documents, and the memory of the training matrix. The size of the model artifact, the memory taken by the vocabulary and embedding table, the size of the classifier itself, peak RAM during training, 95th-percentile (p95) latency and the cost of deploying an index are all absent here.
### cost, on one CPU core, 40,000 training documents (single run, not repeated)
build the vectorise training
representation 10,000 test matrix
tf-idf 0.5 s 0.11 s 9.7 MB
tf-idf 1-2 gram 0.9 s 0.17 s 15.4 MB
tf-idf char_wb 3-5 2.9 s 0.72 s 144.8 MB
word2vec + averaging 14.7 s 0.11 s 16.0 MB
fastText + averaging 31.6 s 0.27 s 16.0 MB
doc2vec 8.5 s 1.73 s 16.0 MBThe training matrix column is the memory of the training matrix only; it does not cover the model, the vocabulary or the embedding table, or peak RAM. The timings come from single runs. When we re-measured the vectorising times in round 1 five times each, the run-to-run difference stayed around ten percent (0.82–0.90 s for the bigram scheme, for example); do not compare small timing differences for that reason.
Two entries deserve comment.
For character n-grams the balance of memory and score depends on the condition. With the full training set, character TF-IDF scored 70.39 macro-F1 from a 144.8 MB matrix (301 non-zeros per document); word TF-IDF scored 71.95 from 9.7 MB. (The bag-of-words matrix has the same sparsity pattern and therefore the same size; only the tf-idf row appears in the table.) At the 1,000-label point, though, the character features were 3.2 points ahead of word TF-IDF. So re-evaluate the memory–score balance against the number of labels and the spelling noise in the text.
doc2vec must run gradient descent to vectorise a new document. 1.73 seconds for 10,000 reviews, against 0.11 for averaging — a 16-fold difference, and unlike training that cost recurs on every new document. Look at the absolute number too: 0.17 milliseconds per document is still small for most services, and the decision belongs to your latency budget. In this setup doc2vec’s macro-F1, 55.13, is the lowest of the dense rows examined. No measurement here supports making doc2vec the default on this short, noisy review task. In chapter 3’s 20 Newsgroups comparison (Chapter 3) doc2vec was ahead of averaging; here it is 6.3 points behind. This chapter did not test whether the difference between the two results comes from document length, language or the amount of data.
The hybrid, and an honest negative result
Scale first. The two blocks of the hybrid arrive on different scales: char_wb TF-IDF rows are L2-normalised (norm 1; ~301 non-zeros per document, averaging 0.05), while the averaged fastText rows are not normalised and have a mean norm of 2.5 (the mean absolute value of their components is 0.2). Handing both to a single L2-penalised C = 1.0 means the relative weight of the two blocks is chosen by the default rather than by us. So we measured both versions: first the plain concatenation, then the one that normalises each block separately and chooses the mixing weight on a validation slice.
The obvious move is to concatenate: character TF-IDF for spelling robustness, averaged fastText for semantics, one classifier over both.
### tf-idf char_wb 3-5 ⊕ averaged fastText (95,039 dimensions)
accuracy 89.36 macro-F1 71.65 F1(neg) 49.24The plain concatenation scores 71.65 macro-F1 on this test; char TF-IDF alone scored 70.39, fastText alone 61.36, and word bigram TF-IDF 76.21. With the paired bootstrap intervals of the re-run: hybrid − char TF-IDF is +1.28 points (0.65 to 1.92), and bigram − hybrid is +4.54 points (3.42 to 5.63). So the plain hybrid is ahead of its own parts and behind the bigrams.
In the version that adjusts the block scales, we L2-normalised the dense block row by row and multiplied it by a coefficient \(\alpha\); we chose \(\alpha\) (0.25, 0.5, 1, 2, 4) and C (1, 3, 10, 30) on the 32,000/8,000 validation split and refitted on all 40,000 documents. The chosen setting (\(\alpha = 2\), C = 10) scores 74.69 macro-F1 on the test set: 3.02 points above the plain concatenation (2.00 to 4.06). Given the same tuning budget, char TF-IDF alone also rises, to 73.46 (C = 30); the hybrid’s gain over it is 1.23 points (0.33 to 2.15). Untuned bigram TF-IDF (76.21) is still 1.52 points ahead of the tuned hybrid (0.20 to 2.80). So adjusting the scales changes the hybrid’s score substantially — reading the plain concatenation as “the hybrid does not help” would have been wrong — but in this setup it is not enough to pass bigram TF-IDF.
On cost, the rough sum of the representation-building times is 2.9 + 31.6 = 34.5 seconds (both from the cost table above), against the bigram model’s 0.9; we did not measure end-to-end training and serving cost. Concatenation is a real technique, and here it gives a measurable gain over its components. But reporting the hybrid as a success by comparing it only against its own parts would be exactly the kind of selective baseline this chapter is arguing against.
4.11 Choosing, on purpose
The seven representations differ in their measured scores, memory and coverage. The table below summarises them together with the conditions of this experiment; the choice of method should follow the task and the data budget.
| Strengths | Weaknesses | Choose it when | |
|---|---|---|---|
| Bag of words | One of the fastest representations to build in this experiment; word features map onto coefficients. At a fixed C = 1.0 it scored above word TF-IDF (73.51 against 71.95). |
Does not normalise for document length by itself; applies no informativeness weighting at the representation stage. Rare forms are separate features. Coefficients alone do not show causes. | One of the first baselines for short-document classification; choose its settings on a validation set. |
| TF-IDF (words) | Learns document frequencies from the training text; word features can be inspected. With C tuned on validation it scored 74.71 macro-F1 and moved ahead of raw counts (gap 1.20; paired bootstrap interval 0.42–1.99). |
Fell behind raw counts under a shared, untuned C; measure both on short text. An unseen word form has no column; the model relies on the familiar words in the same document. Turkish inflections spread over separate columns. |
A general-purpose first sparse baseline; choose the vectoriser and C inside the train/validation boundary. |
| TF-IDF (word unigrams + bigrams) | Highest macro-F1 in this chapter: 76.21. Keeps local two-word sequences as separate features. Still a linear model whose coefficients can be inspected. | 101,659 columns (3.4× the unigram model). With few labels it stayed below unigram TF-IDF (51.7 against 56.6 on average at 1,000 labels). Which sequences contribute how much to the score needs separate analysis; bigrams unseen in training are not represented. | You have a few thousand labels and a short-text task where local patterns may matter — sentiment, intent, triage: a strong sparse candidate to try. |
| TF-IDF (char n-grams) | Can use partial spelling overlap: a misspelt word still shares most of its n-grams with the correct one. On the single-subsample learning curve it was 3.2 points ahead of word TF-IDF at 1,000 labels. Less dependent on a word tokeniser — but lowercasing, Unicode normalisation and broken characters are still language-specific decisions. | 144.8 MB training matrix and 301 non-zeros per document on average. Lower macro-F1 than word TF-IDF at full data (70.39). Unseen n-grams are discarded. N-gram coefficients can be inspected, but they are harder to interpret than word coefficients. | Spelling is variable, and the memory and score you measure are acceptable. |
| Averaged word2vec | 61.39 macro-F1 from a 100-dimensional document vector — 302× fewer coordinates than word TF-IDF. Trained on unlabelled text. In this setup, ahead of word TF-IDF in all ten label samples up to 1,000 labels; the order changes between 1,250 and 1,750 labels, and that range does not transfer. | 46.4% of the distinct test forms are not in the vocabulary; the average skips them. The vector of a rarely seen in-vocabulary form may be poorly estimated. Averaging loses word order and some local distinctions. A dense matrix need not be smaller than a sparse one (16.0 MB against 9.7 MB). | Labels are scarce, or you need dense-vector search; as a candidate to test on the target data. |
| Averaged fastText | Built a vector for every test form this experiment’s tokeniser produced. Highest score at 250 labels on the single-subsample curve (63.10). | Similar spelling can also bring forms of opposite sentiment together: kalitesiz → kalitesi; 60.7% of top-5 neighbours start with the same five letters as the query (6.5% for word2vec). Its macro-F1 could not be distinguished from word2vec’s (61.36 against 61.39). Training took about twice as long as word2vec; the memory of the subword table was not measured. |
Spelling and form variety are high; with task performance and memory measured. |
| doc2vec (PV-DBOW) | Learns a separate vector for each training document; there is no averaging step. | Lowest score in this run, 55.13. Inferring a vector for a new document needs extra optimisation (1.73 s for 10,000 reviews; 16× averaging). This variant was not shown to represent word order. Extra hyperparameters. | A separate experiment shows that learning the document vector directly beats averaging on your data. |
The decision procedure, compressed to something you can apply before writing any code. Note what it is: a list of questions, not a list of thresholds.
### choosing a representation, in order
1. Write down the task, the costs of a false alarm and of a miss, and the
main metric.
2. Fix the data version, the train/validation/test splits and the unlabelled
text you have access to.
3. Keep preprocessing, the vocabulary and IDF away from the test labels;
document which text is used where.
4. Build the cheap baselines:
word counts / tf-idf
word n-grams
character n-grams (especially if the spelling is noisy)
5. If you have enough unlabelled text, or suitable pretrained vectors, add an
embedding candidate.
6. Choose each candidate's settings with the same validation budget; look at
the OOV rate and at error examples (order/negation, or spelling/OOV?).
7. Make the decision on the validation set:
score + interval · latency · memory · explainability need
8. Use the locked test set once, for the chosen solution.The numbers this chapter observed — the change of order between 1,250 and 1,750 labels, the counts-versus-tf-idf ordering on short documents, the timings — are useful as starting expectations at step 5, not as rules at step 1. The same goes for “sparse if you need an explanation” and “dense if you need similarity search over millions of vectors”: both are strong first candidates, not absolutes. Sparse lexical retrieval can be entirely standard over millions of documents, and dense models can be made explainable.
4.13 Intuition checks
Answer before expanding.
Not one model name, but a comparison plan.
First settle the label of a ticket, the business metric and the test design. Two million unlabelled tickets are a resource for both families: you can train word2vec or fastText on that text, or adapt pretrained vectors to the domain; word and character TF-IDF can learn their vocabulary and IDF weights from the same text. The difference is what can be learned: TF-IDF is limited to vocabulary and document frequency, while an embedding model also extracts co-occurrence structure from the same text. Nor is that text “free”: training time and data access are costs.
In this chapter’s product-review experiment the averaged embeddings were about 11 points ahead of word TF-IDF at 500 labels (mean over ten label samples: 62.0 against 50.7), so expecting an embedding candidate to be strong at 400 labels is reasonable. But it is an expectation: support tickets are another domain and the label is another label. Compare word n-gram, character n-gram and embedding candidates with the same validation budget. Try fastText in particular, because the tickets are Turkish text written by users: here 46.4% of the distinct test forms were outside the word2vec vocabulary; measure the rate on your own data.
Then say the second half out loud: decide on further labelling from your own learning curve. Do not treat the change of order at 1,250–1,750 labels in the product-review experiment as a threshold for this domain.
First the accuracy of the majority rule on the same test set, and the confusion matrix.
93% is below the 93.68% you get on this test set by giving every input the majority class. That does not show the model is useless: a model with lower accuracy may be finding most of the negatives (this chapter’s bigram TF-IDF finds 76.3% of them at 92.57% accuracy). So ask for the recall and precision of the negative class, and for the workload its false alarms create at the chosen decision threshold. A decision to ship also needs the cost of a missed complaint and of a false alarm, and the class distribution of the target traffic.
The general form of the question is what does the trivial baseline score? — and you should ask it before you look at any reported score.
In this run word2vec scored 61.39 macro-F1 and fastText 61.36; the difference cannot be distinguished (paired bootstrap interval −0.60 to +0.77; the training seed alone moves either model by 0.5–0.8 points). Coverage went from 53.6% of the distinct test forms to 100%. Read the comparison for what it is: two models trained by different methods, so it does not isolate coverage, and the causal effect of coverage cannot be separated out of the score difference.
There are candidate explanations. fastText’s neighbours shift towards forms close in spelling (60.7% of top-5 neighbours start with the same five letters as the query; 6.5% for word2vec), and kalitesiz (“low quality”) retrieves kalitesi (“its quality”), which carries no negative suffix. In averaged vectors the cosine between a sentence and its negation is high for both models: 0.78–0.96 for fastText, 0.72–0.96 for word2vec. These are candidates; confirming them takes an ablation that removes and restores the out-of-vocabulary tokens within one model, an equal tuning budget and a paired error analysis.
The transferable lesson: an improvement in one property of a representation shows up in the task score only if that property was what limited the task. Measure what limits you before you set out to fix something.
We do not know; these conditions were not measured.
Larger vectors, or pretrained Turkish ones (Grave et al. 2018), may change the neighbourhoods, the cosines of the negation pairs and the task score; neither the direction nor the size can be read off this chapter. The one thing that does not change is the mathematical one: the plain average of two sentences that contain the same words the same number of times is the same whatever fixed word vectors are used, which is why the 1.0000 of the word-order example stays. The 0.78–0.96 negation cosines are different: those sentences differ in which word appears, and other vectors could put beğendim and beğenmedim further apart.
The way to settle it is exercise 13: rerun this chapter’s experiment with 300-dimensional pretrained vectors, and determine the ordering, and the range of labels over which the order changes, from the new experiment.
4.14 Interview questions
These questions exercise which measurements and which limits you bring into your reasoning when choosing a representation. Build every answer from the task, the amount of data, the result of a comparison, and the conditions that could change that result. Almost every answer below rests on something this chapter measured.
In your answer, first name the axis that decides the task: the number of labels, the variety of word forms, the need for order, or cost. Then give the relevant measurement if there is one, and say which new data or experiment would change the conclusion.
Embeddings or TF-IDF for text classification? Build the answer in three steps.
The first question. Ask how many labelled examples there are, what unlabelled corpus is available, and what kind of task it is (document-level or aspect-level; document length; spelling/OOV noise). Word TF-IDF also learns its vocabulary and IDF weights from the unlabelled training text; an embedding model additionally learns co-occurrence and subword parameters from it. Which of the two helps with the task label is measured on the target data.
The number. In this experiment, at 250 labels averaged fastText scored 63.10 macro-F1 against word TF-IDF’s 48.53 — very close to the majority rule; at 40,000 labels that same TF-IDF scored 71.95 against fastText’s 61.36. Over ten label samples the order changes between 1,250 and 1,750 labels.
The limit. The range depends on which sparse model you put on the curve: bigram TF-IDF, which scores 76.21 at 40,000 labels, stays below the unigram model with few labels and passes fastText only between 2,000 and 2,500 labels. This from-scratch averaged-embedding curve flattens after a few hundred labels; word TF-IDF keeps rising until roughly 10,000 labels (72.55 at 10,000 and 71.95 at 40,000 in the single run). Whether more labels, a change of settings or a different model will help is something to determine on your own data, with a repeated learning curve and a validation set.
Why can averaging word vectors fall behind counting words? Shared content can weaken the directional contribution of the one word that makes the difference. Measured on five opposed review pairs: a review and its negation sit at cosine 0.78 to 0.96 under averaged fastText and 0.19 to 0.69 under TF-IDF. A high cosine does not rule out a correct classification: the classifier gave the negated version a probability below 0.5 in all five pairs. Weighted averaging with the first principal component removed (Arora et al. 2017) is a cheap option to try; it does not bring word order back either, and its effect on negation has to be measured separately.
Which candidates does the variety of Turkish word forms bring up? In this experiment 7.27% of test tokens, and 46.4% of the distinct test forms, were outside the word2vec vocabulary; the rate varies by corpus and domain. A form-level OOV rate that high makes subword-based fastText and character n-gram TF-IDF candidates: fastText could build a vector for every form in this test. Then say what you measured: taking coverage to 100% did not change macro-F1 in any distinguishable way (61.39 against 61.36). On your own text, measure coverage and the task score separately.
How would you compare two representations beyond their headline scores? Cross-tabulate the paired predictions. Plain word TF-IDF and averaged fastText agree on 84.1% of the test set; where they disagree, TF-IDF alone is right 1,315 times and fastText 272 — and those two counts are also what a McNemar test needs. Then read examples, but do not draw conclusions about all 272 from six selected reviews: the share of out-of-vocabulary tokens is the same in the two groups of disagreements (9.5%); what differs is the class distribution. For two models with close scores, add a paired bootstrap interval and a systematic labelling of the errors.
When does a difference in cost matter? Quote the numbers together with their scope. On this machine, building the representation for bigram TF-IDF took 0.9 s and its training matrix is 15.4 MB; for fastText, training the word vectors took 31.6 s and the document matrix is 16.0 MB. These are not total model memory or serving latency. At this scale both options look cheap; measure whether cost decides anything together with how often you retrain and with your latency target. At another scale the picture changes: in chapter 12’s own measurement (Section 12.9) a prompted model costs 142× more per document than a fine-tuned encoder. When you carry such a number over, state its task, its hardware and the limits of the measurement too.
If we are going to try a larger encoder as well, why is this comparison useful? Two reasons. First, simple models give strong starting references suited to the task: in chapter 12’s own measurement (Section 12.3) the fine-tuned encoder falls behind the methods of this chapter below roughly 500 labels, and the order changes between 500 and 1,000 labels. Compare that result with your own data, label budget and tuning conditions; do not assume an advantage or a threshold this chapter did not measure. Second, this chapter is where the diagnostic habits come from: the negation-cosine probe, the disagreement cross-tabulation, the distinction between coverage and score, and the cost check. They apply to more complex models unchanged.
4.15 Where to go deeper
| resource | what it is |
|---|---|
| Let’s build GPT: from scratch, in code — Andrej Karpathy | shows, in about two hours, how a language model is built from scratch in code; a complementary resource for the reader who wants to move on from this chapter’s basic methods to sequence-aware models |
| Hugging Face LLM Course | formerly the NLP Course; free and runnable; it starts from the transformer models this book reaches in Part II, so read it as the next step after this chapter’s classical baselines |
fastText source repository |
an implementation of unsupervised subword vectors and of supervised text classification. The repository was archived on 19 March 2024; check its maintenance and release status before using it in production |
| scikit-learn text feature guide and model selection | the first explains the word and character vectorisers used in this chapter, the second the validation and cross-validation machinery |
| Chip Huyen’s blog | on picking baselines and deciding when a model is good enough to ship |
Note that the name “fastText” covers two different things: unsupervised subword word vectors (what round 2 of this chapter used) and the supervised fastText classifier (the first paper below). They come from the same library but they are not the same thing.
- fastText for classification (Joulin et al. 2017). The supervised fastText classifier learns word and optional word-n-gram representations together with the task labels; the paper reports training in minutes on a CPU. It is not the same model as this chapter’s setup, which averages unsupervised subword vectors and fits logistic regression on top. It is a candidate worth adding to a comparison, with CPU cost and task performance measured on the target data.
- Enriching word vectors with subword information (Bojanowski et al. 2017). The paper behind the coverage result above; its experiments centre on morphologically rich languages (Czech, German and Russian among them), the setting where subwords help most.
- Distributed representations of sentences and documents (Le and Mikolov 2014). doc2vec’s original claims, worth reading against this chapter’s measurement of it.
- A simple but tough-to-beat baseline for sentence embeddings (Arora et al. 2017). SIF takes an average weighted by word frequency and removes a common component. It is an alternative that could be tested for this chapter’s dense rows; its effect on the score and its computational cost were not measured here.
- The Hitchhiker’s Guide to Testing Statistical Significance in NLP (Dror et al. 2018). The protocol behind this chapter’s paired bootstrap and McNemar calculations; exercise 8 asks you to extend it over the training split.
- scikit-learn (Pedregosa et al. 2011) and gensim (Řehůřek and Sojka 2010). The two libraries this chapter is built from; the user guides are unusually good.
4.16 Exercises
The label on each exercise marks its level: Basic consolidates a concept, Applied makes you measure something on your own data, Advanced goes past this chapter, and Research is open-ended.
- Examine the crossover under different vocabulary thresholds. (Applied) Rebuild the learning curve of word TF-IDF for
min_df=1,2and5. In every condition keep fixed which training text the vocabulary and the IDF are learned from (all 40,000 reviews in this chapter), and keep the test labels out of any selection. Write down beforehand which way the score will move at the low-label end; then compare with your result and report the number of features together with the validation score. - Kill the imbalance instead. (Basic) Drop
class_weight="balanced"and rerun every model. Report accuracy and macro-F1 for each. Which methods collapse to the majority class? Does the ranking by accuracy change more than the ranking by macro-F1 — and if so, why? - Weighted averaging with SIF. (Advanced) Implement Arora et al. (2017): estimate
p(w)from the training text and average the word vectors with weight \(a/(a + p(w))\) for \(a = 10^{-3}\). Learn the common component to remove from the training document vectors only, and apply the same transformation to the validation and test documents. Compare with plain averaging, word TF-IDF and bigram TF-IDF under the same evaluation protocol; report the gain or loss with one metric fixed in advance and a paired bootstrap interval. - Examine negation pairs. (Applied) Build twenty opposed pairs from your own domain; for each model record the cosine, the label the classifier gives and the decision score. Find a pair that is separated correctly in spite of a high cosine, and explain it by projecting the difference of the two vectors onto the direction of the classifier’s coefficients. Do not derive a general error rate from your twenty chosen examples.
- Test the conditions of character n-grams. (Applied) On the single-subsample curve, character TF-IDF was ahead of word TF-IDF at 1,000 labels and behind it at 40,000. Form a hypothesis for that difference from the number of features and the number of observations per feature. Then sweep
min_dfandCon validation data and test whether the difference is explained by the number of columns alone. - Set up a new, aspect-level task. (Research) Define with an explicit labelling rule which product aspect (shipping, price, quality) a review judges positively or negatively. Split the documents into train/validation/test and evaluate every representation on the same splits. Write down beforehand on which examples sensitivity to word order will be needed; then check against the scores and the error examples.
- Examine the combining-dot damage. (Applied) Running round 1 with
str.lower()on this corpus changes nothing; there is no uppercase left to get wrong. The damage happened earlier: 4,194 documents carry a U+0307, and in this subsample the mark always follows ani. Look through the examples and write a normalisation rule that targets only that brokenisequence (in another corpus U+0307 may be a genuine combining mark). Apply the rule to the training and test texts alike and rerun the round-1 table; report the change in vocabulary size (this chapter found 30,195 → 29,823), example tokens and the difference in macro-F1. Write down beforehand which of the two you expect to move more, and why. - Measure the uncertainty, and extend it. (Advanced — if you do only one exercise, do this one.) First reproduce this chapter’s paired bootstrap table from round 2: for model pairs fixed in advance, resample the same 10,000 test documents in paired fashion and compute the 95% interval of the macro-F1 difference. For the accuracy difference use McNemar’s test on the paired right/wrong outcomes; do not confuse the intervals of individual models with the interval of a difference. When you test many pairs in one table, state the multiple comparisons and the fact that the test set was already examined earlier in the chapter. Then add the source this chapter did not measure: repeat the experiment with five different train/test splits and report how much of the result comes from the split. For the protocol see Dror et al. (2018).
- Choose a decision threshold. (Applied) Evaluate a model chosen in advance at three different decision thresholds on validation data. Compare the workload created by false alarms with the cost of a missed negative review, and set the threshold there. On the locked test set, report the confusion matrix of the chosen threshold once, and write the sentence that justifies the decision to the person who will handle the false alarms. That sentence, not the macro-F1, is the deliverable.
- Is IDF really the culprit? (Applied) Rerun the ablation ladder from this chapter yourself — raw counts, counts + L2, log-TF + L2, TF-IDF — then add one rung: repeat every row with
Ctuned on a validation set. How much of the normalisation effect survives? - Analyzer audit. (Basic) Write ten sentences containing punctuation, digits, an emoji and a U+0307. Put the features produced by
analyzer="word"next to those produced bychar_wb. Write down what extra material the character arm sees, and how that affects the round 1 comparison. - Extend the repeated learning curves. (Research) This chapter used ten stratified subsamples at each label budget and kept the representations fixed on the text of all 40,000 reviews. Raise the number to 20, add character TF-IDF, and report the model difference within the same subsample for every repeat. If the change of order shows up in every repeat, give the distribution of where it happens; if in some repeats it never shows up, show that plainly too. Finally, learn the representation as well from the text of the labelled subset only, and report how the curve changes.
- Test pretrained vectors. (Advanced) Replace round 2’s from-scratch vectors with the 300-dimensional pretrained vectors fastText publishes for Turkish (Grave et al. 2018); check that the preprocessing policy of the vectors (lowercasing, tokenisation) matches yours. Keep the validation budget and the test protocol fixed and redraw the learning curve; report which of improvement, deterioration or no change actually happened. Because
kalite iyi fiyat kötüandkalite kötü fiyat iyicontain the same tokens, their plain averages are equal whatever fixed vector table you use; check the cosine against a numerical tolerance.