A Representation Bake-Off

Chapter 4 · Applied Natural Language Processing

Dr. Batuhan Bardak

After this lecture you can

• build a pipeline from raw text to an evaluated prediction for a real, imbalanced dataset, using the methods of the previous two chapters
• explain why accuracy alone is not enough on this imbalanced task, and compute macro-F1 from its per-class arithmetic
• form a starting expectation for count and embedding representations on a task — and then test that expectation with an experiment
• measure the out-of-vocabulary rate on Turkish text, and evaluate coverage, the quality of the vectors and the task score separately
• show with an example the word-order information that a unigram bag and an average of word vectors both lose

The chapter’s five learning objectives, condensed. In full, the third weighs the amount of labelled and unlabelled data, the structure of the task, the preprocessing and the tuning budget together; the second also asks when a different business goal calls for a different metric.

What we will look at: one dataset of Turkish reviews, with count and embedding representations compared under one shared evaluation protocol. We first settle the metric that suits the class imbalance and the preprocessing, then look at each method’s score, coverage, cost and errors. Finally: how the result changes as the amount of labelled data changes, and which word-order information both representation families are blind to.

The numbers and code outputs come from real runs on one machine, with scikit-learn 1.9.0, gensim 4.4.0, NumPy 2.5.2 and pandas 3.0.5 on Python 3.12.13. The data is public. The code shown is abridged; reproducing every table also needs the full data-splitting, training and reporting steps; the lab works from the chapter notebook. The timings belong to runs on that one machine.

If the lecture is split in two, a natural break is after the slide “Do not read the number of dimensions as memory”: round 1 and round 2 are then complete.

Two chapters, two claims

• Chapter 2: sparse count vectors can be a strong, cheap, interpretable baseline
• Chapter 3: giving every word form its own independent feature can leave a coverage problem on the inflected forms of an agglutinative language like Turkish
• Its measurement: build the vocabulary on 90% of a Turkish Wikipedia sample, test on the remaining 10% — 7.33% of test tokens were never seen in training, and 28.99% of distinct word types
• The two claims do not logically contradict each other; which one dominates on a particular dataset is something you only learn by measuring

A bake-off, in the engineering sense: several candidate methods, one task, one protocol, and the same measuring stick for all of them.

Chapter 2 taught counting words; chapter 3 taught placing them as coordinates. Each chapter ended with a claim. A sparse classifier can be strong while a word-level vocabulary generalises badly across morphology at the same time.

Baseline: the reference result a new method is judged against, on the same data and the same metric. A plain majority-class rule can be a baseline, and so can a strong TF-IDF system. Reading a score also takes the class distribution and the cost of each kind of error.

Agglutinative: a language in which morphemes mostly attach to the word as separable affixes. In Turkish, inflectional and derivational suffixes can give one root or stem many surface forms; how many depends on usage and on the corpus.

The 7.33% and 28.99% belong to that sample and that definition of the vocabulary; they are not constants of every Turkish corpus.

Chapter 3’s own comparison ran on 20 Newsgroups: English, balanced, four classes, 2,327 training documents. The Turkish product reviews here differ in data, task and training protocol all at once. Where the conclusions differ, we cannot say how much each factor contributes without comparisons that change them one at a time.

What is held fixed, and what is not

The same in every row

• the train/test split
• the LogisticRegression class
• the class weighting
• the random seed

Changes from row to row

• the vectoriser’s vocabulary threshold
• the tokenisation path
• the scale of the features
• the separate training of the dense representations on unlabelled text

• Every row uses C = 1.0: TF-IDF rows arrive with norm 1.0, the averaged vectors are not normalised — the same C need not mean the same regularisation

The results compare these concrete setups of the methods; we do not read the gap between them as the effect of the representation family alone.

All seven rows use logistic regression; but the vectorisers do not share the same preprocessing, vocabulary threshold, feature scale or extra training cost. So a difference in score may be decided by the representation together with these accompanying design choices.

Holding a hyperparameter fixed is not the same as being fair to every representation. C is logistic regression’s regularisation setting; in scikit-learn it is the inverse of the penalty strength, so a smaller C shrinks the coefficients harder. How hard a given C actually shrinks depends on the scale of the features. TfidfVectorizer L2-normalises its rows by default; the averaged word2vec and fastText vectors are not normalised and their norms vary.

In the main comparison C was not searched for any row; one later ablation tunes it for two sparse rows only. A fairer design gives each representation a comparable validation budget for its C and its normalisation option; it states which data the vocabulary, the IDF weights and the embeddings were fitted on, and it keeps the test labels out of the selection. This version does not do that, which is a real limitation and is flagged again where it matters.

The chapter’s figure, in words: the 40,000 training documents are turned either into a sparse word TF-IDF matrix with 30,195 columns and 9.7 MB, or into a dense document matrix with 100 columns and 16.0 MB. Logistic regression is applied to both. Those are sizes of the training matrix, not of a model file; we come back to them on the slide about dimensions and memory.

The problem: Turkish product reviews

A binary sentiment task: 235,165 Turkish product reviews, labelled positive or negative.

• Short and informal texts: misspellings, missing diacritics, run-together words; all of the distributed text has already been lowercased

Three reasons for choosing it:

1. It is Turkish: chapter 3’s morphology argument can be tested on real user text
2. Its classes are imbalanced — the positive class is 14.8 times larger — so the metric has to be argued
3. It is large enough to subsample: a learning curve valid for this data

Barmanbay 2020 — Turkish product reviews

We set the comparison up as a binary sentiment task on Turkish product reviews, with imbalanced classes and inconsistent spelling. The dataset is distributed as a single parquet file.

The dataset card says the reviews were collected online; it does not document which site they came from, when, or how the labels were produced — a star-rating threshold, manual annotation, something else. That last gap matters, because the data annotation chapter is built entirely on the label quality of this corpus and has to work without knowing how the labels were made.

Provenance. The licence is stated differently in two places: the licensing section of the card and the licence file of the original GitHub repository say CC BY-SA 4.0, while the licence field at the top of the Hugging Face page says “unknown” (checked on 10 October 2026). CC BY-SA is share-alike, a stricter licence than this book’s own CC BY 4.0, so check the original repository’s licence and the card together before you redistribute the data or anything derived from it.

The third reason in full: the dataset is large enough that we can compare how the methods behave as the amount of labelled training data changes. That comparison is a learning curve valid for this data and these settings.

Check the data before you split it

### checks before de-duplication
missing text or label         0
empty texts                  21
texts with both labels       77

### the raw data
rows as downloaded      235,165
after de-duplication    233,508      (1,657 exact duplicates removed)
label 1 (positive)      218,747      93.7%
label 0 (negative)       14,761       6.3%

• 77 texts carry both labels (297 rows in all); drop_duplicates keeps the first row and positive reviews come first in this file, so all 77 stayed with their positive label
• De-duplicate on the text before splitting, and report the policy you chose: the 1,657 duplicates are only 0.7% of the file; the step is necessary but not sufficient

The first three lines are the checks made before de-duplication. Nothing is missing. 21 rows have an empty text; de-duplication collapses them into one row. 77 texts carry both labels: the same short review is labelled positive in one place and negative in another. In this file every positive review comes before every negative one, so all 77 texts stayed with their positive label. The chapter left that choice as it was; 17 of the 77 texts fall inside the 50,000-review subsample. On your own data, choose the policy for conflicting labels — keep the first, drop them all, or decide by hand — explicitly, and report it.

The class counts can be checked against the source. The dataset card reports 220,284 positive and 14,881 negative reviews. After de-duplicating on the text the counts are 218,747 and 14,761; the differences are 1,537 and 120, and together they make 1,657 — the number of duplicates removed. That agreement shows the counts are consistent; it does not show which records de-duplication kept.

Why before splitting: the same text sitting on both sides of the train/test split can, especially with short reviews, let a memorised answer score as a generalisation. Not sufficient on its own: templated or near-duplicate texts can survive it, and a text that carries two different labels needs its own decision.

The split, and two warnings about it

### the split (stratified, seed 42)
train   40,000    37,471 positive    2,529 negative   rows 8205c2be5ebf
test    10,000     9,368 positive      632 negative   rows 634cdb3bb002

• A stratified 50,000-review subsample of the 233,508 de-duplicated reviews, then an 80/20 split, again stratified
• rows is a digest of the selected row positions: the same 12 characters mean the same rows; the class totals do not show that
• Reproducing it: “stratified with seed 42” does not name a procedure; it names a family of procedures
• The test set and selection: every number below is read off the same 10,000 test reviews — read these results as exploratory

The two calls: train_test_split(df, train_size=50_000, stratify=df["label"], random_state=42), then train_test_split(sub, test_size=0.2, stratify=sub["label"], random_state=42). The digest is the first 12 characters of a SHA-256 over the sorted row positions in the raw file. Many other splits give the same class totals.

Reproducing it. The first version of the chapter recorded the split only as “stratified with seed 42”. The members of that family disagree: it does not say which call did the sampling, in which order, or in how many stages. The two calls were recovered later; the story is in chapter 12 (classification). The numbers 50,000, 40,000 and 10,000, and the 101,659 features from the bigram vectoriser, are strong diagnostic signs — a plausible alternative construction yields 100,350 features instead — but they are not proof that you selected the same rows. The proof is a matching row digest, or the exact sampling code. The clustering chapter, drawing from the same source with the same seed, ended up with a different 50,000 rows.

The test set and selection. Seven representations, nine training sizes, a hybrid, three diagnostics, all on the same 10,000 reviews. You cannot pick a method or a crossover point from these rows and then report an unbiased final score on the same rows. In real work, split train again, select on that validation slice, and use the test set only for the final evaluation.

Subsampling to 50,000 is a convenience, not a necessity: it shortens the experiment. Keeping the same 10,000 test reviews and training on all of the other 223,508 reviews leaves the ranking unchanged, and every row gains (macro-F1, single run): bag of words 73.51 → 74.50; TF-IDF 71.95 → 73.00; bigram TF-IDF 76.21 → 76.69; character TF-IDF 70.39 → 71.12; averaged word2vec 61.39 → 62.89; averaged fastText 61.36 → 62.84; doc2vec 55.13 → 61.99.

The simplest rule: always predict “positive”

Predict “positive” for every review. Never look at the text at all.

### majority baseline, in full
                 predicted neg   predicted pos   total
   actual neg          0             632          632
   actual pos          0           9,368        9,368

precision (neg)   undefined — it never predicts negative
recall    (neg)   0.0%       — it finds none of the 632
F1        (neg)   0.00       — 2TP / (2TP + FP + FN) = 0 / 632
F1        (pos)   96.74
macro-F1          48.37

The rule reaches 93.68% accuracy without using the review text at all, because 93.68% of the test set is positive — and it finds none of the 632 negative reviews.

Before comparing the methods, we use the simplest possible rule — always pick the majority class — to show which errors accuracy can hide on an imbalanced test set.

The 93.68% does not say whether the rule can find the negative reviews. The confusion matrix does: 632 negatives, all predicted positive.

Precision on the negative class is undefined, because the rule never predicts negative. F1 can still be computed from the form written in counts: 2TP / (2TP + FP + FN) = 0 / 632.

The majority rule scores F1 = 0 on the negative class, so its macro-F1 stays at 48.37. That makes visible what the 93.68% accuracy hides. The decision metric for a real system still has to be chosen separately, from the costs of its errors.

Macro-F1, from its per-class arithmetic

\[\text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}, \qquad F_1 = \frac{2\,TP}{2\,TP + FP + FN}\]

\[\text{macro-}F_1 = \frac{1}{K}\sum_{k=1}^{K} F_1^{(k)}\]

• \(F_1\) is the harmonic mean of precision and recall: when either of them drops to zero, so does \(F_1\)
• Majority rule, negative class: \(TP = 0\), \(FP = 0\), \(FN = 632\), so \(F_1^{(\text{neg})} = 0/632 = 0\); with \(F_1^{(\text{pos})} = 0.9674\), macro-F1 \(= (0 + 0.9674)/2 = 0.4837\)

Macro-F1 is the equally weighted average of the per-class F1 scores. Accuracy weights examples equally; macro-F1 weights classes equally.

TP true positives, FP false positives, FN false negatives, for one class. The usual form is F1 = 2 · precision · recall / (precision + recall); the form in counts on the slide is the same quantity.

Harmonic mean: with precision 1.00 and recall 0.00 the arithmetic mean is 0.50 and F1 is 0.00. That property keeps the misses in one class from being hidden behind a single high precision.

Mind the zero denominators. If a class is never predicted (TP + FP = 0) its precision is undefined; F1 can still be computed from the form written in counts. F1 is undefined only when TP = FP = FN = 0, that is, when the class has neither true nor predicted examples; in that case scikit-learn reads what to return from its zero_division argument.

The positive class: F1 = 2 · 0.9368 · 1.0 / (0.9368 + 1.0) = 0.9674.

On a test set where only 6.32% of the reviews are negative, calling everything “positive” gives 93.68% accuracy and catches no complaint at all. A system that triages product reviews has to set the cost of a missed complaint and of a false alarm separately. Macro-F1 is used here as a comparison metric that weights the two classes equally; it is not the only possible product metric.

Which F1 does your library report?

average=	what it computes	the majority rule scores
"binary" (the default for two classes)	the F1 of class 1 only — here the majority positive class	96.74
"micro"	pools all the counts before dividing; here it equals accuracy	93.68
"weighted"	weights each class’s F1 by its number of examples	90.62
"macro"	weights the classes equally	48.37

• Choosing macro-F1 as the headline metric and reweighting the training loss by class frequency (class_weight="balanced") are separate design decisions

In this chapter macro-F1 is the main summary score for comparing the methods, and accuracy is reported alongside it for context. Macro-F1 is not a universal answer: recall on the negative class, precision at a given false-alarm budget, or expected business cost can be more direct decision metrics.

“F1” alone is ambiguous, and in scikit-learn the average argument decides it. Micro equals accuracy for single-label classification with every class included. Always write the argument out explicitly.

Every model below uses class_weight="balanced". It sets the class weights from the inverse of the class counts in the training set. With the 37,471 positive and 2,529 negative training examples here, the class coefficient of a negative example in the loss is about 14.8 times that of a positive one. That does not mean the gradients of individual examples carry the same ratio; a gradient also depends on the features and on the predicted probability. The chapter did not measure the unweighted setup; exercise 2 asks for it.

Do not conflate the two decisions. One sets what the evaluation cares about, the other what training cares about, and picking one does not automatically bring the other. Balanced class weights are a strong default, but compare unweighted, weighted and threshold-tuned variants on a validation set.

Ask students to watch how the accuracy and macro-F1 columns diverge in the tables that follow.

Lowercasing Turkish: the default mapping is not Turkish

TR_LOWER = str.maketrans({"I": "ı", "İ": "i"})
def tr_lower(s):
    return s.translate(TR_LOWER).lower()

### the same string, two lowercasing rules
             str.lower()      tr_lower()
"IYI"        iyi              ıyı
"İYİ"        i̇yi̇             iyi

• Python’s str.lower() implements Unicode’s default case mapping and does not change with the locale: i for I, and i plus a combining dot (U+0307) for İ
• Both run without an error, and both can break tokenisation and vocabulary matching

Case conversion is language-dependent, and a language-specific lowercasing policy is part of the preprocessing and tokeniser contract.

Before the count and embedding models, we define the preprocessing decisions both of them will use.

In Turkish, dotted i/İ and dotless ı/I are different letters that distinguish different words. Lowercasing correctly spelled Turkish text needs the mapping I → ı and İ → i; that is one of the language-specific tailorings the Unicode standard defines and the default does not apply.

The default operation makes IYI into iyi, and it leaves a combining dot in İYİ.

In this experiment’s main score, however, no difference was measured between the two lowercasing routines, because the distributed data is already lowercase — we come to that after the round 1 table. Combining-dot damage that already exists needs a separate repair step.

The operation here is lowercasing; Unicode casefolding is a related but distinct operation intended for caseless matching, and it has its own Turkish tailoring.

Other languages: in Greek, a sigma at the end of a word lowercases to ς, not σ. German ß belongs to a different operation: lower() leaves it unchanged, and it is casefolding that equates ß with ss. That is why the “just lowercase it” step in an English tutorial is not portable advice; but do not treat these examples as the same problem as Turkish I/İ either.

The tokeniser, and what it throws away

TOKEN_RE = re.compile(r"[a-zçğıöşü]+")
def tokenize(s):
    return TOKEN_RE.findall(tr_lower(s))

• The pattern keeps only the listed letters: punctuation, digits and emoji all disappear; â, î, û and combining marks split a token
• !!!, 5/5 or an emoji can carry sentiment; this experiment does not model them, and the effect on the score was not measured
• tr_lower assumes correct spelling: typed on an English keyboard, IYI may well mean “İYİ”, and tr_lower then produces ıyı, a new error

What ASCII-typed text needs is the restoration of Turkish characters (deasciification); the details are in the Turkish NLP chapter.

After Turkish lowercasing, the simple token pattern extracts unbroken runs of a chosen set of lowercase letters.

hâlâ or kâr, for example, does not survive as one token. The effect of these simplifications on the classification score was not measured; on your own corpus, compare alternative patterns.

The limit of tr_lower: in text typed on an ASCII keyboard, the mapping alone cannot tell which letter the writer meant.

The training corpus, in numbers

### the training corpus, 40,000 reviews
tokens                        903,190
distinct word forms            74,317
hapax (seen exactly once)      43,579     58.6% of all forms
mean review length               22.6 tokens
median                             19
95th percentile                    56
longest                           296

• 58.6% of the vocabulary appears exactly once. In chapter 3’s Turkish Wikipedia sample the rate was 50.0%; the gap cannot be put down to typos alone
• 30,738 forms are seen at least twice, yet min_df=2 leaves 30,195 columns: min_df counts document frequency, not corpus frequency

The two corpora differ in topic, document length and preprocessing, so the gap in the hapax rate cannot be put down to typos alone; the spelling variation of user-generated text is one possible factor.

Of the 74,317 distinct forms, 43,579 occur exactly once; the other 30,738 are seen at least twice. After min_df=2 the vectoriser has 30,195 columns. The 543 forms in between were counted one by one: every one of them occurs two or more times in the corpus but only inside one document.

895 word forms behind one five-character prefix

prefix	distinct forms	examples
kulla	895	kullanışlı, kullanıyorum, kullandım
bekle	285	beklediğimden, beklediğim, beklemiyordum
çalış	247	çalışıyor, çalışma, çalışan
çıkar	209	çıkarıyor, çıkar, çıkartıyor
düşün	195	düşünüyorum, düşünmeden, düşünenlere

• In a bag-of-words model every form in the training vocabulary is its own column: the coefficient learned for kullanışlı does not transfer by itself to kullanıyorum

Not 895 inflections of one stem: inflections, derivations and spelling variants together. A five-character prefix is not stem or morpheme analysis.

Group the word forms of the training corpus by their first five characters and the variety of forms becomes visible.

Bag-of-words: count which words a document contains and forget the order they came in. With min_df=2 the forms seen in only one review never become columns at all — they are simply dropped. Other features in the same document, and regularisation, still affect how the model generalises.

The kulla group puts kullanışlı (an adjective), kullanım (a noun) and kullanıcı (a noun) side by side, and the same prefix also collects typos. So this table does not measure how many columns a real stem family is spread over. Finding that out takes morphological analysis and a separate look at the spelling variants; the analysis is classically done with finite-state transducers (Oflazer 1994). Turkish morphological analysers are covered in the Turkish NLP chapter.

The 895-form prefix group is a sign of how far the number of word-level features can grow. How this variety relates to coverage and to the classification score is examined separately below.

Round 1 — the chapter-2 methods

row	vectoriser	vocabulary threshold and tokens	columns
bag of words	CountVectorizer	min_df=2, word pattern	30,195
tf-idf	TfidfVectorizer, sublinear_tf=True	min_df=2, word pattern	30,195
tf-idf 1-2 gram	the same, with ngram_range=(1, 2)	min_df=2, word pattern	101,659
tf-idf char_wb 3-5	analyzer="char_wb", ngram_range=(3, 5)	min_df=3; token_pattern is not used	94,939

• Every vectoriser gets preprocessor=tr_lower, lowercase=False: otherwise scikit-learn calls Python’s str.lower()
• One classifier for all rows: LogisticRegression(max_iter=2000, class_weight="balanced", random_state=42)

We have chosen the evaluation metric and defined Turkish lowercasing explicitly. The combining-dot errors already present in the data are still there; we come back to them after the table. Now the four count representations of chapter 2 with the same classifier.

The word pattern is [a-zçğıöşü]+. sublinear_tf=True replaces a count c by 1 + log c.

The character row is a separate feature-extraction path: token_pattern is not used, and min_df differs. Two slides on.

Round 1 — the results

### round 1: counting
                      accuracy   macro-F1   F1(neg)   dims      fit
majority baseline        93.68      48.37      0.00      —        —
bag of words             92.14      73.51     51.30   30,195    0.7 s
tf-idf                   90.07      71.95     49.41   30,195    0.5 s
tf-idf 1-2 gram          92.57      76.21     56.47  101,659    1.0 s
tf-idf char_wb 3-5       88.63      70.39     47.14   94,939    3.3 s

• All four count methods have lower accuracy than the majority rule’s 93.68%, while scoring 22 to 28 macro-F1 points above it
• If you chose a method by accuracy alone in this setup, the majority rule would look like a strong candidate
• That is not a law: other thresholds and training settings can raise both accuracy and negative-class performance; they were not tested here

Accuracy and macro-F1 part ways over the majority rule. Among the four methods the two metrics give the same order: bag-of-words, for example, has both higher accuracy and higher macro-F1 than TF-IDF. The models evaluated here give up some of the accuracy that ignoring the minority provides, and gain negative-class recall in return.

Macro-F1 averages the F1 of every class with equal weight, so a rare class counts as much as a common one. In this experiment it is the metric that makes the majority rule’s F1 = 0 on the negative class visible.

The rows of the table share the same train/test split and the same classifier, but they differ from one another in three respects — the next slides.

The fit column here and the build the representation column in the cost table near the end do not time the same step. Re-measured five times each, vectorising the training text took 0.82–0.90 s for the bigram scheme and 2.68–2.79 s for the character scheme; fitting the classifier took 0.09–0.10 s and 0.38–0.39 s. fit covers vectorising plus fitting the classifier; the cost table covers vectorising only. The timings in the tables come from single runs; read all of them as approximate values for this machine.

The character row takes a different path

The same input through the two analyzers: "5/5 harika!! ürün çok i̇yi 👍"

analyzer	features	what happened
analyzer="word", token_pattern=[a-zçğıöşü]+	['harika', 'ürün', 'çok', 'i', 'yi']	5/5, !!, the emoji are gone; the combining dot split “i̇yi” in two
analyzer="char_wb", ngram_range=(3, 5)	[' 5/', '5/5', '/5 ', ' ha', 'har', 'ari', ...]	digits, the slash, the exclamation marks and the spaces all survive as features

• In scikit-learn token_pattern is used only when analyzer="word"; char_wb ignores it entirely and runs its own character stream
• The character scheme uses min_df=3 where the three word schemes use min_df=2: a defensible choice, but it was not swept

The score difference of the character row may reflect this preprocessing choice along with the character features.

The character row differs from the word rows not only in feature granularity but in the feature-extraction path itself. The second row of the table shows the first few of the 3-grams.

char_wb does not apply the word token pattern, and it treats digits, punctuation and spaces differently. Keep that in mind when reading the conclusions the chapter draws from character n-grams later on.

On min_df=3: character n-gram counts explode, so the choice is defensible — but it is a choice, and the chapter did not measure what the other setting would have done.

It made no difference on this corpus — write it this way anyway

• With and without lowercase=False, round 1’s table came out byte-identical: not a single uppercase character among the 37,076,865 characters of the 233,508 de-duplicated reviews
• Not the same as the corpus being clean: U+0307 (COMBINING DOT ABOVE) sits in 4,194 documents of the 50,000-review working subsample — 8.4% of it — every time after a letter i
• There tavsiye splits into tavsi + ye; deleting the mark takes the 30,195-column vocabulary down to 29,823: 595 columns exist only because of this splitting

	result	tokens under [a-zçğıöşü]+
'İYİ'.lower()	'i̇yi̇'	['i', 'yi']	split in two
'IYI'.lower()	'iyi'	['iyi']	wrongly merged

The repair is to delete the mark from the text you were given, not to lowercase it differently. Why fix the code anyway? The reader copying it will not have that corpus.

The lowercase=False line was added later, because the first version of the chapter committed exactly the bug it warns about. Both versions were then measured.

No uppercase (counted with str.isupper()): consistent with the distributed text having been lowercased beforehand; the dataset card does not document with which tool, or when. So the two paths coincide here.

Unicode’s default lowercasing produces exactly the i + U+0307 sequence for İ. The earlier processing steps of the data are not documented, so we cannot settle where the mark came from; the observation is consistent with such a conversion. The token pattern does not treat a combining mark as a word character.

The measured effect on the vocabulary: 595 columns such as alişveri and alabi exist only because of the splitting; after the repair, 223 forms that had never appeared in one piece enter the vocabulary. The clustering chapter rediscovers the same damage in its own sample as a clustering result. tr_lower does not delete a combining dot that is already in the text. Exercise 7 asks what the repair does to the scores.

The same warning applies to analyzer="char_wb". Character n-grams are sold as language-independent, but the lowercasing step in front of them is still locale-dependent.

Raw counts ahead of TF-IDF? One change at a time

Between CountVectorizer and TfidfVectorizer three things change at once: IDF weighting, sublinear term frequency (log-TF) and L2 row normalisation. Same C = 1.0:

unigram word features, min_df=2	accuracy	macro-F1	F1(neg)
raw counts (= “bag of words”)	92.14	73.51	51.30
counts + L2	88.07	69.57	45.85
log-TF + L2	88.37	70.00	46.53
counts + idf + L2	90.20	72.16	49.74
log-TF + idf, no norm	92.67	71.83	47.61
log-TF + idf + L2 (= “tf-idf”)	90.07	71.95	49.41

Adding L2 row normalisation to raw counts takes macro-F1 from 73.51 down to 69.57; IDF, added on top of it, brings the score up to 72.16. “TF-IDF fell behind here because of IDF” is not what the experiment shows.

In this setup raw counts are ahead of TF-IDF. The first explanation that comes to mind is IDF: reviews average 22.6 tokens, most words appear once in a document, so IDF may be down-weighting the frequent sentiment words — güzel, iyi, tavsiye. That is a hypothesis, and a testable one.

Sublinear term frequency replaces a count c by 1 + log c. TfidfVectorizer applies L2 row normalisation by default. Each of the three is turned on and off separately, with the same vocabulary, the same classifier and the same C = 1.0.

The drop from adding L2 normalisation is 3.94 points. The result does not support the first explanation: the original gap does not come from IDF alone. Nor does it mean normalisation is harmful in itself: its effect has to be judged together with C.

A likely explanation is that C was not tuned. L2-normalised rows have norm 1 while raw count rows grow with document length, so a fixed C = 1.0 applies a different effective regularisation to the two representations — the same caveat flagged up front for the dense family, now inside the sparse one. Next slide.

With C chosen on validation, the ordering flips

C chosen on a validation split carved out of train (32,000 fit / 8,000 val), then refit on all 40,000 documents:

                     best C on validation    test macro-F1
raw counts                  1.0                  73.51
tf-idf                     10.0                  74.71

• TF-IDF 74.71 against raw counts 73.51: over 10,000 paired bootstrap resamples the 95% interval of the macro-F1 difference runs from 0.42 to 1.99 points
• 151 reviews only TF-IDF gets right, 80 only raw counts get right

The “raw counts ahead of TF-IDF” result did not survive an equal tuning budget: it is not a fixed property of the representations, it depended on a shared, untuned C.

Give both representations the same small tuning budget: split the 40,000 training documents into 32,000 for fitting and 8,000 for validation, choose C on the validation slice, then refit with the chosen C on all 40,000 documents and measure on the test set.

The selection step is spelled out because it changes the result: the test scores in the table belong to models refitted on all 40,000 documents (the 32,000-document models score 73.65 and 73.83 on the same test set). This test set was already used earlier in the chapter, so these rows are not an independent test result opened for the first time.

How solid is the 1.2-point gap? The two models’ predictions were paired on the same 10,000 test reviews. Exact McNemar test on the error rates: p < 0.001; round 2 explains how the two tests differ. So the gap is larger than the sampling noise of this test set. The interval belongs to one training split and one test set; it does not say what would happen on another training sample.

The round 1 table is not wrong — that is genuinely what was measured — but the lesson to take from it is not “TF-IDF falls behind on short text”: a representation comparison run without an equal tuning budget may not be measuring what you wanted it to measure.

Bigrams: the largest gap inside the count family

• In this setup word unigram + bigram TF-IDF scores 4.26 macro-F1 points above unigram-only TF-IDF, and 2.70 above raw counts
• Bigrams keep local sequences such as iyi değil (“not good”) and güzel değil (“not nice”) as separate features: negation built with a separate word
• Part of Turkish negation is a suffix: beğendim (“I liked it”) and beğenmedim (“I did not like it”) are separate features at the unigram level too

This chapter did not measure how much of the 4.26 points comes from which kind of negation; saying that takes a targeted error analysis or an ablation.

Bigram: a pair of words counted as a single unit, so that “not good” can be told apart from “good”.

Not the largest gaps in the chapter — the two families are 14.8 points apart, and embeddings lead counts by 14.6 points at 250 labels. This is the largest difference measured within the count family once the label budget is fixed.

The same bigrams also pick up multi-word sentiment phrases, product names and collocations.

In pairs like tavsiye etmem (“I do not recommend”) or memnun kalmadım (“I was not satisfied”) the negation is again a suffix: etmem and kalmadım are strong negative unigrams in the coefficient table on the next slide, and the bigram adds the tavsiye or memnun context to them.

What the model learned, in words

towards POSITIVE		towards NEGATIVE
gayet (quite)	+8.04	etmiyorum (I do not [recommend])	−8.16
ederim ([I] do)	+6.94	değil (not)	−6.95
mükemmel (perfect)	+6.13	kötü (bad)	−6.63
teşekkürler (thanks)	+6.09	etmem (I would not)	−6.35
harika (wonderful)	+5.93	beğenmedim (I did not like it)	−6.21
memnunum (I am pleased)	+5.84	iade (return / refund)	−5.85
güzel (nice)	+5.48	fakat (however)	−5.52

Global associations learned on this training distribution: not universal sentiment scores, and not a complete explanation of one particular prediction.

TF-IDF + logistic regression, largest coefficients. The chapter’s list continues for two more rows: süper +4.98 and kaliteli +4.32 towards positive; kalitesiz -5.47 and kalmadım -5.33 towards negative.

In this sparse linear model every vocabulary feature has one coefficient. Sorting the coefficients together with their feature names helps you inspect which features the model has associated with each class on this data and with this preprocessing; it does not give the effect of a word in every context. The coefficients are learned together with every other feature.

Two features look as if they are about the domain rather than about language: iade (“refund”) and fakat (“however”). iade may refer to a return that happened; it may also only mention the possibility of one. fakat may be joining a positive judgement to a negative one. The coefficients show a conditional association with the training labels; which reading holds has to be checked by reading example reviews. Nobody wrote these associations down by hand; they were learned from 40,000 labelled reviews.

On a single document, the contribution of feature j to the linear decision score is x_j β_j; the intercept and the contributions of the other features are added to the total score, and the probability is computed by applying the sigmoid to that total. The global coefficient list tells you what the model learned; a single prediction is a separate question.

Keep this table in mind when the dense methods arrive: they do not give the same direct word-to-coefficient table for free.

Round 2 — the chapter-3 methods

word2vec, fastText and doc2vec, trained from scratch on the text of the 40,000 training reviews, without using the labels; no pretrained vectors and no test reviews went into that training.

model	training settings	from there to a document vector
word2vec	vector_size=100, window=5, min_count=2, sg=1, negative=5, epochs=10	the mean of the vectors of its tokens that are in the vocabulary; none at all → the zero vector
fastText	the same, with min_n=3, max_n=6, bucket=500_000	every token goes into the mean: a vector is built for out-of-vocabulary forms too, from subword vectors
doc2vec	vector_size=100, min_count=2, epochs=20, dm=0	training documents: learned directly; test documents: infer_vector at its default settings

All three matrices go to the same logistic regression as in round 1.

Now the same task with the three models of chapter 3. All three use workers=1 and seed=42; dm=0 is PV-DBOW.

Later, in the learning curve, the classifier sees fewer labels while this fixed representation, learned from the text of all 40,000 reviews, stays in use.

The single-round table, for reference: averaged word2vec 79.92 accuracy, 61.39 macro-F1, 34.64 F1(neg), 14.7 s to train the embedding; averaged fastText 79.64, 61.36, 34.79, 31.6 s; doc2vec (PV-DBOW) 71.96, 55.13, 27.66, 8.5 s. All 100 dimensions.

Both rounds, sorted by macro-F1

### both rounds, sorted by macro-F1
tf-idf 1-2 gram          92.57      76.21     56.47  101,659
bag of words             92.14      73.51     51.30   30,195
tf-idf                   90.07      71.95     49.41   30,195
tf-idf char_wb 3-5       88.63      70.39     47.14   94,939
averaged word2vec        79.92      61.39     34.64      100
averaged fastText        79.64      61.36     34.79      100
doc2vec (PV-DBOW)        71.96      55.13     27.66      100
majority baseline        93.68      48.37      0.00        —

• Unlike the sparse rows, the dense rows cannot be reproduced to the last digit: a re-run with the same seed gave 61.37 macro-F1 for word2vec, 61.29 for fastText and 55.44 for doc2vec
• Five embedding seeds kept word2vec between 61.57 and 62.06, and fastText between 60.86 and 61.68: read the decimal differences in the dense rows together with that spread

The columns are accuracy, macro-F1, F1 on the negative class and the number of dimensions.

The re-run used the same code and the same seed (42) while the chapter was being revised. The second measurement changed only the embedding seed, over five seeds.

Which row moves: later runs of the chapter’s notebook gave the same 61.37 and 61.29 every time, while doc2vec moved again (55.57, 55.58, 55.54). gensim starts infer_vector for each test document from a vector seeded with Python’s string hash, and that hash changes from process to process unless PYTHONHASHSEED is fixed.

On chapter 3’s 20 Newsgroups task the sparse method in the comparison was also ahead of the averaged static embedding. Seeing a similar ordering on two different datasets is a useful observation; but the language, the class structure, the amount of training data and the preprocessing all differ, so no general rule about document classification follows from these two results.

Which gaps are larger than the noise of this test set?

Paired bootstrap, 10,000 test reviews: macro-F1 difference and 95% interval.

tf-idf 1-2 gram     - bag of words         +2.69   [ +1.60,  +3.82]
bag of words        - tf-idf               +1.56   [ +0.53,  +2.55]
tf-idf              - tf-idf char_wb 3-5   +1.57   [ +0.59,  +2.54]
tf-idf char_wb 3-5  - averaged word2vec    +9.01   [ +7.97, +10.01]
averaged word2vec   - averaged fastText    +0.08   [ -0.60,  +0.77]
averaged fastText   - doc2vec (PV-DBOW)    +5.85   [ +5.02,  +6.69]
tf-idf 1-2 gram     - averaged word2vec   +14.83   [+13.65, +16.01]

• The gap between the families, and the adjacent gaps inside the sparse family, are larger than the sampling noise of this test set; the gap between word2vec and fastText is not
• These intervals cover test sampling only: not a change of training split, and not the spread over training seeds

Every row of the ranking comes from one split and one run. To measure the sampling uncertainty of the test set, the two models’ predictions on the same 10,000 reviews were paired and the test rows resampled 10,000 times; both models’ macro-F1 is recomputed on each resample. The dense rows are taken from the re-run.

The wobble of the learning curve between 250 and 40,000 labels is not an uncertainty interval of this kind; those points differ both in training size and in which documents were sampled.

Error rates need a separate test. McNemar’s test compares the two methods’ error rates, so it is about the accuracy column, not the macro-F1 one the ranking is sorted on. A macro-F1 difference is not a paired count of errors; it needs a paired bootstrap, as here, or a randomisation test. Dror et al. (2018) lay the protocol out in detail, and exercise 8 asks students to reproduce these intervals in their own run and extend them over the training split.

The shape of the gap: more negatives found, more false alarms

	bigram TF-IDF	averaged word2vec
macro-F1	76.21	61.39
F1 on the negative class	56.47	34.64
recall on the negative class	76.3%	84.2%
precision on the negative class	44.8%	21.8%

• The gap between the best sparse method and the best dense one is 14.8 macro-F1 points; on the negative class it is 21.83 points of F1
• Averaged word2vec actually finds more negatives; but only 21.8% of the reviews it flags as negative really are negative

These are aggregate rates over the test set. That pattern of false alarms may have as much to do with class_weight="balanced", the decision threshold and the choice of C as with how well the representation separates the classes.

Recall: of the reviews that really were negative, the share the model found. Low recall goes with a lot of misses. Precision: of the reviews the model called negative, the share that really were. Low precision goes with a lot of false alarms.

A coincidence to disarm: 21.83 is a difference in F1 points, 21.8% is a precision. The two numbers are unrelated.

In this run averaged word2vec finds 84.2% of the negative reviews — more than any sparse model. Do not read the aggregate rates as the probability that one new prediction is right. Telling the causes of the false-alarm pattern apart takes threshold and class-weight ablations.

Do not read that ordering as a general ranking

• The three dense rows are averaged static vectors trained from scratch on this corpus’s 0.9 million tokens, not pretrained ones
• The scale of the dense vectors was not equalised and C was tuned for no row at all; every row comes from one split and one run
• The measurement is precisely: static embeddings trained from scratch on 0.9M tokens fell behind, in this setup, counts extracted from the same 40,000 documents — not “static embeddings fall behind counts”
• Vectors pretrained on large Turkish corpora may give a different result; this chapter did not measure the direction or the size of the difference

What does not depend on corpus size is the averaging defect: that failure is in the pooling step, not in the vectors. That is the transferable result of this chapter; the ranking is the local one.

The row names are short; in full, the dense rows are from-scratch averaged static vectors. Recall what the C experiment did on the sparse side.

All three round-2 models were trained from scratch on the 903,190 tokens of the training reviews of this corpus only; work such as Bojanowski et al. (2017) uses corpora at Wikipedia scale. Exercise 13 is where students measure the pretrained case themselves.

The averaging defect: no amount of training data makes kalite iyi fiyat kötü and kalite kötü fiyat iyi stop having cosine 1.0000. We get to that example at the end of the lecture.

The rest of the chapter looks at how the ordering is affected, in this Turkish review setup, when the number of labels and the kinds of error change.

Do not read the number of dimensions as memory

	word TF-IDF matrix	dense document matrix
columns	30,195	100
non-zeros per row	19.8 on average	100
measured memory of the training matrix	9.7 MB	16.0 MB

• A sparse matrix stores only the non-zero values and their indices
• 100 dimensions is about 1,017 times fewer coordinates than bigram TF-IDF’s 101,659 columns and about 302 times fewer than word TF-IDF’s 30,195 — not, by themselves, ratios of memory or latency

61.39 macro-F1 from 100 numbers per document, against 76.21 from 101,659 columns. Measure the costs in your target use separately.

Where the dense methods clearly differ is the number of dimensions.

A row with 19.8 non-zeros on average holds 19.8 values, not 30,195.

Fewer coordinates can be an advantage in some dense-vector structures — a nearest-neighbour index, approximate-nearest-neighbour (ANN) search, the first layer of a network; the actual memory and latency of search have to be measured with the chosen index, the number of documents and the sparsity.

The ratio depends on which sparse model you compare against; 302 is the figure in the summary table later in the chapter. These ratios are not, by themselves, ratios of memory, latency or product architecture: in the target use, measure the costs of sparse retrieval, dense retrieval and classification separately.

If the lecture is split in two, this is the place.

The out-of-vocabulary test

### coverage of the 10,000 test reviews
test tokens                     229,055
outside the word2vec vocabulary  16,660     7.27% of tokens
distinct test word forms         32,573
outside the vocabulary           15,098     46.4% of forms
test reviews with no usable token     7

• 46.4% of the distinct word forms in the test set are not in the vocabulary word2vec learned; the document average skips such tokens
• Seven reviews have no token in the vocabulary: they get the zero vector, and all seven are misclassified
• The 7.27% is not chapter 3’s 7.33%: counted with that definition — never seen in training — the rates here are 5.10% of tokens and 33.5% of forms

We test chapter 3’s subword claim in two stages: can fastText produce a vector for a form that is not in the learned word vocabulary, and is the vector it produces useful on this classification task?

The first rate is computed over tokens, the second over distinct forms. Accessing an out-of-vocabulary form directly with w2v.wv[word] raises a KeyError. The consequence of skipping shows up not as an error but as lost information and the seven empty documents. All seven are labelled positive, and the classifier calls the zero vector negative (intercept −0.83). The rate varies from corpus to corpus and domain to domain.

The two definitions. Chapter 3’s 7.33% token rate was about forms never seen in training. The 7.27% here is the rate of test tokens not in the vocabulary of a word2vec trained with min_count=2. Counted with the same definition: 5.10% of test tokens (11,681 tokens) never occur in training; 33.5% of distinct test forms (10,921 forms) were never seen, and another 4,177 forms were seen exactly once in training and cut by min_count=2. So the like-for-like counterparts of the 7.33% and 28.99% in chapter 3’s Wikipedia sample are 5.10% and 33.5% here. The corpus, the tokenisation and the sample size all differ, so the numbers can be read side by side but do not confirm each other.

fastText builds a vector for every form

### nearest neighbours for unseen word forms
kargolanmamış  ("not having been shipped")
  word2vec   KeyError
  fastText   kargolanmış, kargolandı, kargolanma, kargolanıp, kargolu

tavsiyelerim  ("my recommendations")
  word2vec   KeyError
  fastText   tavsiyeleri, tavsiyeler, tavsiyeederim, tavsiyedir

hızlıcana  (colloquial "nice and fast")
  word2vec   KeyError
  fastText   hızlıca, hızlıda, hızlısı, hızlıydı, hızlı

• Coverage goes to 100%; the quality of the vector is not guaranteed. On average 84.5% of the n-grams of the 15,098 out-of-vocabulary test forms also occur in at least one vocabulary word

The first neighbour of kargolanmamış (“not yet shipped”) is kargolanmış (“shipped”), which carries no negative suffix: similarity of form can overshadow the negation.

For every test form this experiment’s tokeniser produces, gensim’s fastText can build a vector: the vector of a form that is not in the vocabulary is assembled from the vectors in the buckets its character n-grams fall into.

Why quality is not guaranteed: an n-gram never seen in training falls either into a bucket that was never updated (still at its random initialisation) or into a hash bucket shared with other n-grams (bucket=500_000). The measurement is over the 3-to-6-character n-grams: for 24.7% of the forms every n-gram is familiar, and for 4.8% fewer than half are.

The three forms on the slide never occur in training. fastText returns a vector and brings back neighbours related in spelling (Bojanowski et al. 2017). It does not follow from three examples that every unseen form lands in a meaningful stem family.

Two models, two kinds of neighbour

query	word2vec	fastText
kargo (“shipping”)	teslimat, gönderim, gönderi, kargoda, kargoyla, kargolama — different words, same topic	kargocu, kargolu, kargosu, kargonuz, kargom, kargomun
kalitesiz (“low quality”)	dandik, kullanışsız, dayanıksız, uyduruk, kırılacak — shoddy, unusable, flimsy, fake, about-to-break	kalitesizmiş, kalitesizliği, kalitel, kalitesi, kalitesindeki — all forms of the same stem

• Measured on 500 words drawn at random from the 2,806 vocabulary words with at least six letters and at least 20 occurrences: the share of top-5 neighbours that share the query’s first five characters is 6.5% for word2vec and 60.7% for fastText

fastText’s list contains kalitesi (“its quality”) — not a synonym: a noun form without the negation that -siz carries. The measurement counts orthographic overlap only; it does not say how appropriate the neighbours are in meaning.

Both words are in both vocabularies. On these two queries word2vec’s neighbours are related words from other stems, and fastText’s are forms close in spelling. The fastText neighbours of kargo: courier, with-shipping, its-shipping, your-shipping, my-shipping. Asked what is like kalitesiz, word2vec answers with five other ways to say a product is bad. The first eight letters of kalitesiz are kalitesi itself.

Two words are an illustration, not a measurement, hence the 500-word sample. For the first four characters the shares are 8.6% and 67.4%. So fastText returning neighbours close in spelling is a general tendency of this model. Appropriateness in meaning takes human judgement or a task-based measurement.

This is chapter 3’s stated weakness — it assumes spelling similarity implies meaning similarity — becoming visible. That the subword parameters are spent on modelling form, at the expense of information about meaning, is a plausible explanation; the measurements in this chapter do not prove it.

Coverage 53.6% → 100%. Macro-F1: 61.39 → 61.36

	word2vec	fastText
distinct test forms the model has a vector for	53.6%	100%
macro-F1 in this run	61.39	61.36

• A difference of −0.03 points: the paired bootstrap interval (−0.60 to +0.77) contains zero, and the training seed alone moves each model by about 0.5–0.8 points
• The two models differ in more than coverage, so this result cannot be put down to out-of-vocabulary coverage alone

Producing a vector does not guarantee the right meaning. Measure intrinsic properties, coverage and task performance separately.

Coverage went to 100%; macro-F1 did not change in any distinguishable way: the interval is the one from the round 2 table. Producing a vector does not guarantee that the new form is represented with the right meaning. When the score does not move, look at representation quality, pooling and the classifier settings together.

Intrinsic measurements look at properties of the representation in its own space, such as similarity or analogy; vocabulary coverage is a separate measure of usability. Extrinsic (task-based) evaluation measures success on the target job, such as classification. The two answer different questions; the choice of model is decided by the target task and the conditions of use.

One more diagnostic: being in the vocabulary does not by itself show that a representation is reliable. kalitesizmiş occurs only twice in training. Its word2vec neighbours are ayakkabıların, kaymak, yapışır, temizlenmemiş, zorlamadım (of the shoes, cream, it sticks, uncleaned, I did not force): neither the sentiment nor the form of the word. fastText brings back kalitesiz, kalitesizliği, kalitesinide, kalitesi. Past the vocabulary boundary a direct lookup raises a clear error; just inside it, a rarely seen form can return a poorly estimated vector without any error. So the 46.4% out-of-vocabulary rate may not show the whole of the representation problem. A general conclusion about the quality of rare forms takes a wider evaluation by frequency band.

Where the two models disagree

• Plain word TF-IDF and averaged fastText give the same label on 84.1% of the test documents
• On the 1,587 documents where they differ, TF-IDF is right 1,315 times and fastText 272
• McNemar’s test: \(\chi^2 = (|1315 - 272| - 1)^2 / (1315 + 272) = 684.2\) — a statement about error rates, that is, about accuracy
• The macro-F1 gap needs a paired bootstrap: 10.66 points, with a 95% interval from 9.52 to 11.77

Both results hold only for the two setups examined and this test data. The interesting question is what the 272 look like.

The table shows that the count methods score higher in this setup; it does not show why. Measuring how much each cause contributes takes separate experiments. Here we look at the predictions the two methods disagree on, at negation examples, and at the limits of the representations.

The two rows: plain word TF-IDF is the 90.07-accuracy row, averaged fastText the 79.64 one. With two classes, whenever the two disagree exactly one of them is right, so 1,315 − 272 = 1,043 documents is exactly their accuracy gap: 10.43 points. McNemar’s test uses exactly these off-diagonal counts.

The intuition of the test: if the two methods were equally accurate, their disagreements would split roughly half and half, and McNemar’s test on paired predictions (McNemar 1947) asks how far from half-and-half the split is. One degree of freedom; 684.2 is far past any conventional threshold. That is strong evidence that the two setups make errors at different rates on the same test set.

The uncertainty of the macro-F1 gap does not come out of that test; it was measured with a paired bootstrap over the test rows, with the fastText side taken from the re-run mentioned in round 2.

Six disagreements, and two properties counted over all of them

review	gold	who is right
"en guvenli alısveriş sitesi"	positive	fastText — diacritics partly dropped
"çok hizli ve temi̇z tesli̇mat"	positive	fastText — broken diacritics mid-word
"çok sessiz çalışıyor makina ."	positive	fastText — “makina”, colloquial for “makine”
"sıkıcı edebi yonu cok zayif"	negative	TF-IDF — fastText says positive
"koku ne çok kötü ne çok iyi"	negative	TF-IDF — “neither… nor…”
"cok guzel! kedi bayildi buna"	positive	TF-IDF — fastText says negative

• Share of tokens outside the TF-IDF vocabulary: 9.49% in the reviews only fastText gets right, 9.48% in those only TF-IDF gets right — it does not separate which model turns out right
• What does differ is the class: 31% of the reviews only fastText gets right are negative (6.3% in the whole test set), while 98% of those only TF-IDF gets right are positive

These six selected examples show the kinds of text on which the two models can part ways: missing diacritics, colloquial spelling, and scope relations such as ne … ne … (“neither … nor …”). Which feature decided each prediction does not follow from the list; that takes looking at decision contributions, making controlled changes to the text, or labelling and analysing all of the disagreements. Texts with missing diacritics appear in both directions, so no single explanation is enough.

The two properties were counted with the fastText model of the re-run, where the counts are 1,323 and 272. In the reviews both models get right the out-of-vocabulary share is 6.51%: out-of-vocabulary forms are more frequent among the disagreements, but they do not separate which model turns out right.

The class pattern is consistent with the averaged-embedding setup working at higher recall and lower precision.

Cosine between a review and its negation

### cosine between a review and its negation
                                                  averaged     tf-idf
                                                   fastText
ürünü çok beğendim / ürünü hiç beğenmedim            0.776       0.193
kesinlikle tavsiye ederim / … tavsiye etmem          0.919       0.474
kargo çok hızlı geldi / kargo çok yavaş geldi        0.943       0.572
fiyatına göre gayet kaliteli / … hiç kaliteli değil  0.892       0.690
bu üründen memnun kaldım / … memnun kalmadım         0.962       0.582

• On these five opposed pairs: averaged fastText 0.78 to 0.96, word TF-IDF 0.19 to 0.69. Averaged word2vec gives 0.72–0.96 on the same pairs, so the high values are not specific to fastText’s subwords

Averaging can reduce the effect of the one word that changed, through the words the two sentences share — but the whole gap cannot be put down to averaging alone.

Negation examples let us test the representations directly.

The two representations use different weights and different vectors; that is why the whole gap cannot be attributed to averaging. Nor does a high cosine mean the classifier will give the two reviews the same label — next slide.

In word unigram TF-IDF, beğendim and beğenmedim are two distinct columns; a document containing one has a zero in the other’s column. That helps keep the two forms from being confused. TF-IDF is not immune to dilution either: with L2 normalisation, two n-term documents differing in exactly one term (each term used once, all weights equal) sit at cosine (n−1)/n, which also goes to 1 as n grows. The 0.19–0.69 values on the real reviews depend on the IDF weights of the terms and on the token sequences.

Why averaging can dilute negation

\[\mathbf{d}_+ - \mathbf{d}_- = \frac{1}{5}\left(\mathbf{v}_{\text{liked}} - \mathbf{v}_{\text{disliked}}\right)\]

• A five-word review and its negation differ in one word. What matters is the ratio: one differing vector against four shared ones — at twenty words, nineteen shared ones
• Not guaranteed: it depends on the sum of the shared word vectors dominating the words that differ
• A subword model adds shared character n-grams: the cosine between beğendim and beğenmedim is 0.74 in fastText and 0.53 in word2vec
• A high cosine does not rule out separation along the decision direction: “ürünü çok beğendim” gets probability 0.82 of being positive, its negation 0.02 — at cosine 0.776

Averaging can weaken some signals; these examples do not measure its share of the total 14.82-point gap.

The two document vectors are the means of five word vectors each; four are shared (v1 to v4) and one differs. The formula does not hand you a fixed length threshold — that depends on the norms and directions of the vectors — but it shows how the dilution scales as the shared content grows.

Be careful reading the 1/5: it scales the shared words too, and cosine is invariant to a common factor, so the factor itself is not the problem. At twenty words the difference vector is scaled by 1/20. If the shared contributions cancel or point in opposing directions, the cosine need not approach 1 however long the review gets. The five examples show that a high cosine can happen.

Subwords: beğendim and beğenmedim share n-grams such as beğ, eğe, ğen. The two words also have separate whole-word components and n-grams that differ, so shared spelling alone does not settle the distance. For kaldım and kalmadım the cosines are 0.69 and 0.56. For ederim / etmem, which do not share a stem, the two models are in the same place: 0.56 and 0.57.

The decision direction: in the re-run, the probability of the negated version stays below 0.5 for all five pairs (between 0.02 and 0.33). Logistic regression can find a direction in which the small difference between the two averages is discriminative.

The analogy. Averaging word vectors is taking the mean colour of a painting. Two paintings that differ only in that one is a sunrise and the other a sunset have nearly the same mean colour, and the mean colour is the only thing your classifier gets to see. A bag of words is instead a list of which pigments appear — rougher about relationships, but it keeps the pigment that changed as a separate entry.

The crossover: fewer labels, a different ordering

### macro-F1 by labelled training size (same test set of 10,000)
    n     negatives   tf-idf   tf-idf char   word2vec   fastText
   100        6        48.37       48.37       53.51      53.87
   250       16        48.53       48.84       61.59      63.10
   500       32        49.28       50.60       61.02      61.16
 1,000       63        55.40       58.55       59.52      59.93
 2,500      158        66.61       68.36       60.91      61.19
 5,000      316        71.16       70.47       61.52      60.80
10,000      632        72.55       71.29       61.80      61.56
20,000    1,264        72.42       71.26       60.94      60.96
40,000    2,529        71.95       70.39       61.39      61.36

• Both families use the text of the 40,000 training reviews, without looking at the labels; what varies is the number of labels given to the classifier
• Every row comes from a single subsample: with retrained vectors word2vec gives 58.23 at 100 labels and 65.21 at 250

Everything so far has compared methods with 40,000 labelled reviews. In practice there are often far fewer labels.

TF-IDF learns its vocabulary and IDF values from the training text, and the embedding models learn their co-occurrence and subword parameters from it. So the question is: with access to unlabelled text held fixed, how do additional labels change the decision boundary? What TF-IDF takes from the unlabelled text is limited to document frequencies; the embeddings also take co-occurrence relations between words.

For each n, a stratified subsample of the training set is drawn (train_test_split(train_size=n, stratify=…, random_state=42)) and the classifier is trained with the labels of those rows only. The negatives column is the number of negative examples among them.

The word2vec column moves between 59.52 and 61.80 from 250 to 40,000 labels. The table does not say how much of that wobble comes from the number of labels, from which documents were selected, or from the training of the vectors; nor can it be used as a significance threshold for pairs of models. The re-run values 58.23 and 65.21 stand in place of the table’s 53.51 and 61.59. The sparse columns come out identical in the re-run.

From the figure caption: at 250 labels fastText is 14.57 points above word TF-IDF (word2vec 13.06); at 40,000 labels it is 10.59 points below.

Ten label samples per size: mean macro-F1

n	tf-idf	tf-idf 1-2 gram	fastText	tf-idf > fastText
250	48.98	48.70	60.96	0 / 10
1,000	56.57	51.68	61.96	0 / 10
1,250	59.59	53.43	62.00	2 / 10
1,500	61.87	55.63	61.90	4 / 10
1,750	63.95	57.81	61.64	10 / 10
2,500	67.52	63.42	61.55	10 / 10
5,000	70.97	71.49	61.18	10 / 10

• At the same size, changing only which documents are labelled moves the scores by a few points: at 1,000 labels TF-IDF ranges from 55.14 to 59.80 and fastText from 61.01 to 63.15

In this setup the ordering changes between 1,250 and 1,750 labels. Read any ordering that rests on a single subsample together with that spread.

To measure the uncertainty, ten separate stratified label samples were drawn at each training size, with the representations fixed; the dense columns use the vectors of the re-run. The chapter’s table also has rows for 100, 500 and 2,000 labels, a word2vec column, and the minimum and maximum next to every mean.

A few of the ranges: at 250 labels TF-IDF [48.35, 49.63] and fastText [56.55, 64.83]; at 1,500 labels TF-IDF [60.50, 65.01] and fastText [60.36, 63.52]; at 5,000 labels TF-IDF [69.72, 72.27], bigram TF-IDF [70.06, 72.88] and fastText [59.98, 62.40]. The word2vec means stay close to fastText’s: 61.69 at 250 labels, 62.15 at 1,000, 61.48 at 5,000.

The last column counts in how many of the ten samples word TF-IDF beats fastText: none at 1,000 labels, two of ten at 1,250, four at 1,500, all ten at 1,750.

Do not read the crossover range as a transferable number

• In the single sequence of subsamples the lines cross at about 1,500 labels: a value read between two measurements, not a measured point
• For word bigram TF-IDF, the chapter’s highest-scoring model, the range lies further right: it beats fastText in two of ten samples at 2,000 labels and in nine at 2,500
• All of these numbers were read off the same test set. Plot your own curve for your own task

What transfers is not the threshold but the shape: with few labels a representation learned from unlabelled text can have a head start; as labels accumulate the sparse linear model closes it.

In the first table, word TF-IDF is below the embedding methods at 1,000 labels and above them at 2,500.

With few labels bigram TF-IDF stays below unigram TF-IDF: it has 3.4 times as many columns to estimate from a handful of negatives.

To choose a method, split train again and decide on the validation slice; use the test set only to evaluate the method chosen beforehand.

The chapter’s chart shows word TF-IDF, averaged fastText and averaged word2vec against the number of labelled training reviews on a log axis: nine training sizes, the same test set throughout, a single subsample at each size. Character TF-IDF is left off for readability. In this setup the number of labels is related to the ordering; the same crossover should not be expected for other methods and settings.

The left edge: 250 labels, 16 of them negative

• Word TF-IDF scores 48.53 macro-F1 and the majority rule 48.37: this model calls exactly one of the 10,000 test reviews negative
• Averaged fastText scores 63.10 with the same 250 labels; over ten label samples the gap averages 12.0 points (between 8.2 and 16.1)
• One explanation consistent with this pattern: 16 negative examples are 16 sparse rows drawn from a 30,195-column space; the embedding offers a shared structure learned beforehand from unlabelled text

Testing this explanation means looking at the low-label coefficients and errors, and at stronger regularisation options; this chapter did not do that.

The one review the TF-IDF model calls negative really is negative. So in this run it behaves, in effect, like the majority rule; the paired bootstrap interval of the 0.16-point difference runs from 0.00 to 0.51.

The explanation in full: with few labels, the coefficients of the rare features of a word-based linear model may not be estimated reliably. The embedding’s shared structure comes from the unlabelled text of 40,000 reviews: word2vec, for example, places kalitesiz, dandik and dayanıksız close to one another, and the classifier has far fewer coefficients to estimate in 100 dimensions.

Self-supervised training extracts information from the patterns in which words occur together, without looking at labels. Those patterns can be useful to a classifier with few labels; that does not mean the model has learned all of Turkish grammar, or sentiment polarity. Which examples the learned information actually fixes on this task has to be examined separately. This is the small-scale version of the general principle behind the pretrained models in this book.

Then the curve flattens

• Averaged fastText scores 61.16 macro-F1 with 500 labels and 61.36 with 40,000; over ten samples the means between 500 and 5,000 labels stay in the 61.2–62.0 band
• Word TF-IDF keeps rising until about 10,000 labels and then drops slightly in this single run
• What was measured is not “dense representations” but this setup: averaged 100-dimensional static vectors trained from scratch

What generalises is the question itself: on this task, am I limited by labels, by the representation, or by the settings?

That closeness suggests more labels bring limited gain with the current pooling and settings. To say it is a fixed ceiling of the representation, you would have to compare normalisation, C, the decision threshold, different seeds and alternatives that keep word order.

The two curves have different shapes: in this setup the averaging-based static dense representation gains almost nothing after a few hundred labels, while the sparse linear model keeps gaining as labels arrive. Keeping the sentence that narrow matters.

What it costs

On one CPU core, 40,000 training documents; single run, not repeated.

                      build the        vectorise      training
                      representation   10,000 test    matrix
tf-idf                     0.5 s          0.11 s        9.7 MB
tf-idf 1-2 gram            0.9 s          0.17 s       15.4 MB
tf-idf char_wb 3-5         2.9 s          0.72 s      144.8 MB
word2vec + averaging      14.7 s          0.11 s       16.0 MB
fastText + averaging      31.6 s          0.27 s       16.0 MB
doc2vec                    8.5 s          1.73 s       16.0 MB

• training matrix is the memory of the training matrix only: it does not cover the model, the vocabulary or the embedding table, or peak RAM
• The timings come from single runs; run-to-run difference stayed around ten percent, so do not compare small timing differences

Choosing a method means weighing training and inference cost along with the score. This experiment measured three things: the time to build the representation, the time to vectorise the 10,000 test documents, and the memory of the training matrix.

Absent here: the size of the model artifact, the memory taken by the vocabulary and embedding table, the size of the classifier itself, peak RAM during training, 95th-percentile (p95) latency and the cost of deploying an index.

The ten percent comes from re-measuring the round 1 vectorising times five times each (0.82–0.90 s for the bigram scheme, for example).

The bag-of-words matrix has the same sparsity pattern as the tf-idf one and therefore the same size; only the tf-idf row appears in the table.

Two entries deserve comment

• Character n-grams: with the full training set, character TF-IDF scored 70.39 macro-F1 from a 144.8 MB matrix (301 non-zeros per document); word TF-IDF scored 71.95 from 9.7 MB
• At the 1,000-label point, though, the character features were 3.2 points ahead of word TF-IDF: re-evaluate the memory–score balance against the number of labels and the spelling noise in the text
• doc2vec must run gradient descent to vectorise a new document: 1.73 seconds for 10,000 reviews, against 0.11 for averaging — a 16-fold difference, and unlike training that cost recurs on every new document
• In this setup doc2vec’s macro-F1, 55.13, is the lowest of the dense rows examined

No measurement here supports making doc2vec the default on this short, noisy review task. Look at the absolute number too: 0.17 milliseconds per document is still small for most services.

For character n-grams the balance of memory and score depends on the condition.

On doc2vec’s cost: the decision belongs to your latency budget.

In chapter 3’s 20 Newsgroups comparison doc2vec was ahead of averaging; here it is 6.3 points behind. The chapter did not test whether the difference between the two results comes from document length, language or the amount of data.

The hybrid, and an honest negative result

setup	macro-F1
averaged fastText alone	61.36
char TF-IDF alone	70.39
char TF-IDF ⊕ averaged fastText, plain concatenation (95,039 dimensions)	71.65
char TF-IDF alone, with the same tuning budget (C = 30)	73.46
hybrid with the dense block L2-normalised and scaled (\(\alpha = 2\), C = 10)	74.69
word bigram TF-IDF, untuned	76.21

• Adjusting the block scales puts the hybrid 3.02 points above the plain concatenation (2.00 to 4.06) — yet untuned bigram TF-IDF is still 1.52 points ahead of the tuned hybrid (0.20 to 2.80)

Reporting the hybrid as a success by comparing it only against its own parts would be exactly the kind of selective baseline this chapter is arguing against.

The obvious move is to concatenate: character TF-IDF for spelling robustness, averaged fastText for semantics, one classifier over both.

Scale first. The two blocks arrive on different scales: char_wb TF-IDF rows are L2-normalised (norm 1; about 301 non-zeros per document, averaging 0.05), while the averaged fastText rows are not normalised and have a mean norm of 2.5 (the mean absolute value of their components is 0.2). Handing both to a single L2-penalised C = 1.0 means the relative weight of the two blocks is chosen by the default rather than by us. So both versions were measured.

The plain concatenation: accuracy 89.36, macro-F1 71.65, F1(neg) 49.24. With the paired bootstrap intervals of the re-run: hybrid − char TF-IDF is +1.28 points (0.65 to 1.92), and bigram − hybrid is +4.54 points (3.42 to 5.63). So the plain hybrid is ahead of its own parts and behind the bigrams.

The tuned version: the dense block was L2-normalised row by row and multiplied by a coefficient α; α (0.25, 0.5, 1, 2, 4) and C (1, 3, 10, 30) were chosen on the 32,000/8,000 validation split and the model refitted on all 40,000 documents. The hybrid’s gain over the tuned char TF-IDF is 1.23 points (0.33 to 2.15). Reading the plain concatenation as “the hybrid does not help” would have been wrong — but in this setup adjusting the scales is not enough to pass bigram TF-IDF.

On cost, the rough sum of the representation-building times is 2.9 + 31.6 = 34.5 seconds, against the bigram model’s 0.9; end-to-end training and serving cost was not measured. Concatenation is a real technique, and here it gives a measurable gain over its components.

Choosing, on purpose: the sparse candidates

	measured in this experiment	choose it when
Bag of words	At a fixed C = 1.0 it scored above word TF-IDF (73.51 against 71.95). Does not normalise for document length by itself	One of the first baselines for short-document classification; choose its settings on a validation set
TF-IDF (words)	With C tuned on validation it scored 74.71 and moved ahead of raw counts; fell behind under a shared, untuned C	A general-purpose first sparse baseline; choose the vectoriser and C inside the train/validation boundary
TF-IDF (word unigrams + bigrams)	Highest macro-F1 in this chapter: 76.21. With few labels it stayed below unigram TF-IDF	A few thousand labels and a short-text task where local patterns may matter — sentiment, intent, triage
TF-IDF (char n-grams)	3.2 points ahead of word TF-IDF at 1,000 labels; 144.8 MB training matrix; lower macro-F1 at full data (70.39)	Spelling is variable, and the memory and score you measure are acceptable

The seven representations differ in their measured scores, memory and coverage. The chapter’s table summarises strengths, weaknesses and when to choose each, together with the conditions of this experiment; this slide and the next condense it.

Bag of words: one of the fastest representations to build in this experiment; word features map onto coefficients. It applies no informativeness weighting at the representation stage; rare forms are separate features; coefficients alone do not show causes.

TF-IDF (words): the gap after tuning is 1.20, with a paired bootstrap interval of 0.42–1.99. An unseen word form has no column; the model relies on the familiar words in the same document. Turkish inflections spread over separate columns.

Unigrams + bigrams: 101,659 columns, 3.4× the unigram model; 51.7 against 56.6 on average at 1,000 labels. Which sequences contribute how much to the score needs separate analysis; bigrams unseen in training are not represented.

Character n-grams: a misspelt word still shares most of its n-grams with the correct one. Less dependent on a word tokeniser — but lowercasing, Unicode normalisation and broken characters are still language-specific decisions. 301 non-zeros per document on average; unseen n-grams are discarded; n-gram coefficients are harder to interpret than word coefficients.

Choosing, on purpose: the dense candidates

	measured in this experiment	choose it when
Averaged word2vec	61.39 macro-F1 from a 100-dimensional document vector. Ahead of word TF-IDF in all ten label samples up to 1,000 labels. 46.4% of the distinct test forms are not in the vocabulary	Labels are scarce, or you need dense-vector search; as a candidate to test on the target data
Averaged fastText	Built a vector for every test form. Highest score at 250 labels on the single-subsample curve (63.10). Its macro-F1 could not be distinguished from word2vec’s	Spelling and form variety are high; with task performance and memory measured
doc2vec (PV-DBOW)	Lowest score in this run, 55.13. Inferring a vector for a new document needs extra optimisation (16× averaging)	A separate experiment shows that learning the document vector directly beats averaging on your data

The choice of method should follow the task and the data budget.

Averaged word2vec: 302× fewer coordinates than word TF-IDF; trained on unlabelled text. The order changes between 1,250 and 1,750 labels, and that range does not transfer. The average skips out-of-vocabulary forms; the vector of a rarely seen in-vocabulary form may be poorly estimated; averaging loses word order and some local distinctions. A dense matrix need not be smaller than a sparse one (16.0 MB against 9.7 MB).

Averaged fastText: similar spelling can also bring forms of opposite sentiment together (kalitesiz → kalitesi); 60.7% of top-5 neighbours start with the same five letters as the query (6.5% for word2vec). 61.36 against 61.39. Training took about twice as long as word2vec; the memory of the subword table was not measured.

doc2vec: learns a separate vector for each training document; there is no averaging step. 1.73 s to infer vectors for 10,000 reviews. This variant was not shown to represent word order. Extra hyperparameters.

Choosing a representation, in order

1. Write down the task, the costs of a false alarm and of a miss, and the main metric
2. Fix the data version, the train/validation/test splits and the unlabelled text
3. Keep preprocessing, the vocabulary and IDF away from the test labels
4. Build the cheap baselines: word counts / tf-idf, word n-grams, character n-grams
5. With enough unlabelled text, or suitable pretrained vectors, add an embedding candidate
6. Choose each candidate’s settings with the same validation budget
7. Decide on the validation set: score + interval · latency · memory · explainability need
8. Use the locked test set once, for the chosen solution

A list of questions, not a list of thresholds. The numbers this chapter observed are useful as starting expectations at step 5, not as rules at step 1.

The decision procedure, compressed to something students can apply before writing any code.

Details the slide leaves out. Step 3: document which text is used where. Step 4: character n-grams especially if the spelling is noisy. Step 6: look at the OOV rate and at error examples — order and negation, or spelling and OOV?

The numbers meant in the takeaway: the change of order between 1,250 and 1,750 labels, the counts-versus-tf-idf ordering on short documents, the timings. The same goes for “sparse if you need an explanation” and “dense if you need similarity search over millions of vectors”: both are strong first candidates, not absolutes. Sparse lexical retrieval can be entirely standard over millions of documents, and dense models can be made explainable.

The limit both share: word order

### two reviews, same words, opposite meaning
a = "kalite iyi fiyat kötü"      (quality good, price bad)
b = "kalite kötü fiyat iyi"      (quality bad, price good)

cosine, averaged fastText   1.0000
cosine, tf-idf              1.0000

• Unigram TF-IDF and a plain average of word vectors both produce the same vector for the two reviews: addition does not depend on the order of its terms
• A classifier that takes only this vector as input cannot tell the two reviews apart — not with more labels, and not with a stronger classifier

Document-level sentiment can tolerate this loss better than aspect-level tasks; for “which component is the customer complaining about” it directly blocks the task.

Closing the chapter on “the count methods scored higher” would leave something out, because even the highest-scoring model still misses something. A unigram bag of words and a plain average of word vectors reduce some changes of order to the same document representation.

The two reviews contain the same tokens the same number of times. Under the same tokenisation, the same vocabulary and the same weights the vectors are equal, which is why the cosine is not approximately but exactly 1 — in practice up to floating-point rounding, far below the fourth decimal. A model that also sees the order, the bigrams or the original text can tell the reviews apart.

That most reviews are entirely positive or entirely negative is a reasonable expectation, but the chapter did not measure it.

Label the 1.0000 precisely: the two rows are unigram TF-IDF and averaged embeddings. Word bigrams are a different matter — next slide.

Bigrams keep local order, not long range

• The bigrams of the two reviews share nothing; four of the six are in the vocabulary, and the cosine of the two reviews drops from 1.0000 to 0.3247
• Separate representations do not mean a different, or a correct, label: this model calls both reviews negative (probabilities of being positive 0.26 and 0.15)
• In the pump the engineer replaced last week failed, five words separate pump from failed: covering that link with one contiguous n-gram takes a seven-token sequence

For longer relations the representation has to be a function of the sequence: sequence-aware encoders, where Part II starts.

The bigrams: kalite iyi, iyi fiyat, fiyat kötü against kalite kötü, kötü fiyat, fiyat iyi. fiyat kötü and kötü fiyat did not pass the min_df=2 threshold. If the threshold removes all the bigrams of a pair, the order information is lost for that pair too.

A bag of bigrams keeps local order; it does not directly encode dependencies further apart. Patterns as long as seven tokens recur very rarely in small datasets. The number of possible n-gram types grows very fast with n; the vocabulary actually stored is limited to the patterns seen and by the thresholds.

A function of the sequence: something that reads kalite, then iyi, and produces a different state than if it had read kalite, then kötü. Language models are one important way of learning such encoders from large amounts of unlabelled text — but a sequence-aware encoder does not have to be a language model. The mechanism that makes that state depend on every other word at once is attention, the subject of the chapter after the language-models chapter.

Two useful properties to look for in the models that follow: the sensitivity of sparse representations to decisive local forms, and the ability of embeddings to build structure shared across word forms. Which of these gains actually materialise still has to be measured on the target task.

Intuition check 1 of 4

Your client has 400 labelled support tickets in Turkish and 2 million unlabelled ones. Which representation?

• Not one model name, but a comparison plan: first settle the label of a ticket, the business metric and the test design
• The unlabelled tickets are a resource for both families. TF-IDF is limited to vocabulary and document frequency; an embedding model also extracts co-occurrence structure
• Here the averaged embeddings were about 11 points ahead of word TF-IDF at 500 labels (62.0 against 50.7): an expectation, not a threshold for this domain
• Compare word n-gram, character n-gram and embedding candidates with the same validation budget; decide on further labelling from your own learning curve

Give the room a minute, then reveal the answer (one click).

The unlabelled text is not “free”: training time and data access are costs. You can train word2vec or fastText on it, or adapt pretrained vectors to the domain.

The 62.0 against 50.7 is the mean over ten label samples. Support tickets are another domain and the label is another label. Try fastText in particular, because the tickets are Turkish text written by users: here 46.4% of the distinct test forms were outside the word2vec vocabulary; measure the rate on your own data.

Do not treat the change of order at 1,250–1,750 labels in the product-review experiment as a threshold for this domain.

Intuition check 2 of 4

A colleague reports 93% accuracy and wants to ship. What do you ask for?

• First the accuracy of the majority rule on the same test set, and the confusion matrix
• 93% is below the 93.68% you get on this test set by giving every input the majority class. That does not show the model is useless: this chapter’s bigram TF-IDF finds 76.3% of the negatives at 92.57% accuracy
• So ask for the recall and precision of the negative class, and for the workload its false alarms create at the chosen decision threshold
• The general form: what does the trivial baseline score? — ask it before you look at any reported score

A model with lower accuracy may be finding most of the negatives.

A decision to ship also needs the cost of a missed complaint and of a false alarm, and the class distribution of the target traffic.

Intuition check 3 of 4

fastText raised form coverage to 100%. Why did the score not go up?

• word2vec scored 61.39 macro-F1 and fastText 61.36; the difference cannot be distinguished (paired bootstrap interval −0.60 to +0.77)
• Two models trained by different methods: the comparison does not isolate coverage
• Candidate explanations: fastText’s neighbours shift towards forms close in spelling; in averaged vectors the cosine between a sentence and its negation is high for both models
• An improvement in one property of a representation shows up in the task score only if that property was what limited the task

Coverage went from 53.6% of the distinct test forms to 100%. The training seed alone moves either model by 0.5–0.8 points. The causal effect of coverage cannot be separated out of the score difference.

The candidates in numbers: 60.7% of fastText’s top-5 neighbours start with the same five letters as the query (6.5% for word2vec), and kalitesiz (“low quality”) retrieves kalitesi (“its quality”). The negation cosines: 0.78–0.96 for fastText, 0.72–0.96 for word2vec. Confirming them takes an ablation that removes and restores the out-of-vocabulary tokens within one model, an equal tuning budget and a paired error analysis.

The transferable lesson: measure what limits you before you set out to fix something.

Intuition check 4 of 4

Would 300-dimensional vectors, or pretrained ones, have changed the verdict?

• We do not know; these conditions were not measured
• Larger vectors, or pretrained Turkish ones, may change the neighbourhoods, the cosines of the negation pairs and the task score; neither the direction nor the size can be read off this chapter
• The one thing that does not change is the mathematical one: the plain average of two sentences that contain the same words the same number of times is the same whatever fixed word vectors are used — the 1.0000 of the word-order example stays
• The way to settle it is exercise 13

Pretrained Turkish vectors: Grave et al. (2018).

The 0.78–0.96 negation cosines are different from the 1.0000: those sentences differ in which word appears, and other vectors could put beğendim and beğenmedim further apart.

Exercise 13: rerun the chapter’s experiment with 300-dimensional pretrained vectors, and determine the ordering, and the range of labels over which the order changes, from the new experiment.

Interview-style questions for discussion

Three of the chapter’s six questions. Almost every answer rests on something this chapter measured.

1. Embeddings or TF-IDF for text classification?
2. Why can averaging word vectors fall behind counting words?
3. How would you compare two representations beyond their headline scores?

First name the axis that decides the task: the number of labels, the variety of word forms, the need for order, or cost. Then give the relevant measurement, and say which new data or experiment would change the conclusion.

Suggested format: pairs, one question per pair, two minutes; then compare with the chapter’s answers.

Question 1, in three steps. The first question: how many labelled examples, what unlabelled corpus, what kind of task (document-level or aspect-level; document length; spelling/OOV noise). The number: at 250 labels averaged fastText scored 63.10 macro-F1 against word TF-IDF’s 48.53; at 40,000 labels that same TF-IDF scored 71.95 against fastText’s 61.36; over ten label samples the order changes between 1,250 and 1,750 labels. The limit: the range depends on which sparse model you put on the curve — bigram TF-IDF passes fastText only between 2,000 and 2,500 labels.

Question 2. Shared content can weaken the directional contribution of the one word that makes the difference: cosine 0.78 to 0.96 under averaged fastText and 0.19 to 0.69 under TF-IDF on five opposed pairs. A high cosine does not rule out a correct classification. Weighted averaging with the first principal component removed (Arora et al. 2017) is a cheap option to try; it does not bring word order back either.

Question 3. Cross-tabulate the paired predictions: agreement on 84.1% of the test set; TF-IDF alone right 1,315 times and fastText 272 — the two counts a McNemar test needs. Then read examples, but do not draw conclusions about all 272 from six selected reviews. For two models with close scores, add a paired bootstrap interval and a systematic labelling of the errors.

The other three questions, as homework: which candidates the variety of Turkish word forms brings up; when a difference in cost matters (on this machine, 0.9 s and 15.4 MB for bigram TF-IDF against 31.6 s and 16.0 MB for fastText — not total model memory or serving latency); and why this comparison is useful if a larger encoder will be tried as well.

What to remember

• Before any reported score, ask what the trivial baseline scores: here the majority rule reaches 93.68% accuracy and 48.37 macro-F1
• A representation comparison run without an equal tuning budget may not be measuring what you wanted it to measure: “raw counts ahead of TF-IDF” depended on a shared, untuned C
• The local result: in this setup, counts extracted from 40,000 documents were ahead of static embeddings trained from scratch on 0.9M tokens
• Coverage and the task score are separate measurements: fastText took coverage to 100%, and macro-F1 did not change in any distinguishable way
• With few labels the averaged embeddings were ahead; in this setup the ordering changes between 1,250 and 1,750 labels. The shape transfers, the threshold does not
• The transferable result: a unigram bag and a plain average of word vectors give the same vector to the same words in a different order

A summary of the lecture in six sentences, each tied to a measurement shown earlier.

The diagnostic habits come from this chapter and apply to more complex models unchanged: the negation-cosine probe, the disagreement cross-tabulation, the distinction between coverage and score, and the cost check.

A pointer forward from the chapter’s interview questions: in chapter 12’s own measurement the fine-tuned encoder falls behind the methods of this chapter below roughly 500 labels, and the order changes between 500 and 1,000 labels. Students should compare that result with their own data, label budget and tuning conditions.

Lab and exercises

• The chapter has 13 exercises, each labelled Basic, Applied, Advanced or Research
• If you do only one, do exercise 8: reproduce the paired bootstrap table from round 2, then repeat the experiment with five different train/test splits
• Exercise 2: drop class_weight="balanced" and rerun every model
• Exercise 13: replace round 2’s from-scratch vectors with 300-dimensional pretrained Turkish vectors and redraw the learning curve

The chapter: A Representation Bake-Off. The lab follows the chapter notebook, en/chapters/04-bake-off/04-bake-off.ipynb in the notebooks repository.

Basic consolidates a concept, Applied makes you measure something on your own data, Advanced goes past the chapter, and Research is open-ended.

Exercise 8 in full: for model pairs fixed in advance, resample the same 10,000 test documents in paired fashion and compute the 95% interval of the macro-F1 difference; for the accuracy difference use McNemar’s test on the paired right/wrong outcomes. When many pairs are tested in one table, state the multiple comparisons and the fact that the test set was already examined earlier in the chapter.

Exercise 7: write a normalisation rule for the combining-dot damage and rerun the round-1 table. In this subsample the mark always follows an i; in another corpus U+0307 may be a genuine combining mark. The chapter found the vocabulary going from 30,195 to 29,823 columns.

The other exercises: the crossover under different min_df thresholds; weighted averaging with SIF (Arora et al. 2017); twenty negation pairs from your own domain; the conditions of character n-grams; a new aspect-level task; choosing a decision threshold; rerunning the ablation ladder with C tuned; an analyzer audit; and extending the repeated learning curves.

Papers named in the chapter for going deeper: Joulin et al. 2017 (the supervised fastText classifier — not the same thing as the unsupervised subword vectors used in round 2), Bojanowski et al. 2017, Le and Mikolov 2014, Arora et al. 2017 and Dror et al. 2018.