16  Turkish and Morphologically Rich Languages

Chapter status: drafted. The measurements new to this chapter were run on the WMT16/17/18 Turkish–English test sets — 9,007 professionally translated sentence pairs, which give the same content in two languages and so allow comparisons the usual monolingual corpora cannot. Where a number was measured in an earlier chapter it is cited rather than repeated. Software: sacrebleu 2.6.0, snowballstemmer 3.1.1, transformers 5.15.1, PyTorch 2.13.0 on Python 3.14.5.

Learning objectives. After this chapter you can:

16.1 What agglutination actually costs

Turkish is agglutinative: it builds words by stacking suffixes onto a stem, each one adding a piece of grammar, in an order fixed by rule and with vowels that harmonise with the stem — vowel harmony, which is why the same suffix surfaces as -ler or -lar.

Agglutinative — a language that builds words by concatenating suffixes that each carry one piece of grammar, rather than by fusing them or by using separate words. Turkish, Finnish and Hungarian; the reason a Turkish vocabulary never closes. ↩︎

Vowel harmony — the rule that a suffix’s vowels agree with the stem’s last vowel in backness, and — for four-way suffixes — in rounding as well, so one suffix has two or four surface forms (evler but kitaplar; evim, kitabım, okulum, gözüm). It multiplies the number of distinct strings a tokenizer must account for. ↩︎

Ev is house; evler houses; evlerimiz our houses; evlerimizden from our houses. One Turkish word, four morphemes — the stem plus plural, possessive and case — where English needs three words, one of them the preposition from, to say the same thing.

Everyone knows this. What it costs is rarely measured, and the measurement needs a corpus where both languages say the same thing — which is exactly what a professional translation is.

Vocabulary growth, on identical content

So take one. Nine thousand news sentences, translated by professionals, and count the distinct word forms on each side:

WMT16/17/18 tr-en, 9,007 aligned sentence pairs

  Turkish   141,412 tokens   32,866 distinct forms
  English   182,429 tokens   16,652 distinct forms

Turkish produces twice the vocabulary from 78% of the tokens. Same content, same facts, same news stories. And the ratio is not fixed — it grows with the corpus:

   tokens seen     TR forms    EN forms    ratio
         5,000        2,471       1,655     1.49
        10,000        4,192       2,575     1.63
        25,000        9,342       5,016     1.86
        50,000       15,963       7,827     2.04
       100,000       25,664      11,848     2.17
       141,412       32,866      14,251     2.31

Those six rows are two curves. Both follow Heaps’ law (Section 2.3.2) and both bend; they differ in how fast:

Two vocabularies over the same content, and the gap that keeps widening
DISTINCT WORD FORMS AGAINST TOKENS READ — THE SAME 9,007 SENTENCE PAIRS, COUNTED ON BOTH SIDES 010k 20k30k 5k25k 50k100k 141k 32,866 14,251 Turkish English What the two shapes mean Both curves follow Heaps' law and both bend; their exponents: 0.78 vs 0.65. English flattens faster, so the multiple between them keeps widening. THE SAME THING AS A MULTIPLE — TURKISH FORMS DIVIDED BY ENGLISH FORMS, AT EACH POINT ABOVE 5k10k 25k50k 100k141k 1.491.63 1.862.04 2.172.31

Both curves are counted at equal token counts, so the gap is not an artefact of one language needing more words to say the same thing. Both curves bend, but English bends faster: the Heaps exponent fitted on the log-log slope is about 0.78 for Turkish and about 0.65 for English, so the multiple between them is still growing when the corpus runs out.

That widening is the important part. Neither language saturates — even the English side adds 2,401 new forms between 141k and 182k tokens — but English slows faster; Turkish keeps producing forms it has not used before at a higher rate, because the forms are generated by rule rather than drawn from a list. A vocabulary is a list, and productive morphology does not fit in one.

The tail shows the same thing:

  forms occurring exactly once     Turkish   58.6%      English   44.1%

A form that occurs exactly once is called a hapax (plural hapax legomena). Chapter 4 measured 58.6% hapax on a completely different Turkish corpus — product reviews rather than news, and six times larger — and Chapter 3 50.0% on Turkish Wikipedia. Treat the exact match as a coincidence: the hapax share depends on corpus size and on how noisy the text is. What holds across all three is that Turkish sits at or above half, well over the 44.1% of English news above.

One stem, many words

Vocabulary size is one view of the cost. The other is what sits behind it — how many of those distinct forms are the same word wearing different suffixes. Stemming the corpus and counting the forms each stem produced:

  32,866 distinct forms  ->  17,891 stems       mean 1.84 forms per stem

  arkadaş   39 forms   arkadaş, arkadaşı, arkadaşım, arkadaşın, arkadaştı, ...
  ülke      35 forms   ülke, ülkem, ülkede, ülkeyi, ülkeye, ülkeden, ülkeler, ...
  el        34 forms   el, eli, ele, elim, elde, elin, elimi, ...
  çocuk     31 forms   çocuk, çocuğu, çocuğa, çocukla, çocuğun, çocukta, ...

The mean of 1.84 understates the effect, for two reasons. Most stems in a corpus this small appear exactly once, which drags the mean towards 1 regardless of the language. And the stemmer under-merges wherever the stem changes shape in a way its rules do not undo (rule 4 below), so some forms of one word are counted under two stems. The informative row is the top: thirty-nine forms of one everyday noun in 141,000 tokens, and the list is nowhere near exhausted. Chapter 4, on a larger corpus, found 895 forms sharing the stem kulla.

The rules behind the forms

Four rules generate almost every Turkish surface form, and all four are already at work in the list above. Naming them matters here, because the fourth is exactly where the stemmers later in this chapter go wrong.

1. Suffix order is fixed. In the standard productive pattern a Turkish noun is STEM (+PLURAL) (+POSSESSIVE) (+CASE) (+CLITIC), and that order does not vary: ev-ler-imiz-den, never *ev-den-ler-imiz. (Derivational suffixes and compounds complicate the picture — a derived stem can itself carry inflection — but the inflectional slots keep this sequence. One notable exception is -ki, which renominalises an inflected noun and restarts the plural–possessive–case cycle: ev-de-ki-ler-in, even evdekilerinki.)

2. Backness harmony (büyük ünlü uyumu) gives two-way suffixes: the plural -lAr surfaces as -ler after a front vowel (ev-ler) and -lar after a back one (kitap-lar).

3. Rounding harmony (küçük ünlü uyumu), layered on backness harmony, gives four-way suffixes: the first-person possessive -Im surfaces as -im (ev-im), -ım (kitab-ım), -um (okul-um) and -üm (göz-üm). This is the rule that multiplies forms fastest, and it is the one the usual one-sentence summary of vowel harmony leaves out.

4. Sounds change on both sides of the seam. In many words the last consonant of the stem becomes voiced before a vowel (kitap → kitab-ı, çocuk → çocuğ-u; but top → top-u), the first consonant of the suffix loses its voicing after a voiceless consonant (arkadaş + -(y)DI → arkadaş-tı), a buffer consonant is inserted between two vowels (ülke + -I → ülke-y-i), and in some words the last vowel of the stem drops out (ağız → ağz-ı, burun → burn-u). None of these is a different suffix: it is the same stem, or the same suffix, changing shape.

Rule 4 is the one that breaks stemmers. The stem is not constant, so a rule-based stemmer that only strips suffixes has no way to merge kitap with kitabı, and every such failure is a document a query will not find. The Snowball Turkish stemmer used in this chapter patches the commonest case: after stripping, it turns a final b, c, d or ğ back into p, ç, t or k, which is why çocuğu sits under çocuk in the list above and kitabı stems to kitap. The other changes it does not undo: ağzı and burnu stay apart from ağız and burun, and rengi from renk.

Where this shows up

Agglutination is not one problem; it is the same problem arriving in four places. The last row comes from a second Turkish peculiarity independent of agglutination — the alphabet (agglutinative Finnish does not have it) — but it breaks the same pipelines. This book has now measured each of them:

Five Turkish traps: four from agglutination, one from the alphabet
where what happens measured in
Tokenization with an English-centric tokenizer, more tokens for the same content, so more cost and less effective context Chapter 8
Vocabulary twice the types, a longer tail, more out-of-vocabulary here
Lexical retrieval a query form that differs from the document form matches nothing Section 16.4
Word-level metrics fewer tokens per sentence, so each error costs more Chapter 15
Case folding one character breaks the whole pipeline Section 16.2

What we will look at. First normalization, where two steps are not preferences but correctness bugs, and both survive code review because the code looks right. Then morphological analysis, which subword tokenizers mostly replaced, and the four jobs it is still the right tool for. Then search, where one preprocessing step moves recall@1 from 28% to 95%. Then the pattern this book keeps walking into — a standard tool, silently wrong outside English, caught by a control rather than by reading its source. Then a short map of Turkish corpora, models and benchmarks. And finally monolingual against multilingual, settled by running both, with a result I did not expect.

16.2 Turkish normalization, and the two bugs that survive review

Normalization in English is a preprocessing decision. In Turkish two of its steps are correctness bugs, and both are invisible in code review because the code looks right.

Bug one: str.lower() is wrong in both directions

Turkish has two letters where English has one — the dotted and dotless i. Dotted İ/i and dotless I/ı are different letters with different sounds, and the case mapping between them is not the ASCII one:

        English rules          Turkish rules
  I  →  i                      ı
  İ  →  i + combining dot      i

Dotted and dotless i — Turkish has two i letters where English has one: dotted i/İ and dotless ı/I. They are distinct letters, so case conversion between them does not follow the ASCII rule, and str.lower() gets both capitals wrong: I becomes i instead of ı, and İ becomes i plus a combining dot. ↩︎

Python’s str.lower() and JavaScript’s toLowerCase() apply the locale-independent Unicode default mapping — the “English rules” above — which is wrong for Turkish. Java’s toLowerCase() and .NET’s ToLower() use the machine’s locale by default, so the same code gives one result on a Turkish-locale laptop and another on the server (the classic “Turkish I” bug: "TITLE" becomes "tıtle"). Both are dangerous. On the WMT16 Turkish source:

  tokens in the WMT16 Turkish source                      46,243
  tokens containing I or İ                                   438   (0.95%)
  of those, where str.lower() differs from Turkish rules     438   (100%, by construction)

    'İtiraf'      str.lower() 'i̇tiraf'       Turkish 'itiraf'
    'İngilizce'   str.lower() 'i̇ngilizce'    Turkish 'ingilizce'
    'İşlerin'     str.lower() 'i̇şlerin'      Turkish 'işlerin'

The 100% is not a finding. The two case mappings differ for both letters — I goes to ı in Turkish and i in English, İ goes to i in Turkish and i plus a combining dot in English — so every token containing either letter must come out different. It is arithmetic, and reporting it as a measurement would be the kind of thing this book keeps telling you to check for.

The finding is the 0.95%. One token in a hundred is silently corrupted, and they are not a random one in a hundred: a capital dotted İ appears disproportionately in proper nouns and in every sentence-initial word starting with i, which are exactly the tokens a retrieval or NER system cares about most.

Lowercase it yourself — English rules against Turkish rules

Your browser has the correct mapping built in — toLocaleLowerCase("tr") — and so do Java, .NET and ICU; Python’s standard library does not (see below). The bug is never that the fix is unavailable; it is that the default is wrong and silent.

Bug two: the result is not even one word any more

The first bug is a wrong string. The second is worse, and it is the one Chapter 11 found in real data:

  'İSTANBUL'.lower()  =  'i̇stanbul'

    code points  U+0069 U+0307 U+0073 U+0074 U+0061 U+006E U+0062 U+0075 U+006C
                   i     ̇      s      t      a      n      b      u      l
    categories   Ll     Mn     Ll     Ll     Ll     Ll     Ll     Ll     Ll

  re.findall(r"\w+", 'İSTANBUL'.lower())   →   ['i', 'stanbul']

U+0307 COMBINING DOT ABOVE is a nonspacing mark, and Python’s re does not count it as a word character, so a tokenizer built on \w+ splits the word in two at that point. Check your own engine before assuming it behaves the same way: \w is defined per implementation, and some engines fold nonspacing marks into it while others do not. İstanbul becomes i plus stanbul, and no amount of downstream modelling recovers from it — the model never sees the word.

And it does not go away when you normalize:

  unicodedata.normalize("NFC",  'İSTANBUL'.lower())   →   ['i', 'stanbul']
  'TAVSİYE'.lower()                                   →   ['tavsi', 'ye']

NFC does not recombine, because NFC only composes a sequence into a single code point when such a code point exists, and none exists for i + combining dot. There is no fix at the level of Unicode normalization; the fix is a library that applies Unicode’s language-specific casing rules (the Turkish and Azeri conditions in SpecialCasing.txt), such as ICU.

There is no standard-library fix either, and the folk answer is wrong: str.casefold() is not locale-aware, and on TAVSİYE it produces the identical eight codepoints that lower() does, combining dot included. A reader who reaches for it because “casefold is the more correct lowercasing” gets exactly the bug they were trying to avoid. Python has no Turkish casing at all; ICU does (PyICU, icu.UnicodeString(s).toLower(icu.Locale("tr"))). For most pipelines the cheaper answer is to not create the problem:

import unicodedata

def tr_lower(s):
    # NFC first: a decomposed İ (I + U+0307) has to become the precomposed
    # character, or the replace below misses it and the combining mark survives.
    s = unicodedata.normalize("NFC", s)
    return s.replace("İ", "i").replace("I", "ı").lower()

def tr_upper(s):
    s = unicodedata.normalize("NFC", s)
    return s.replace("i", "İ").replace("ı", "I").upper()

Six lines, applied before anything else touches the text — and the NFC line is not optional. Text arrives decomposed more often than you would like: from HTML forms, from some macOS filenames, from OCR. Without the normalization, tr_lower on a decomposed İstanbul returns ı̇stanbul — the combining dot is still there and it is now over the wrong vowel, which is worse than what you started with.

This is why the bug survives: the code that causes it is text.lower(), which is correct-looking, ubiquitous, and inside every library you did not write. Chapter 11 found it because a cluster came back 94.7% full of one artefact — not because anyone was looking for it. Check for it by lowercasing a word with İ and printing the code points.

Diacritics: deasciification and what it merges

Turkish keyboards are not universal, and a large amount of real Turkish is written without diacritics — guzel for güzel, cok for çok. Handling it means either restoring the diacritics (deasciification) or stripping them everywhere so both forms match.

Deasciification — restoring Turkish diacritics to text typed without them (guzel → güzel). The inverse, stripping diacritics so both forms match, is easier and merges word pairs that are genuinely different. ↩︎

Stripping is the tempting one-liner. Measured on the same corpus:

  words containing a Turkish-specific character   19,318 / 46,243   (41.8%)
  distinct forms                                          15,173
  distinct forms after stripping diacritics               15,101
  ascii forms now ambiguous between 2+ real words             72

     'oldu'   <-  oldu, öldü          (it happened / he died)
     'bas'    <-  bas, baş            (press / head)
     'sac'    <-  saç, sac            (hair / sheet metal)
     'yasa'   <-  yasa, yaşa          (law / live!)

Only 72 collisions, which sounds harmless until you read them. Oldu and öldü are “it happened” and “he died”. Not every merge in that list of 72 costs something, and it is worth checking rather than counting: mi and mı also collapse together, but they are the same question particle under vowel harmony, so nothing is lost. The merge rate is low and the cost per merge is total, which is the worst shape a preprocessing decision can have: it will not show up in aggregate metrics and it will produce individually catastrophic errors.

Restore rather than strip, if you can afford it. If you must strip, strip the query and the document both, and know that you have made those pairs indistinguishable.

16.3 Morphological analysis, and whether you still need it

Before subword tokenizers, handling Turkish meant analysing it: taking evlerimizden and returning the stem ev plus an ordered list of the grammatical functions stacked on it. Turkish has one of the field’s great success stories here — Oflazer (1994) built a two-level finite-state analyser that covers the language’s morphology properly, and its descendants still work. (“Two-level” means the rules relate two strings directly: the dictionary-level form, such as kitap + -I, and the surface form kitabı. That is how an analyser undoes rule 4 above rather than stumbling over it.)

Two things make analysis harder than it sounds:

Ambiguity. Yüzü is at least four different words: yüz “face” with the third-person possessive (his or her face), yüz “face” with the accusative (the face), and yüz “hundred” with either. (The difference between yüz “face” and yüz “surface” is lexical, not morphological: an analyser gives both the same reading.) A sharper case is koyun, which is three separate stems in one surface form: koyun “sheep”, koy “bay” with a possessive (your bay) or genitive (of the bay), and koy- “put” in the second-person plural imperative. A morphological analyser returns all the readings; picking the right one is morphological disambiguation, and it needs sentence context. So the tidy pipeline — analyse, then model — has a sentence-level model inside its first step.

Coverage. An analyser knows the stems it was built with. New words and derivations, brand names, loanwords, typos and slang fall outside it, and the failure mode is a hard one: no analysis at all.

Stem or lemma. Chapter 2 separated stemming (chopping endings off to get a matching key) from lemmatization (returning the dictionary form). In Turkish the gap between the two is wide. The Snowball stemmer’s output need not be a word — burun “nose” becomes bur, and ağız “mouth” becomes ak, which is a different word (“white”). It also targets the endings nouns take, so verbs keep their tense and aspect: okuyorlar “they are reading” becomes okuyor, and gidiyor “is going” comes back unchanged. A lemmatizer built on an analyser, such as Zemberek’s, returns burun for burnu and git for gidiyor (the root; a Turkish dictionary’s headword is gitmek, as in Chapter 2) — but only after choosing among the readings, which is the ambiguity problem above.

So do you need it in 2026?

For most modelling, no. Subword tokenization (Section 8.3) handles morphology statistically rather than linguistically: it does not know that -lerimizden is three suffixes on a stem, but it learns that the piece recurs, which is most of what a model needs. That is why BERTurk works without a morphological analyser anywhere in the stack.

Reach for a real analyser when you need something a subword model does not give you:

  • Lexical retrieval, where you need the stem to index on — Section 16.4 measures exactly how much this is worth.
  • Linguistic annotation, where the grammatical analysis is the output.
  • Rule-driven products — grammar checkers, spelling correction, dictionary lookup — where a wrong answer must be explainable.
  • Very low data regimes, where the inductive bias of knowing that evlerimizden and evimiz share a stem is worth more than the data you do not have.

The compromise most projects land on is a stemmer rather than an analyser: crude, fast, no ambiguity to resolve, and — as Section 16.4 shows — carrying most of the benefit for retrieval. The Snowball Turkish stemmer used throughout this chapter is a few hundred lines of rules. It under-merges, it over-merges, and it takes recall@1 from 28.0% to 95.3%.

16.4 Turkish retrieval: how much of the gap is morphology

Search in Turkish is harder than search in English, and practitioners generally know it. What is worth knowing is how much of the difficulty is morphology alone — because that part has a cheap fix, and the rest does not.

The experiment isolates it. Take 7,165 Turkish sentences as documents, index them with BM25 — the standard word-matching ranking function, which scores a document by the query words it contains (Chapter 2) — and query with three content words drawn from a target document, but in a different surface form than the document uses. That is the real case: the user types İstanbul, the document says İstanbul’da. The score is recall@1, the share of queries whose target document comes back in first place (recall@10: anywhere in the top ten).

7,165 Turkish documents (sentences of 8+ words)
1,481 queries of 3 terms each
69% of query terms are a different inflection than the document uses

index / query scheme      recall@1   recall@10   terms/doc
surface form                 28.0%       54.4%        15.7
5-char prefix                74.6%       92.6%        15.7
Snowball stem                95.3%       99.8%        15.7
character 4-grams            74.9%       94.9%        55.7

Surface-form BM25 finds the right document 28% of the time. Stemming takes it to 95.3%. One preprocessing step, a 3.4× improvement, and no model involved at all.

Read the setup before you take the headline anywhere, because three separate things could inflate that 3.4× on their own.

How the queries were made decides whether the number is circular. If a query term were produced by applying a suffix rule, the same rule-based stemmer would be scored against its own inverse and 95.3% would be near-tautological. The forms here are drawn from elsewhere in the corpus rather than generated, which avoids that — but the design is still known-item retrieval: the query comes from the target document, so a stem match is guaranteed once stemming works at all. Read 95.3% as an upper bound on what stemming can recover, not as what a user’s query will do.

And part of the gain is string surgery rather than morphology. Since version 3.0.0, snowballstemmer’s Turkish algorithm truncates a token at its first apostrophe (ignoring one in the first two characters, so O’Neill’in keeps O’Neill). So the chapter’s own headline example, İstanbul'da → İstanbul, is not a morphological analysis; it is a split("'"). On a news corpus full of inflected proper nouns that step alone is worth a share of the gap, and we did not separate it. Two rows would: apostrophe truncation with no stemming, and Snowball with the apostrophe step disabled. They are the first thing to run before quoting 3.4× anywhere. (One practical trap: only the ASCII apostrophe ' counts. The typographic ’, common in edited text, is not recognised, so replace it with ' before stemming.)

Finally, 69% of query terms were deliberately chosen to be a different inflection, and a query can only rank its target first if enough of its three terms match the indexed form. That is harsher than an average query and gentler than the worst one — a user searching a Turkish corpus usually types the dictionary form, which the corpus often does not contain in that shape.

Three practical consequences:

  1. Stemming is not optional in Turkish lexical search — on queries like these. In English it is usually reported as a marginal tweak. Here it is the difference between a search box that works and one that does not. Keep the condition attached — the 69% above is what makes exact matching fail — and on a corpus whose users type the same forms the documents use, the gap shrinks. Measure it on your own query log before quoting this as a universal requirement.
  2. Character n-grams are the robust fallback. 74.9% recall@1 with no linguistic knowledge at all, at the cost of 3.5× the index size (55.7 terms per document against 15.7). When you have no stemmer for a language, this is the move.
  3. Prefix truncation is a surprisingly good approximation. Keeping the first five characters gets 74.6% — nearly the same as character n-grams, at no index cost. It is crude, it is wrong on short words, and it is what to reach for when a real stemmer is not available.

This is also the strongest single argument for dense retrieval in Turkish: an embedding of İstanbul’da and an embedding of İstanbul are close whatever the tokenizer did, because the model learned them from context rather than matching strings. Chapter 19 builds that properly, including the hybrid retriever that combines a word-matching (sparse) half with an embedding (dense) half. The measurement here says what the sparse half needs in Turkish: stemming. Without it, that half does not pull its weight.

16.5 The five-chapter pattern

Something has happened repeatedly in this book, and it is worth naming as a pattern rather than leaving as a handful of anecdotes. In each case a standard, widely-used tool produced a number that was silently wrong on Turkish, and in each case a control measurement — not a better model — is what exposed it.

Six silent failures, and the control that caught each
chapter the tool what it did on Turkish how it surfaced
Chapter 4 str.lower() IYI → iyi under English rules, ıyı under Turkish comparing the two case mappings side by side
Chapter 11 str.lower() İ becomes i + a combining mark, splitting the word a cluster that was 94.7% one artefact
Chapter 13 WikiANN 13.06% of the test set sitting verbatim in training hashing test inputs against training inputs
Chapter 14 rouge-score 44.4% of Turkish words mutilated by its tokenizer two implementations of one metric disagreeing
Chapter 14 a faithfulness proxy reported human journalists as 56% unfaithful scoring copied text, which cannot be unfaithful
Chapter 15 BLEU about 22.5% fewer word tokens per sentence (English uses 29% more), so each error costs more a comparable error, run in two languages (exploratory)

Three things generalise, and none of them is about Turkish specifically.

The tools were not broken; they were untested outside their assumptions. rouge-score reproduces the original Perl behaviour faithfully. BLEU does exactly what its definition says. WikiANN was built as described. Each is correct on the language it was validated on, and each carries an assumption its documentation does not state.

Every one was caught by a control, not by inspection. Two case mappings side by side; a cluster that should not exist; test inputs hashed against training; two implementations of one metric compared; copied text that must score perfectly; a comparable error in two languages. When you cannot verify a metric directly, feed it an input whose correct answer you already know. That habit found every one of them, and reading the code found none.

The failures were silent by construction. No exception, no warning, no missing field — just a number that was 60% too high, or a benchmark that was 13% memorisation. A pipeline that fails loudly is a pipeline you will fix. The ones to worry about are the ones that return a plausible float.

If you work in a language that is not English, expect this and budget for it. The practical version: before trusting any tool on your language, run it on an input whose answer is known — identical strings, empty strings, a copied document, a single-token change — and check that the number is what it must be. Ten minutes, and it is the highest-return ten minutes in a non-English NLP project.

16.6 Turkish corpora, models, and benchmarks

This section dates faster than anything else in the book. Treat the names as a map of kinds of resource rather than a leaderboard, and check what is current before committing to one.

Knowing what to distrust is half the job. The other half is knowing what exists, and there is more Turkish infrastructure than most people outside Turkey assume.

Encoders. BERTurk (Schweter 2020) remains the default Turkish encoder and is what Chapter 13 and Chapter 12 fine-tuned. It exists in cased and uncased variants, and for Turkish the cased one is almost always right. The uncased checkpoint’s configuration only says “lowercase”; loaded with the library defaults, its tokenizer then lowercases with the English rules (Section 16.2) and, because accent stripping is switched on by default whenever lowercasing is, also removes the diacritics. IŞIK comes out as isik, and öldü as oldu — the merge the diacritics section warns about — even though the model’s vocabulary is full of pieces containing ş, ü or ç that then never get used. If you do use it, pass strip_accents=False and lowercase the text yourself with tr_lower first.

Multilingual encoders. mBERT and XLM-R (Conneau et al. 2020) cover Turkish among a hundred other languages. Section 16.7 measures what that costs.

Generative models. The Turkish-specific generative landscape — TURNA, Kanarya, Trendyol’s and Cosmos’s LLMs, and the Turkish-adapted Llama and Qwen derivatives — changes every few months. What does not change is the question to ask about each: what was it trained on, and can you use it? Licence terms on regionally-released models vary far more than on the big international releases, and a model you cannot deploy commercially is not a candidate.

Benchmarks. Mukayese (Safaya et al. 2022) assembled Turkish baselines across tasks precisely because the field kept reporting Turkish numbers that could not be compared with each other. Turkish versions of the standard suites — question answering, multi-task knowledge — exist and are used.

Two cautions about all of them, both of which this book has already demonstrated on other corpora:

  1. Translation artefacts. A benchmark translated from English tests translated-English competence, not Turkish. Translationese has different word order, different sentence length, and vocabulary chosen by a translation system. Check whether the benchmark was written in Turkish or rendered into it.
  2. Contamination. Chapter 13 found 13.06% of a standard Turkish NER benchmark’s test set sitting verbatim in its own training set — in a resource used by hundreds of papers. Hash your test inputs against training before you believe a score, and against the model’s pretraining corpus if you can.

16.7 Multilingual or monolingual: deciding with a measurement

The question arrives on every Turkish project, and it is usually answered by preference. Here it is answered by running both.

bert-base-multilingual-cased and dbmdz/bert-base-turkish-cased are comparable models — same architecture, same era, same fine-tuning recipe — differing in what they were pretrained on. Chapter 13’s NER setup, identical in every other respect, gives the comparison.

WikiANN Turkish NER, 20,000 training sentences, identical recipe

model                     span P  span R  span F1   macro   train s   infer s
BERTurk (monolingual)      91.49   92.35    91.92   91.82     476.6      10.3
mBERT  (multilingual)      92.25   93.27    92.76   92.67     537.5       8.8

The multilingual model came out ahead, by 0.84 span F1. That is the opposite of what this section was set up to show, and it is worth working through rather than explaining away — starting with what the number can carry. Both rows are a single training run at a single seed, and this book has not measured how much a different seed moves span F1 on this task (Chapter 13 says the same of its own table). For BERT-base fine-tuning on NER the spread is commonly a few tenths of a point; if that holds here, 0.84 is around one or two standard deviations of it: enough to notice, not enough to settle. Five seeds per row is the measurement this table is missing, and until they exist the defensible statement is that this experiment did not separate the two models. Everything below is about why a gap this small is unsurprising rather than about who won.

First, the comparison is fairer than it looks

The obvious objection is size — mBERT has 177.3M parameters against BERTurk’s 110.0M. But look at where the difference is:

model                    total    word-embedding table   everything else
BERTurk                 110.0M                  24.6M             85.5M
mBERT                   177.3M                  91.8M             85.5M

(“Everything else” is the twelve transformer layers plus the small position embeddings and the tagging head; the word-embedding table is vocabulary size × 768.)

The transformer bodies are identical. Twelve layers, same width, same 85.5M parameters doing the actual computation. The entire difference is the embedding table, which is a lookup indexed by vocabulary size — 32,000 entries against 119,547. mBERT is not a bigger model; it is the same model carrying a larger dictionary.

And it wins despite a worse tokenizer for Turkish, measured on the same test set:

model                     vocab    subwords/word   mean sequence length
BERTurk                  32,000             1.66                   14.5
mBERT                   119,547             1.97                   16.8

mBERT spends 19% more tokens on the same Turkish text, with a vocabulary 3.7× larger — because most of that vocabulary belongs to other languages.

Second, the confound that most likely explains it

WikiANN is built from Wikipedia, and both models saw Wikipedia — in very different proportions.

mBERT was pretrained on Wikipedia alone, in 104 languages. BERTurk’s 35 GB corpus is a filtered OSCAR dump plus OPUS corpora plus a Wikipedia dump plus a corpus contributed by Kemal Oflazer, so Wikipedia is a minority of its training data rather than all of it. The domain match is therefore a difference of degree rather than of kind — which is exactly the shape of thing that produces a gap of under a point, and the most plausible systematic reason for one this size, if more seeds show that the gap is real.

The honest reading is therefore not “multilingual models are better for Turkish”. It is: on a Wikipedia-derived task, the model whose pretraining was entirely Wikipedia edged ahead by 0.84 points from one seed, despite tokenizing the language worse — and one seed cannot tell that from noise.

So what this section actually gives you is a protocol

There is no general verdict here, and anyone offering one is selling something. What there is, is a decision procedure — four measurements, in this order, because the cheap ones often decide it before the expensive one runs:

  1. Fertility on your text. Tokens per word for each candidate, on your own domain. Convert to cost and to effective context length (Chapter 8). This is an hour’s work and it sometimes ends the discussion.
  2. Domain overlap. What was each model pretrained on, and how close is it to your data? The measurement above is mostly a demonstration of how much this matters.
  3. The task itself, both models, identical recipe. Same data, same hyper-parameters, the same set of several seeds. Anything else is not a comparison.
  4. The operational columns. Training time, inference latency, memory. Here mBERT trained 13% slower and — despite longer sequences — ran inference slightly faster, which is a reminder to measure rather than reason about throughput.

Rust et al. (2021) is the systematic version of this question: how much of the monolingual advantage is the tokenizer rather than the model. Their answer, across languages, is a lot of it — which makes step 1 the right place to start.

Note what the result does not undermine. mBERT still fragments Turkish 19% worse, which is a permanent tax on cost and context that no accuracy number removes. If this had been a generation task billed per token rather than a tagging task scored per span, the same tokenizer measurement could have decided the choice on its own — before any accuracy was compared.

16.8 Intuition checks

Answer each one before you open it. All six are questions this chapter settled with a measurement rather than an argument.

Not the ranker, and not the embeddings. Look at whether you are indexing surface forms.

Measured on 7,165 Turkish documents with queries whose terms are inflected differently from the documents: surface-form BM25 gets 28.0% recall@1, Snowball stemming gets 95.3%. A user types the dictionary form; the corpus contains an inflected one; exact string matching finds nothing.

If you have no Turkish stemmer, character 4-grams get 74.9% at 3.5× the index size, and truncating to five characters gets 74.6% for free.

Two reasons, and the second is worse.

Turkish has four letters where English has two — İ i I ı — and the case mapping is not the ASCII one. Every word containing I or İ comes out differently under English rules — that follows from the two mappings — and on the WMT16 Turkish source such words are 0.95% of all tokens, many of them names.

Then: 'İSTANBUL'.lower() produces i followed by U+0307 COMBINING DOT ABOVE, which is a nonspacing mark and not a word character. So re.findall(r"\w+", ...) returns ['i', 'stanbul'] — the word is split in two, and NFC normalization does not put it back. The model never sees istanbul at all.

Only with your eyes open. On the WMT16 source it merges 15,173 forms into 15,101 — just 72 collisions — which looks like a free win until you read them: oldu / öldü is “it happened” / “he died”. bas / baş is “press” / “head”.

The shape of that trade is the worst one available: too rare to show up in an aggregate metric, and catastrophic in each individual case. Restore the diacritics if you can afford it. If you must strip, strip both sides and document that those pairs are now indistinguishable.

On identical content — 9,007 professionally translated sentence pairs — Turkish produced 32,866 distinct forms from 141,412 tokens against English’s 16,652 from 182,429. Twice the vocabulary from 78% of the tokens.

The number that matters more is that the ratio grows: 1.49 at 5,000 tokens, 2.31 at 141,000. Both vocabularies follow Heaps’ law and keep growing, but Turkish has the larger exponent (about 0.78 against 0.65) because it generates forms by rule, so the ratio widens as the corpus grows. Any design that assumes a vocabulary can be enumerated — a fixed word list, a closed feature set, an exact-match index — is fighting that.

Three questions, in order.

Was the benchmark written in Turkish or translated into it? A translated benchmark measures competence on translationese.

Is the test set in the training set? Chapter 13 found 13.06% of a widely-used Turkish NER benchmark’s test set verbatim in its own training split. Hashing takes four lines.

What did the tokenizer do? Chapter 14 found the standard ROUGE implementation mutilating 44.4% of Turkish words. If the score came out of a tool that was validated on English, the score is a hypothesis.

When you need one deployment for many languages, when you need cross-lingual transfer from a language with more labelled data, or when no monolingual model exists for your language.

What you give up is measurable rather than mysterious: a multilingual tokenizer spends more tokens per Turkish word, which costs money and context (Chapter 8), and it spends vocabulary on languages you are not using.

But do not assume the accuracy follows. Section 16.7 ran both on the same task and the two came out within 0.84 span F1 of each other, with the multilingual one ahead despite needing 19% more tokens for the same text — on a benchmark whose domain matched its pretraining corpus, from a single seed. Measure fertility first, domain second, and the task third; and run more than one seed before you call a gap that size a result.

16.9 Interview questions

On sourcing — the same caveat as Chapter 2.

Testing whether you understand the language

Q: Explain the Turkish i problem to someone who has never seen it.

Turkish has four letters where English has two: dotted i/İ and dotless ı/I. They are different letters, not different cases of one letter, so the mapping between them is not the ASCII one — I lowercases to ı, and i uppercases to İ. The consequence is the part interviewers are listening for: str.lower() is not a style choice in Turkish, it is a correctness bug. It turns IŞIK into işik where Turkish requires ışık. That every word containing one of these letters comes out differently is guaranteed by the two mappings; the measured number is how often such words occur — about one token in a hundred on WMT16 news, concentrated in names and sentence-initial words. It fails a second way too: 'İstanbul'.lower() returns i plus a combining dot above, nine code points where istanbul has eight, so the two compare unequal while rendering identically on screen — and a \w+ tokenizer then splits it into ['i', 'stanbul'], which NFC normalization does not repair. The fix is a locale-aware lowercaser, applied everywhere, including inside your metric code.

Q: Why do subword tokenizers struggle with Turkish, and what does it cost?

Because agglutination generates word forms combinatorially rather than listing them: one stem plus a stack of suffixes, so arkadaş appeared in 39 distinct forms in a single corpus. A tokenizer trained mostly on English either lacks pieces for those suffixes or has pieces that do not always line up with morpheme boundaries (the im|iz split in Section 8.3.3). The cost is concrete and worth naming in three places: more tokens for the same text, which is money; less content per context window, which is capability; and a faster-growing vocabulary — both languages follow Heaps’ law, but this chapter measured the Turkish/English type ratio growing from 1.49 at 5,000 tokens to 2.31 at 141,000, because English flattens faster.

Q: Stemming or morphological analysis for Turkish?

Start with the stemmer and be surprised by how far it goes: Snowball is crude, over-merges and under-merges, and still took recall@1 from 28.0% to 95.3% on this chapter’s retrieval task — a 3.4× improvement from one preprocessing step, no model involved. Reach for a real morphological analyser when you need the analysis itself rather than a matching key: recovering case to decide who did what to whom, disambiguating yüzü (her face / the face / the hundred), or generating forms. The engineering answer is that the stemmer is the default and the analyser is a requirement you should be able to justify.

Testing whether you have shipped something

Q: You inherit a Turkish NLP system with no tests. What do you do first?

Not train anything. Run the controls, because this chapter’s whole argument is that tools fail silently outside English. Lowercase a list of words with I and İ and diff it against a correct lowercaser. Feed your metric a copy of the reference and check that it scores a perfect one; that catches a harness that processes the two sides differently, like the faithfulness proxy of mine that called professional journalists 56% unfaithful (Chapter 14). It does not catch a tokenizer that damages both sides the same way: rouge-score, whose tokenizer splits 44.4% of Turkish words, still gives a Turkish reference a perfect score against itself. For that, score two sentences that differ in one Turkish word, or compare against a second implementation, and print the tokens — on güzel against özel, which share no word, rouge-score reports a ROUGE-1 of 0.67 instead of 0. Then measure fertility on real production text. Each of those is an afternoon, and each one has historically found a real bug.

Q: How do you budget an LLM feature for Turkish?

By measuring fertility on your own text and multiplying, rather than by translating an English estimate. Be careful converting fertility into a bill, because they are not the same ratio. Fertility is tokens per word, and Turkish says the same thing in fewer words — Chapter 15 measured English needing 1.29× as many word tokens for identical content. Combine the two and a 1.5–2× fertility ratio comes out at roughly 1.2–1.6× the total tokens on matched content. Treat that as an illustration rather than a measured invoice: the two ratios come from different samples, and the bill also separates input from output pricing, caching and retries. If you want the number, count total tokens for both languages on one parallel sample. The same total ratio shrinks the effective context window, which is the constraint that actually breaks features. Then add that the choice of model moves this number: a language-specific tokenizer is the single largest lever, and it is fixed at pretraining time, so it is a procurement decision rather than something you tune later.

Q: Your Turkish benchmark shows 92. Do you trust it?

Ask what the control says: what does a simple baseline score, and how far does a second seed move the number? This chapter fine-tuned a monolingual and a multilingual model on the same Turkish task and the multilingual one came out ahead, 92.76 to 91.92 F1 from a single seed, against the folklore and against my own expectation — but the two share a transformer body of the same size, so the honest reading is that the comparison is about pretraining data, not architecture, and one task is not a verdict. What earns trust is a protocol: fertility measured on your text, domain overlap checked, both models fine-tuned on your data, and a contamination audit before any number is quoted.

16.10 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below.

The stack these numbers came from
library version used what it is for here
snowballstemmer 3.1.1 the Turkish stemmer behind the 28.0% → 95.3% retrieval result
transformers 5.15.1 fine-tuning BERTurk and mBERT in Section 16.7
datasets 5.0.1 the Turkish splits used throughout

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
resource current version when you would reach for it
zeyrek 0.1.3 a pure-Python morphological analyser and lemmatizer — full analyses when a stem is not enough
zemberek-python 0.2.3 a partial pure-Python port of Zemberek, the reference Turkish NLP toolkit (no Java needed): analysis with disambiguation, word generation when you need to produce inflected forms, tokenization, normalization of noisy text
turkish-morphology 1.2.5 Google’s finite-state analyser, if you want a second opinion on an analysis
trnlp 0.2.3a0 stemming, analysis and suggestions for words typed with ASCII letters instead of Turkish ones
BERTurk / mBERT on the Hub the two models compared above; start with both, not with one
Mukayese on GitHub a Turkish benchmark suite — use it for the tasks it standardises, and read what each task actually measures before quoting a score

16.11 Lab: a Turkish evaluation suite you can trust

The point of this lab is not to rank models. It is to build an evaluation you would defend, on a language the tools were not validated on.

Step 1 — write three tasks, in Turkish, yourself

Fifty items each. Classification, extraction, and something generative. Write them in Turkish rather than translating from English, because a translated suite measures translationese. Keep them boring and unambiguous — you are building a measuring instrument, not a challenge set.

Step 2 — validate the scorer before scoring anything

Feed each metric a perfect prediction and confirm it returns a perfect score. Then feed it a copied input, an empty prediction, a prediction that differs by one character, and a pair of sentences that differ in one whole Turkish word. Write down what each should return before you run it.

This step is the whole lab — and the perfect-prediction check alone is not enough. Chapter 14 found a standard metric mutilating 44.4% of Turkish words, and that metric passes the perfect-prediction test: both sides are split the same way, so a reference still matches itself exactly. What exposes it is a pair with a known non-perfect answer (güzel against özel should share no word, yet scores 0.67), a second implementation that disagrees, or simply printing the tokens.

Step 3 — check your own text for the two normalization bugs

Lowercase your suite and search for U+0307. Count words containing İ and compare str.lower() against tr_lower() on each. If your suite already contains the bug, every model you evaluate will be scored on damaged input.

Step 4 — measure fertility and convert it to money

For each model you plan to test, tokens per Turkish word on your own text. Multiply by your request volume. This number often decides the choice before accuracy does.

Step 5 — run three models, including one multilingual

A Turkish-specific model, a multilingual one, and an LLM. Report accuracy, tokens per request, latency, and cost per thousand requests — four columns, because the first one alone has never decided a deployment.

Step 6 — audit for contamination

Hash your suite’s items against anything you know about the models’ training data. You will usually not be able to check a pretraining corpus; say so explicitly rather than leaving it implied.

What to hand in

The four-column table, the scorer-validation log from step 2 with its predicted-versus-actual column, and a paragraph naming which model you would ship and what would change your mind.

16.12 Exercises

  1. Vocabulary growth in your language. Take any professionally translated parallel corpus and plot distinct forms against tokens for both sides. Report the ratio at three corpus sizes and say whether it is widening.

  2. Break the lowercaser. Find a Turkish word where str.lower() produces something re.findall(r"\w+", ...) splits in two. Print the code points. Then check whether the libraries in your own stack have the same problem.

  3. The retrieval fix, on your data. Index a Turkish corpus with BM25 twice, surface forms and stems. Query with dictionary forms. Report recall@1 for both, and decide what your search box should do.

  4. The diacritic collision list. Strip diacritics from a Turkish vocabulary and list every ascii form that now maps to two or more real words. Rank them by how badly the confusion would matter in your domain.

  5. Fertility to invoice. Take a Turkish document and its English translation, tokenize both with the tokenizer your product actually uses, and convert the difference into a monthly bill at your request volume. Then do it with a Turkish-native tokenizer and report the difference (Chapter 8 has the method).

  6. Audit a Turkish benchmark. Pick one you would use. Check for translation artefacts, hash its test set against its training set, and run its scoring script on a perfect prediction to confirm it returns a perfect score — then on two Turkish sentences that differ in one word, because a tokenizer that damages both sides equally still passes the first check. Report what you find, including if the answer is “nothing wrong”.

  7. The control habit. Take any metric you rely on and feed it an input whose correct answer you know for certain — identical strings, a copied document, an empty prediction. Write down what it should return and what it did. This exercise has a low hit rate and an enormous payoff.

16.13 Where to go deeper

Start by running str.lower() over a list of Turkish words containing I and İ and diffing it against a locale-aware lowercaser. Everything else in this chapter is downstream of believing that result.

Beyond the papers
resource what it is
Zemberek the reference Turkish NLP toolkit — morphology, spelling, tokenization; the thing to reach for when a stemmer is not enough
Mukayese a Turkish benchmark suite with baselines; read what each task measures before quoting a score
BERTurk the monolingual models used here, with their corpora and training documented
TDD — Türkçe Doğal Dil İşleme Turkish corpora, models and tools collected in one place
Speech and Language Processing — Jurafsky & Martin free online; the “Words and Tokens” chapter covers morphemes and subword tokenization. The finite-state treatment of morphology is in the printed second edition, not in this draft
The primary sources
topic source
Turkish morphology, done properly Oflazer (1994)
the Turkish benchmark suite Safaya et al. (2022)
the monolingual encoder used throughout Schweter (2020)
what a language-specific tokenizer is worth Rust et al. (2021)
the multilingual model it was measured against — bert-base-multilingual-cased Devlin et al. (2019)
XLM-R, the other multilingual encoder named above Conneau et al. (2020)

16.14 What to remember

  • Agglutination is measurable, and it compounds. On identical content, Turkish produced 32,866 distinct forms from 141,412 tokens against English’s 16,652 from 182,429 — twice the vocabulary from 78% of the tokens. The ratio grows with corpus size (1.49 → 2.31): both curves follow Heaps’ law, but Turkish has the larger exponent (≈0.78 vs ≈0.65), because it generates forms by rule rather than drawing them from a list. Any design that assumes a vocabulary can be enumerated is fighting that.

  • str.lower() is a correctness bug in Turkish, twice over. Every word containing I or İ comes out wrong under English rules — that part is arithmetic — and such words are about one token in a hundred (0.95% of the WMT16 source), concentrated in names. And 'İSTANBUL'.lower() emits i plus a combining mark, so \w+ returns ['i', 'stanbul'] — the word splits and NFC does not repair it. A three-line tr_lower before anything else touches the text.

  • Stemming is not optional in Turkish lexical search — on queries like the ones tested here. Sixty-nine percent of that query set was built to differ from the document in inflection, so measure the gain on your own query logs before treating it as a requirement. Surface-form BM25: 28.0% recall@1. Snowball stemming: 95.3%. Character 4-grams get 74.9% with no linguistic knowledge, at 3.5× the index. In English this is usually reported as a tweak; here it is whether the search box works.

  • Stripping diacritics causes only 72 collisions, and some of them are total. oldu and öldü — “it happened” and “he died” — become the same string (a few, like mi/mı, cost nothing). Rare enough to pass every aggregate metric, severe enough to matter every time it hits.

  • A multilingual model came out ahead of the monolingual one on this task, 92.76 against 91.92 span F1 — with identical transformer bodies (85.5M each; the parameter gap is all embedding table) and despite tokenizing Turkish 19% worse — from a single seed, so not yet a separation. If the gap holds up, the likely reason is domain: the benchmark is built from Wikipedia and mBERT was pretrained on Wikipedia alone. There is no general verdict here, only a protocol — fertility on your text, domain overlap, both models on your task with an identical recipe, then the operational columns.

  • Subword tokenization replaced morphological analysis for modelling, not for everything. Reach for a real analyser when the grammar is the output, when a wrong answer must be explainable, or when you need the stem to index on — and note that a stemmer of a few hundred lines carried most of the retrieval benefit.

  • The pattern, across five chapters: standard tools fail silently outside English. str.lower(), rouge-score, BLEU, a benchmark’s own splits, and a faithfulness proxy written for this book each produced a plausible wrong number on Turkish. None raised an error. Every one was caught by a control measurement — an input whose correct answer was known in advance — and none by reading the code. The last entry on that list is ours, which is the point: the failure mode is not other people’s carelessness.

  • So build the control first. Before trusting a tool on your language, give it something whose answer cannot be in doubt: identical strings, copied text, a perfect prediction, two texts that share no word, a majority-class floor. A perfect score on identical inputs is necessary, not sufficient — it cannot see a tokenizer that damages both sides alike. Ten minutes, and it is the highest-return ten minutes in a non-English NLP project.