Appendix F — Turkish NLP Datasets and Tools
This catalogue changes faster than any other page in the book. Model names change, repositories are archived, and licences are revised. Treat it as a map of what kinds of resource exist and check each before committing to it.
F.1 Corpora
Licence is a column of its own because it is usually the reason a dataset cannot be used — in class or in a product.
| dataset | task | size | licence | link |
|---|---|---|---|---|
| XL-Sum Turkish | summarization | 27,176 train / 3,397 test, plus a validation split the card ships and this book did not use | CC BY-NC-SA 4.0 | csebuetnlp/xlsum |
| Turkish product reviews | sentiment | ~235k reviews | check the card — the licence section and the metadata field have disagreed; do not assume CC BY | fthbrmnby/turkish_product_reviews |
| WikiANN (Turkish) | NER | 20k train / 10k validation / 10k test | check the card — commonly cited as ODC-BY, and the card itself does not state one | unimelb-nlp/wikiann |
| SQuAD-TR | question answering | translated SQuAD | CC BY-NC-SA 4.0 | boun-tabi/squad_tr |
| UD Turkish BOUN | dependency parsing | ~9.7k sentences | CC BY-SA 4.0 | UD_Turkish-BOUN |
| WMT newstest tr-en | machine translation | size varies by year; Chapter 15 used newstest2016 | see the WMT release terms for the year you use | via sacrebleu (Chapter 15) |
Three cautions, each of which this book demonstrated on one of these datasets rather than in the abstract.
Check for contamination before quoting a score. Chapter 13 found 13.06% of WikiANN Turkish’s test set sitting verbatim in its own training split — in a resource used by hundreds of papers. Hashing test inputs against training inputs takes four lines.
Check whether the benchmark was written in Turkish or translated into it. SQuAD-TR is translated, and that is fine as long as you say so: it measures competence on translationese, which has different word order, sentence length and vocabulary from text a Turkish speaker wrote.
Read the non-commercial clauses — and read them at the source. NC in a licence means the dataset cannot be used in a commercial product, which is discovered surprisingly late in surprisingly many projects. The licence column above is a starting point and not an authority: a Hub dataset carries a licence in its card prose and a separate metadata field, the two disagree more often than you would like, and neither is the upstream LICENSE file. Two rows here are marked check the card for exactly that reason. Before a dataset goes into anything commercial, follow it to the original release and record what you found and when.
F.2 Tokenizers and morphological analyzers
Section 16.3 argues that a stemmer carries most of the retrieval benefit and an analyser is a requirement you should be able to justify. This is the shortlist for both.
| tool | what it does | state |
|---|---|---|
snowballstemmer |
rule-based Turkish stemmer, ~100 lines | maintained; took recall@1 from 28.0% to 95.3% in Section 16.4 |
zemberek-nlp |
the reference Turkish toolkit — morphology, spelling, tokenization | Java, and no longer actively developed; still the widest coverage of the options this appendix lists |
zeyrek |
pure-Python morphological analyzer, a partial Zemberek port | lightly maintained; the easiest way to get real analyses in Python |
turkish-morphology |
Google’s finite-state analyzer | archived; useful as a second opinion, not as a dependency |
trnlp |
analysis plus generation of inflected forms | maintained; reach for it when you need to produce forms, not just parse them |
Several well-known Turkish analyzers are unmaintained, and it is better to know that now than mid-project. Zemberek in particular still has the widest coverage of the options this appendix lists and is not receiving fixes; plan for that rather than discovering it when a bug matters.
For tokenization specifically, the finding that decides most projects is in Chapter 2 and Chapter 16: a multilingual tokenizer spends noticeably more tokens on Turkish, and for a checkpoint you are handed that ratio comes with it. Changing it is possible — extend the vocabulary, initialise the new embeddings, continue training — and the compute depends on how much you change and the quality you need; it is not an ordinary fine-tune, which is why Chapter 8 treats it as a procurement decision. Measure fertility on your own text before choosing a model.
F.3 Pretrained models
| model | kind | note |
|---|---|---|
| BERTurk | encoder, 110M | the default Turkish encoder; Chapter 13 and Chapter 12 fine-tune it. Use the cased variant — see Section 16.2 for what uncasing does to İ |
| mBERT | encoder, 177M | the multilingual comparison in Section 16.7, where it won on a Wikipedia-derived task despite tokenizing Turkish 19% worse |
| TURNA | encoder-decoder, 1.1B | Turkish-specific generative model from Boğaziçi, built on UL2. Released for non-commercial academic research only — the clearest example on this page of a model you can benchmark and cannot ship |
| Kanarya | decoder, 2B | Turkish generative model; small enough to run locally |
ytu-ce-cosmos |
several | Yıldız Technical University’s Turkish model family, including GPT-2 and Llama derivatives |
Trendyol |
several | industry-released Turkish LLMs; check licence terms, which vary more than international releases |
The question to ask about each, and the reason this table gives no rankings: what was it trained on, and can you use it? Licence terms on regionally released models vary far more than on the big international ones, and a model you cannot deploy commercially is not a candidate — however good its benchmark numbers.
This table deliberately does not carry a licence column, and that is a limit rather than a decision: for the families listed — ytu-ce-cosmos, Trendyol — the licence varies per model, and derivatives of Llama carry Llama’s terms rather than their publisher’s. A column would have to be wrong for at least one row. Read the model card before you build on it, and note the date you read it. TURNA above is the case that catches people: a good model, freely downloadable, and not licensed for the product you were planning.
F.4 Benchmarks
| resource | what it is |
|---|---|
| Mukayese | Turkish baselines across tasks, assembled (Safaya et al. 2022) because the field kept reporting Turkish numbers that could not be compared with each other |
| TDD — Türkçe Doğal Dil İşleme | Turkish corpora, models and tools collected in one place; the best single starting point |
turkish-bert |
the BERTurk models with their corpora and training documented — read it for what “trained on Turkish” actually meant |
Those three are a catalogue, a baseline collection and a model repository — useful, and none of them is a current benchmark for a Turkish LLM. The suites that are arrived after 2024, and they answer Mukayese’s original complaint directly:
| benchmark | what it measures | native or translated |
|---|---|---|
| TurkishMMLU (2024) | multi-task knowledge, ~10k high-school questions across nine subjects | native — written from Turkey’s national curriculum rather than translated |
| TR-MMLU (2024) | multi-task knowledge, 6,200 questions across 62 categories including law and health | native, sourced from standardised exams |
Cetvel (EACL 2026, KUIS-AI/cetvel) |
23 tasks in seven categories, generative as well as multiple-choice — grammatical error correction, machine translation, questions rooted in Turkish history and idiom | native and adapted |
| TurkBench (2026) | general Turkish LLM evaluation | native |
Cetvel is the one to start from if you are comparing models, and the reason is the distinction this book keeps returning to: TurkishMMLU and TR-MMLU are multiple-choice, so they rank models on knowledge, and a Turkish system in production usually fails on generation. A benchmark that only asks questions with four options cannot see the failure you are going to have.
Before quoting any score from any of these, validate the scorer — and use the right control, because the obvious one is weaker than it looks. Feeding the script a perfect prediction and confirming it returns 1.0 catches a scorer that tokenizes the two sides differently, which is a real and common bug (Chapter 14 hit it twice in one faithfulness proxy). It cannot catch a scorer that mangles both sides the same way: split every Turkish word into fragments on both sides and the identity check still returns a perfect score.
For that failure you need two more checks, both a few lines. Print the scorer’s own token output for a Turkish sentence and look at it — görüştü should not come back as ['g','r','t']. And score minimal pairs that differ only in the characters at issue — açık/acik, şirket/sirket, İstanbul/Istanbul — and confirm the score drops for every pair your metric is meant to tell apart. If it ignores diacritics on purpose, write that down: a pair that scores as a match should be a decision you made, not one the tokenizer made for you. Chapter 14 found the standard ROUGE implementation splitting 44.4% of Turkish words into fragments, and what surfaced it was comparing its output against a Turkish-correct tokenizer, not an identity check.
F.5 Contributing
This page is reviewed once a semester, before the course begins, and that is not often enough — a link here may already be stale.
If you find a dead link, a changed licence, an archived repository or a resource that should be listed, open a GitHub issue with [resources] in the title. Include what you found and when you checked it; a dated observation is worth more than a correction with no date, which is the same argument Chapter 38 makes about regulatory instruments.