15  Machine Translation

Chapter status: drafted. Every number on this page was measured on WMT16 newstest2016 Turkish–English — 3,000 professionally translated sentence pairs in each direction, downloaded by sacrebleu itself so that the test set is identified rather than described. Systems are scored on the first 1,000 sentences unless a table says otherwise, and the sample size differs by experiment: the decoding sweep uses the first 500, the morphological perturbation draws its subsets from the first 1,000, and the LLM head-to-head uses the first 400 because a 9B model at one second per sentence sets the budget. Every table states its own \(n\). Section 15.5 measures why the field’s default metric is harsher on Turkish than on English, with an exploratory test that puts a comparable error into both languages. Software: sacrebleu 2.6.0, transformers 5.15.1, PyTorch 2.13.0 on Python 3.14.5; the MT model on an Apple M-series GPU, the LLM through ollama 0.24.0.

Learning objectives. After this chapter you can:

• explain why translation is the task that produced the attention mechanism, and what the bottleneck was that attention removed;
• pick a decoding strategy suited to translation rather than to chat, and say why the same setting is right in one and wrong in the other;
• compute BLEU by hand, state precisely what it can and cannot see, and report it alongside something that sees more;
• quantify why a word-level metric is harsher on an agglutinative language than on English, rather than asserting that it is;
• decide between a dedicated MT model and an LLM on measured quality, latency and cost;
• choose between translate-train, translate-test and neither, and say where the errors accumulate in each.

15.1 Translation is where attention came from

Chapter 7 taught attention as a mechanism. This chapter is where it gets its origin story, because attention in its modern form was not proposed as a general idea — it was proposed to fix one specific, measurable failure in translation. (Similar soft-addressing ideas were being tried in other tasks at the time, handwriting generation for one.)

The 2014 architecture (Sutskever et al. 2014) was an encoder RNN that read the source sentence and compressed it into a single fixed-length vector, and a decoder RNN that generated the translation from that vector alone.

Elegant, and it worked — up to a point that was easy to find. Everything the decoder would ever know about a forty-word sentence had to fit in the same number of floats as everything it would know about a four-word one.

The symptom was unmistakable: translation quality fell off a cliff as source sentences got longer, and it fell off in a specific way — the model would translate the beginning well and then drift away from the source, because a vector of fixed size cannot hold every detail of a long sentence, and the details it has lost are the ones the decoder needs later.

Bahdanau et al. (2015)’s fix was to stop compressing. Keep one vector per source token, and let the decoder, at each output step, compute a weighted average over all of them — weights that depend on what it is currently trying to produce.

That is attention, and the reason it was obviously right here is that translation has a property the mechanism mirrors exactly: output words correspond to input words. When the decoder emits Istanbul, the useful information is concentrated in one source position, and an average that can concentrate its weight there is a better interface than a fixed summary.

This is why the first attention papers could publish alignment heatmaps that looked like word alignments — and why later attempts to read transformer attention heads as explanations disappointed. In a translation encoder-decoder the alignment interpretation is earned by the task’s structure. In a 24-layer language model it is a hypothesis, and whether attention weights are a faithful explanation remains contested (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).

Two years later Vaswani et al. (2017) removed the recurrence and kept only the attention, which is Chapter 7’ subject. The lineage is worth holding on to: the dominant architecture in machine learning began as a repair to a translation bug.

What we will look at. First the three eras of translation systems, briefly, because each one left something the next kept. Then the metric the field runs on, computed by hand on a sentence that is a correct translation and scores 50. Then decoding, where the settings that make a chat reply feel natural cost a translation half its BLEU score. Then the measurement this chapter exists for: how much harsher a word-level metric is on Turkish than on English, from an exploratory test that puts a comparable error into both. And finally an LLM against a model roughly forty times smaller, and where each of them belongs.

15.2 Three eras, and what each got right

Three answers to the same problem, each replacing the last, and each leaving behind something the next era still needs. (Before all three came rule-based MT, from the 1950s onwards: bilingual dictionaries plus grammar rules written by linguists. It was precise where its rules applied and brittle everywhere else, and writing enough rules for a whole language never became affordable.)

Statistical MT began with word-based alignment models (Brown et al. 1993) and matured into phrase-based systems (Koehn et al. 2003). A phrase-based system learned a phrase table — this source phrase maps to that target phrase with this probability — plus a target-language model to keep the output fluent, and a reordering model. It was assembled from parts you could inspect. If a translation was wrong you could find the phrase-table entry responsible and read it.

Word alignment is the thing worth carrying forward from that era. It was the last time the field had a fully interpretable account of what a translation system was doing, and it is still what terminology checks and localisation QA are built on. (Translation memories are a close relative: they store sentence-aligned pairs.)

A system that cannot say which source words produced which target words cannot be audited by a human translator, and in regulated localisation that is a hard requirement.

Neural MT replaced the pipeline with one trained model. It fixed reordering and fluency almost immediately — statistical output read like statistical output — and it introduced failure modes the phrase table could not have: fluent inventions, dropped clauses, and a habit of translating a rare word by whatever plausible thing it saw in training instead. Koehn and Knowles (2017) catalogued them.

General-purpose LLMs were not built to translate and translate anyway, because a next-token model trained on a multilingual web has seen the same content in many languages. What they add over a dedicated model is context: a document-level view, a glossary in the prompt, a register instruction. What they lose is cost and determinism, and Section 15.6 measures both.

15.3 Evaluation, and how BLEU actually works

Translation has no single right answer, so every reference-based automatic metric compares against one or more human translations and reports a similarity. A reference-free family also exists: quality estimation (QE) metrics predict a score from the source and the output alone, which makes them the automatic signal you can use in production, where there is no reference. The metric you pick decides which systems you will believe are better, so it is worth knowing exactly what each one counts.

BLEU by hand

BLEU (Papineni et al. 2002) is modified n-gram precision, geometrically averaged over n = 1..4, multiplied by a brevity penalty. That is the whole definition, and it fits in a worked example.

BLEU — the share of the candidate’s word n-grams that appear in the reference, averaged over n = 1 to 4, with a penalty for being too short. The field’s default metric since 2002, and blind to any correct paraphrase that uses different words. ↩︎

Two words in that definition need unpacking first. Modified (or clipped) precision means a candidate n-gram is credited at most as many times as it occurs in the reference. Without the clip, the candidate the the the the would get a perfect unigram precision against the cat sat on the mat; with it, only the reference’s two thes count, so the precision is 2/4. The brevity penalty exists because precision alone rewards saying very little: four correct words offered for a twenty-word reference have perfect precision. With \(p_n\) the clipped n-gram precision, \(c\) the candidate length and \(r\) the reference length:

\[ \text{BLEU} = \text{BP}\cdot\exp\Big(\frac{1}{4}\sum_{n=1}^{4}\log p_n\Big), \qquad \text{BP} = \begin{cases} 1 & \text{if } c > r \\ e^{\,1 - r/c} & \text{if } c \le r \end{cases} \]

The exponential of an average of logs is simply the geometric mean of the four precisions, and the result is usually multiplied by 100. On a whole test set, BLEU is not the average of sentence scores: the matched and total n-gram counts, and the lengths \(c\) and \(r\), are summed over all sentences first, and the formula is applied once. Every table in this chapter reports that corpus BLEU. A single-sentence score like the one below is a teaching device, and it needs smoothing whenever one n-gram order has no match at all, because \(\log 0\) is undefined.

REF: In Istanbul , heavy rain caused flooding on Monday evening .   (11 tokens)
HYP: Heavy rain in Istanbul caused flooding on Monday evening .     (10 tokens)

The candidate is a correct translation. It moved the location phrase to the middle, which English permits and which changes nothing about the meaning. Here is what BLEU makes of it:

  1-gram precision   8/10 = 0.8000
  2-gram precision   5/9  = 0.5556
  3-gram precision   4/8  = 0.5000
  4-gram precision   3/7  = 0.4286

  brevity penalty    len(hyp)=10, len(ref)=11  ->  exp(1 - 11/10) = 0.9048
  geometric mean of the four precisions        =  0.5555

  BLEU = 0.9048 x 0.5555 = 50.27

Edit the translation and watch BLEU move

viewof bleuRef = Inputs.text({label: "reference",
  value: "In Istanbul , heavy rain caused flooding on Monday evening .", width: 560})

viewof bleuHyp = Inputs.text({label: "candidate",
  value: "Heavy rain in Istanbul caused flooding on Monday evening .", width: 560})

viewof bleuLower = Inputs.toggle({label: "lowercase both first", value: false})

bleuScore = {
  const prep = t => (bleuLower ? t.toLowerCase() : t).trim().split(/\s+/).filter(x => x);
  const ref = prep(bleuRef), hyp = prep(bleuHyp);
  const grams = (a, n) => a.slice(0, Math.max(0, a.length - n + 1))
                           .map((_, i) => a.slice(i, i + n).join(" "));
  const rows = [];
  let logsum = 0;
  for (let n = 1; n <= 4; n++) {
    const h = grams(hyp, n), r = grams(ref, n);
    const need = new Map();
    for (const g of r) need.set(g, (need.get(g) ?? 0) + 1);   // clipped counts
    let hit = 0;
    for (const g of h) if ((need.get(g) ?? 0) > 0) { hit++; need.set(g, need.get(g) - 1); }
    const p = h.length ? hit / h.length : 0;
    rows.push({n: `${n}-gram`, matched: hit, total: h.length, precision: p});
    logsum += Math.log(Math.max(p, 1e-12));
  }
  const bp = hyp.length > ref.length ? 1 : Math.exp(1 - ref.length / Math.max(hyp.length, 1));
  const geo = Math.exp(logsum / 4);
  return {rows, bp, geo, bleu: bp * geo * 100, hlen: hyp.length, rlen: ref.length};
}

Inputs.table(bleuScore.rows, {
  columns: ["n", "matched", "total", "precision"],
  header: {n: "order", matched: "matched", total: "in candidate", precision: "precision"},
  format: {precision: v => v.toFixed(4)},
  width: {n: 90, matched: 90, total: 110, precision: 100},
  rows: 4})

md`brevity penalty **${bleuScore.bp.toFixed(4)}** (candidate ${bleuScore.hlen}
tokens, reference ${bleuScore.rlen}) × geometric mean
**${bleuScore.geo.toFixed(4)}** = **BLEU ${bleuScore.bleu.toFixed(2)}**`

50.27 for a correct translation. Two things did the damage, and both are worth naming because they recur constantly:

• Moving in Istanbul broke every n-gram that crossed the move. The unigrams survived; the 4-grams did not.
• Heavy did not match heavy, and in did not match In. BLEU is case-sensitive by default. That alone cost two unigrams out of ten and two bigrams (heavy rain, in Istanbul). Switch on the lowercase toggle above and the same pair scores 57.81; the remaining loss is the reordering.

For the same pair, chrF2 scores 83.23 — character n-grams of length 1 to 6 with \(\beta = 2\), sacrebleu’s default, and the variant every chrF number in this chapter uses. It scores this high because at the character level almost everything still matches. The chrF++ variant, which adds word unigrams and bigrams, scores the pair lower (78.07), because the reordering the pure character metric forgave becomes visible again as soon as words re-enter.

Reproduce it and you will get 50.27 exactly — but only if you say which BLEU. Tokenization, casing, and smoothing all change the number, which is why sacrebleu exists and why Post (2018) had to be written: a BLEU score without its signature is not comparable to another BLEU score. Report the signature, always — which obliges us. Every BLEU in this chapter is sacrebleu 2.6.0 at its defaults, case:mixed | tok:13a | smooth:exp | nrefs:1; every chrF is chrF2, case:mixed | nc:6 | nw:0 | eff:yes. That tok:13a matters more than usual for the Turkish measurements below: under 13a an apostrophe does not split a word, so İstanbul'da is one token and a suffix edit destroys all of it; under intl the apostrophe splits and the edit destroys less. The perturbation experiment in Section 15.5 is therefore a tok:13a result, and re-running it under intl is a worthwhile exercise that will move the number.

The metric family, and what each one sees

BLEU’s blind spots are not a reason to stop measuring, only a reason to know what else is available. Four options, in increasing order of what they see and what they cost:

Four ways to score a translation

metric	unit	sees	misses
BLEU	word n-grams	exact phrase matches	any paraphrase; partially-correct words
chrF (Popović 2015)	character n-grams	partially-correct words, morphology	meaning; a fluent sentence built of right characters
COMET (Rei et al. 2020)	learned	meaning, as far as its training data taught it	anything its training data did not contain; and it is slow without a GPU
Human MQM (Multidimensional Quality Metrics) (Freitag et al. 2021)	annotated errors	the widest range, by category and severity	whatever its annotators miss or disagree on — report inter-annotator agreement with the score; and it costs money and days

chrF — an F-score over character n-grams. Unlike BLEU, which is precision only with a length penalty bolted on, chrF combines precision and recall and weights recall \(\beta\) times as heavily; sacrebleu’s default is \(\beta = 2\) over character n-grams of length 1 to 6, which is chrF2. That recall term is why it notices a missing suffix, which is the case an agglutinative language produces constantly. ↩︎

COMET — a neural metric: a model fine-tuned to predict human quality judgements from the source, the hypothesis and the reference. Correlates far better with humans than BLEU, and inherits every bias of the judgements it was trained on. ↩︎

Human evaluation comes in two main forms, and they are worth telling apart. In Direct Assessment (DA), raters score each translation on a 0–100 scale; it is cheap enough for many raters, and the widely used COMET models were trained to predict these scores. In MQM, trained translators mark each error span, give it a category (accuracy, fluency, terminology, …) and a severity, and the score is a weighted count of errors. MQM is slower and more expensive, but Freitag et al. (2021) found it more reliable than crowd ratings and it tells you what went wrong, not only how much.

The 2026 consensus is: report chrF and a neural metric; report BLEU as well because everyone else does; never rank systems on BLEU alone. Kocmi et al. (2024) is the current guidance on how large a difference in each metric has to be before it means anything.

This chapter follows two thirds of that rule and you should hold the missing third against it. Every table below reports BLEU and chrF2 and no neural metric, because everything here ran on one laptop with no CUDA GPU — the MT model uses Apple’s MPS backend. COMET would still run there, only slowly; that slowness is itself the honest cost of the neural metrics rather than an argument against them. Read the rankings below as provisional, and add a COMET column before you ship a decision based on one.

15.4 Decoding for translation

Chapter 18 will argue that sampling is what you want for open-ended generation. Translation is the counterexample, and the contrast is worth having concretely rather than as a rule of thumb.

The difference is in the task. Open-ended generation has many good answers and you want the variety, so sampling from the distribution is the point. Translation also has many valid renderings — that is why the metrics in this chapter compare against a reference, one of many possible correct renderings, rather than against a single truth — but the variety is not what the user asked for, and a sampled token is as likely to be a worse rendering as a different-but-equal one. What you want is the sequence the model itself rates highest. So translation uses beam search: keep the k best partial hypotheses at each step and extend all of them, instead of committing to the single best token.

Here is the whole decoding space on 500 sentences of newstest2016 tr-en with opus-mt-tc-big-tr-en:

kind      value    BLEU    chrF  len ratio  ms/sent
beam          1   29.45   58.01      1.015       51
beam          2   30.06   58.36      1.021       66
beam          4   30.17   58.41      1.022       80
beam          8   30.23   58.42      1.021      119
beam         16   30.25   58.41      1.022      166
beam         32   30.36   58.44      1.020      273
beam         64   30.63   58.58      1.018      459

lenpen      0.2   30.51   58.30      1.007       76
lenpen      0.6   30.44   58.42      1.013       74
lenpen      1.0   30.17   58.41      1.022       73
lenpen      1.4   29.83   58.37      1.031       75
lenpen      2.0   29.25   58.39      1.041       74

sample      0.3   28.95   57.42      1.016       45
sample      0.7   25.90   55.17      1.013       45
sample      1.0   14.34   45.27      1.111       53

How to read the rows. beam 1 keeps a single hypothesis, which is greedy decoding: take the most probable token at every step. The lenpen rows hold the beam at 4 and vary the length penalty, explained below. The sample rows draw each token at random at the given temperature, which rescales the model’s probabilities before the draw: below 1 it sharpens them towards the most likely token, and at 1.0 the draw follows the model’s own distribution. len ratio is output length divided by reference length.

Sampling costs you half the score here

BLEU 30.17 at beam 4 against 14.34 at temperature 1.0. The same model, the same sentences, and 52.5% off the score. Say the score, not the quality: BLEU is an n-gram overlap statistic, not a ratio scale on which half the number means half the translation. Pure sampling picks a token proportional to its probability, so on every token there is a chance of taking a worse one, and across a whole sentence that chance compounds. In a chat reply, the same behaviour is called not being repetitive.

The lesson generalises past translation: decoding parameters are not model settings, they are task settings, and copying a temperature from a chat example into a translation pipeline is a way to lose between four BLEU (at 0.7) and sixteen (at 1.0) without touching the model.

Beam search: the folklore is wrong here

The received wisdom is that quality degrades at large beam widths — the “curse of beam search”, where the model’s highest-probability sequence turns out to be a short, degenerate one. This model does not show it. BLEU rises monotonically from beam 1 to beam 64, and the length ratio stays flat at about 1.02, so nothing is collapsing.

What the data does show is severe diminishing returns:

  beam 1 -> 2      +0.61 BLEU     for  1.3x the time
  beam 2 -> 64     +0.57 BLEU     for  7.0x the time

One extra beam buys as much as the following sixty-two — but before you carry that anywhere, notice what has not been done to it. These are 0.6-point BLEU differences on 500 sentences, and neither has been tested. A paired bootstrap over 500 sentences puts a BLEU interval of roughly a point around each of them, and Kocmi et al. (2024) — cited two sections ago for exactly this purpose — gives thresholds that both of these differences sit below. The defensible reading of this table is that no beam width was shown to be measurably better than its neighbours — a more cautious claim than the raw numbers invite, and the one to write down. Beam 4 stays the right default, not because it is measurably better than beam 2 but because it costs almost nothing and the wide settings cost a lot.

The most likely explanation for the missing curse is the decoder, not the model. generate() scores a beam by its log-probability divided by its length raised to length_penalty, and that argument defaults to 1.0 — full length normalisation, one of the standard corrections for the length bias that produces the curse; Murray and Chiang (2018) analyse that bias and propose a tuned per-word reward as an alternative. This is a hypothesis, not a result: nothing here ran the beam sweep with normalisation switched off. What the next subsection does show is that the knob is a generate() argument, not a property of the Marian weights. To actually go looking for the curse, set length_penalty=0.0 and sweep the beam again; Koehn and Knowles (2017) documented the degradation, and Meister et al. (2020) argue that beam search works well precisely because it does not find the exact highest-probability sequence.

Length penalty moves BLEU without moving quality

The length penalty divides a hypothesis’s score by its length raised to a power, so a higher value tolerates longer output. Sweeping it while holding everything else fixed:

  lenpen 0.2 -> 2.0     BLEU  30.51 -> 29.25    a 1.26-point swing
                        chrF  58.30 -> 58.39    a 0.09-point swing
                        length ratio 1.007 -> 1.041

BLEU moves fourteen times as much as chrF, and it moves in lockstep with output length. Two metrics disagreeing does not by itself settle which one is right — a flat chrF is not proof that quality held. What makes the reading safe here is the third line: the length ratio moves with BLEU and not with chrF. Note which mechanism is moving: the ratio is above 1 in every row, so the output is longer than the reference and the brevity penalty never engages (BP = 1); what moves BLEU is the precision denominator, since every extra word adds n-grams whether or not they match. A human or MQM check would settle it; the length ratio is the cheap evidence that it is worth checking. Tuning the length penalty against BLEU pulls output length toward the reference length; for this system, whose output runs long, that means slightly shorter output (had the output run short, it would push the other way), which is a real and common way to buy a point of BLEU that no reader would notice.

15.5 Why BLEU is worse than usual for Turkish

“BLEU is unfair to morphologically rich languages” is repeated often enough to be folklore. It is true, the mechanism is simple, and the size of the effect is measurable — so here it is measured rather than asserted.

The mechanism, in two numbers

Take the 3,000 professionally translated sentence pairs of newstest2016 and count both sides:

                    words        characters      median words/sentence
  Turkish          46,243           335,110              13
  English          59,661           341,494              18

  the same content needs   1.29x as many WORDS in English
  but only                 1.02x as many CHARACTERS

The same meaning, essentially the same number of characters, and 29% more word tokens in English. Turkish packs into suffixes what English spells out as separate words: evinizden is from your house, one token against three.

That is the whole mechanism. A word-level metric divides a sentence into tokens and asks how many match. English uses 29% more tokens for the same content, so Turkish gives the metric \(1 - 1/1.29 = 22.5\%\) fewer tokens to work with — the two percentages are the same fact read from opposite ends, and only one of them is 29. What follows is that each Turkish token is 1.29× as large a share of the score, and one wrong word costs proportionally more.

A character-level metric sees essentially the same amount of material in both languages — the 1.02 ratio — so it has no such asymmetry.

The mechanism, measured

Predictions are cheap; this one is testable. Take a correct translation — the reference itself, which scores 100 — and introduce exactly one morpheme of error, the same semantic error in both languages: locative becomes ablative, in X becomes from X.

In Turkish that is a suffix: İstanbul'da → İstanbul'dan. To be certain the edit is a suffix and not a corrupted stem, only words with an explicit apostrophe were perturbed, where the morpheme boundary is unambiguous. In English it is a preposition: in → from.

Two things about that pairing that the table cannot show, and both cut against the section’s own BLEU claim.

First, the match is not exact. Turkish 'da is unambiguously locative, while English in is locative only some of the time — interested in, in 2016, in fact all get rewritten too, and some of those become ungrammatical rather than merely wrong.

Second, the two rows are not the same sentences. The filters ran independently, so 264 Turkish sentences qualify against 997 English ones, and the English side very likely carries more than one edit per sentence where the Turkish side carries one.

Both effects make the English perturbation the harsher of the two, which biases the BLEU comparison against the claim below. The Turkish penalty coming out larger anyway is the result; the exact multiplier is not something this design can pin down. The same bias works for the chrF reading, though: with one edit per English sentence, the English chrF penalty would shrink and the 0.57 below would rise. So treat the whole comparison as exploratory. Running the English side on the English halves of the same 264 pairs, editing only the first match in each, is the version to trust, and it is the first thing to fix here.

                              BLEU      chrF     BLEU/chrF     n
  Turkish   X'da -> X'dan    -14.90    -3.05        4.89x      264
  English   'in' -> 'from'   -12.65    -5.34        2.37x      997

Three readings, in order of how surprising they are.

BLEU punishes the Turkish error 1.18× as hard. The direction is what the 1.29 token ratio predicts, and the size is in the neighbourhood — do not read 1.18 as derived from 1.29, because nothing here derives it: the two numbers come from different subsets and a single-edit penalty is not a linear function of the token count. What the section establishes is the sign and the rough scale, which is enough for the operational point below.

chrF punishes the Turkish error only 0.57× as hard. Changing 'da to 'dan adds one character to an eleven-character word; changing in to from replaces a two-character word with a four-character one. At the character level the Turkish error is genuinely smaller, and chrF says so.

The disagreement between the two metrics is more than twice as large in Turkish — 4.89× against 2.37×. This is the number that matters operationally. In English, BLEU charges this error about two and a half times what chrF charges; in Turkish, nearly five times. The two metrics therefore rank Turkish systems differently more easily, so which metric you choose can decide which system you ship.

The whole argument fits in one picture, and every number in it is from the two tables above:

Why the same error costs more in Turkish — and why chrF disagrees

The four cost bars are drawn on one scale, so the crossing is the whole point: the metric built on words punishes the Turkish error harder and the metric built on characters punishes it softer, for a change that means exactly the same thing in both languages.

This is why chrF was proposed (Popović 2015) and why the WMT community moved to character-level and neural metrics for morphologically rich languages. It is also why comparing a Turkish BLEU score against an English one — “our Turkish system gets 23 and our German system gets 34, so Turkish is harder” — is not a valid comparison at all. The metric is measuring different amounts of the sentence in each language.

The second problem: word order

Turkish word order is freer than English. The unmarked order is subject-object-verb, but constituents move for emphasis, and several orders are fully grammatical for the same sentence. English permits far less movement.

The consequence for a word n-gram metric is the one Section 15.3 already showed: moving a phrase destroys every n-gram that crosses the move, while leaving the meaning intact. A correct English paraphrase that reorders one phrase scored 50.27 (57.81 once casing is taken out of the picture). In a language where reordering is a normal stylistic choice rather than an occasional one, a translator’s legitimate variation is charged in a similar way every time.

Both problems have the same fix and it is not a better word n-gram metric: score at the character level, which softens both penalties without removing them, and score with a model that was trained on human judgements.

15.6 LLMs as translators

A general-purpose model can translate without being asked to be a translator. The question is not whether it works but what it costs and where it wins, so here is the head-to-head: qwen3.5:9b prompted zero-shot against opus-mt-tc-big, on the identical 400 sentences. opus-mt-tc-big is really two 237M-parameter models, -tr-en and -en-tr, one per direction, and each does nothing else.

Before the table, the two things that decide how to read it. State the decoder. Section 15.4 just measured what sampling costs a translator — 30.17 BLEU at beam 4 against 14.34 at temperature 1.0, the same model and the same sentences — and ollama samples by default. Its global fallback is temperature 0.8 with top-\(k\) 40, but a library model can ship its own settings, and the qwen3.5:9b entry lists temperature 1.0, top-\(k\) 20, top-\(p\) 0.95 and a presence penalty of 1.5 at the time of writing. Unless the request overrode them, the LLM row was sampled at the very temperature that cost the Marian model half its BLEU — softened somewhat by top-\(k\) and top-\(p\), which restrict the draw to the most likely tokens (Chapter 18 covers them). Benchmarking an LLM at a chat default and concluding that LLMs translate badly is the most common way to get this comparison wrong, and the LLM row below is only interpretable if its decoding settings are published beside it. They are not, and neither are the prompt text or whether the thinking mode of qwen3.5:9b, a reasoning model, was on; that is a gap this chapter owes you: treat the LLM row as a measurement of this model rather than as its ceiling until it has been re-run with the decoder pinned.

And state the precision. ollama serves an untagged model 4-bit quantised by default, while opus-mt-tc-big runs unquantised through transformers. So the headline is not 237M against 9B but 237M unquantised against a 4-bit 9B. Quantisation plausibly hurts more when the model must generate a morphologically rich language, which would put its cost in the en-tr column where the gap is largest — but that has not been measured here. Both the sampling and the quantisation push the LLM’s score down, so the 6-BLEU figure more likely overstates than understates the gap between the models themselves; contamination, discussed below, pulls the other way.

wmt16 tr-en                          BLEU    chrF   s/sentence
  opus-mt-tc-big   greedy           31.44   59.53      0.013
  opus-mt-tc-big   beam-4           32.33   60.08      0.024
  qwen3.5:9b       zero-shot        29.37   58.32      1.088
  copy the source (control)          2.09   21.11

wmt16 en-tr                          BLEU    chrF   s/sentence
  opus-mt-tc-big   greedy           22.50   56.70      0.016
  opus-mt-tc-big   beam-4           23.13   57.41      0.064
  qwen3.5:9b       zero-shot        17.10   53.67      2.620
  copy the source (control)          2.10   19.87

The control row is there because a metric you have not sanity-checked is a metric you do not understand: leaving the source untranslated scores 2.09, so the numbers above are measuring translation rather than incidental overlap.

One control the table cannot run, and one warning about its last column. newstest2016 has been public since 2016 — in sacrebleu, on GitHub, in HuggingFace datasets, with its reference translations. Any LLM trained on a 2020s web crawl has plausibly seen it, and that contamination inflates the LLM’s score. Which means it works against nothing in the conclusion below: the dedicated model wins despite the benchmark possibly sitting in its opponent’s training data. The dedicated side is cleaner, since Helsinki-NLP excludes WMT test sets from OPUS-MT training, but note that “zero-shot” here means no examples in the prompt and not never seen this data. If you need an uncontaminated number, use a test set published after your model’s cutoff.

As for the seconds: they are not measured the same way as the ms/sent column in Section 15.4, which is why the same model at the same beam width appears there at 80 ms and here at 24. The same holds for BLEU: the same model and setting score 30.17 there and 32.33 here because the two tables use different subsets (the first 500 and the first 400 sentences). Compare within a table, never across, and when you publish a latency say whether it is one request or a batch.

Three things to take from this.

The small dedicated model wins both directions. 237M parameters beat 9B, by 3 BLEU into English and by 6 into Turkish, at 41× to 164× the speed depending on direction and beam width — 41× for the slowest honest comparison (beam-4 into Turkish) and 164× for the fastest (greedy into Turkish); at the beam-4 setting you would actually ship into English it is 45×. A model trained on nothing but this language pair is very hard to beat on this language pair.

The gap is twice as large translating into Turkish. That is the asymmetry to expect and to check for: a general multilingual model has seen far more English than Turkish, and generating a morphologically rich language is harder than parsing one. Never assume your MT quality is symmetric — measure both directions.

Translating into Turkish is harder for every system here. opus-mt drops from 32.33 to 23.13 when the target changes. Part of that is genuine difficulty and part is Section 15.5’s measurement artefact: the same quality of translation scores lower when the target language gives BLEU 22.5% fewer tokens to match. chrF, which does not have that bias, drops far less — 60.08 to 57.41. The same check applies to the previous point: in chrF the LLM’s deficit is 1.76 into English and 3.74 into Turkish, so the doubling is not only BLEU’s harshness on Turkish.

What an LLM buys that the dedicated model cannot

The table makes the LLM look strictly worse. On sentence-level news translation it is. The reasons to reach for one anyway are things this benchmark cannot express, because WMT is a list of isolated sentences:

• Document context. A dedicated sentence-level model translates each sentence independently and so cannot resolve a pronoun whose antecedent was two sentences ago, or keep a consistent formality register across a page. Turkish makes this concrete: o means he, she or it, so translating o bir doktor into English forces a choice the sentence alone cannot make. Without context, a system falls back on whatever its training data made most frequent, which is how o bir doktor becomes he is a doctor — a well-documented source of gender bias in MT (Prates et al., 2019 tested exactly this with Turkish and other languages whose pronouns carry no gender).
• Terminology adherence. Put the glossary in the prompt and the model will mostly follow it. Enforcing terminology in a dedicated model means constrained decoding or retraining.
• Register and audience. Translate this formally, for a legal audience, keeping the second person plural is an instruction, not a model.
• Any pair, immediately. There is no opus-mt for most of the pairs a real product needs, and no data to train one. There is a middle option worth checking first: massively multilingual dedicated models such as M2M-100 (Fan et al., 2021) and NLLB-200 (NLLB Team, 2022) translate directly between 100 and 200 languages from one model. They are still far smaller than an LLM, but read the licence before shipping: the NLLB-200 weights are released for non-commercial use.

Vilar et al. (2023) is the systematic study of how much prompting choices matter here; the short version is that examples in the prompt help, and that the quality of the examples matters more than the number.

The practical architecture in 2026 is usually neither alone: a dedicated model for volume, an LLM for the documents where context and terminology matter, and a router in front. The cost table decides the split — 0.024 seconds against 1.088 is not a rounding difference at scale, it is the difference between one machine and forty-five.

15.7 Low-resource translation

Turkish–English is not low-resource; OPUS has millions of pairs. But most translation problems that arrive in industry are: a language pair nobody publishes data for, or — far more often — a domain nobody publishes data for. Legal Turkish, clinical Turkish, and the internal vocabulary of one company are each low-resource regardless of how much news text exists.

The resource that does exist is monolingual target text. You have your own documents; you do not have their translations.

Back-translation (Sennrich et al. 2016) turns that into training data. Train a rough system in the reverse direction, use it to translate your monolingual target-language text into the source language, and add those synthetic pairs to training with the synthetic side as input.

The synthetic source may be poor; it does not matter as much as you would expect, because the model only has to learn to produce the target side, and the target side is genuine human text.

That asymmetry is the whole trick and the reason it works: the noisy half is on the input, where the model is allowed to be robust, and the clean half is on the output, where it is being taught what to write.

Two adjacent moves worth knowing:

• Fine-tuning on in-domain pairs. A few thousand genuine in-domain pairs usually beat a large amount of synthetic data. Collect them from translation memories before generating anything.
• Terminology injection. Domain adaptation frequently is not a fluency problem at all but a vocabulary one: the model translates hisse as share when your house style is stock. That is a constraint to enforce at decoding or in a prompt, not a reason to retrain — and it is where Section 15.6 has a genuine advantage.

Terminology drift is the failure that gets a translation project cancelled, and it is invisible to every metric in Section 15.3: the model’s word and your house word are both plausible translations, so BLEU against a generic reference cannot see the difference. Measure it directly — take your glossary, and count how often the system used your term.

15.8 Translate-train, translate-test, or neither

You have a task — classification, retrieval, extraction — in Turkish, and the best model or the labelled data is in English. Three options, and the choice is an error-propagation question rather than a modelling one.

Three pipelines, three failure profiles

approach	what happens	where errors accumulate
Translate-train	translate the English training data into Turkish once, train a Turkish model	translation noise is baked into training, but the model sees it consistently and can partly learn around it
Translate-test	keep the English model, translate every incoming Turkish input at inference	a translation error at request time is a wrong prediction, every time, with no chance to recover
Neither	a multilingual model applied directly (Chapter 8)	no translation error at all; whatever the multilingual model gives up on Turkish

translate-train — machine-translate the labelled training data into the target language, then train there. The translation cost is paid once, offline, and the noise it introduces is at least consistent. ↩︎

translate-test — keep the model in its original language and translate each incoming request into it. Nothing to retrain, and every user-facing prediction now depends on a translation you did not check. ↩︎

The arithmetic that decides it: translate-test pays the translation cost per request, forever, and inherits the translation’s error rate on every prediction. Translate-train pays once. If your system is a chatbot handling a thousand requests a second, that is not a close call.

Two more considerations that decide it in practice:

1. Which direction is your MT better in? They are not symmetric. Measure both before assuming.
2. What is the task? Translate-test destroys anything that depends on surface form — Chapter 13’s span offsets do not survive translation, so extraction can use it only with an extra step that maps the predicted spans back to the original text through a word alignment, and that step adds its own errors. Document classification survives it comparatively well.

Chapter 16 measures the third row for Turkish — a multilingual model against a monolingual Turkish one on the same task — and finds no general verdict, only a protocol. Translate-train and translate-test are not measured in this book; this chapter’s contribution is the mechanics and the arithmetic, and exercise 7 is where you measure translate-test for your own task (for translate-train, translate the same task’s training data, train on it, and compare on the same Turkish test set).

15.9 Intuition checks

Answer each one before you open it. The first five are questions this chapter settled with a measurement rather than an argument; the sixth is settled by arithmetic.

Note1. Your Turkish system scores 23 BLEU and your German one scores 34. Is Turkish harder?

You cannot conclude that, because BLEU is not measuring the same amount of each sentence in the two languages. English needs 29% more word tokens for the same content, so Turkish gives a word n-gram metric 22.5% fewer slots to match and each error costs more of the score.

Measured here: the same semantic error — locative to ablative — costs 14.90 BLEU in Turkish and 12.65 in English. Some of your eleven-point gap is the metric.

Report chrF, which sees essentially the same amount of material in both languages (a 1.02 character ratio against a 1.29 word ratio), and a neural metric. Then compare.

Note2. You copy the temperature from your chat pipeline into your translation pipeline. What happens?

At a chat-style temperature of 1.0, the score falls by more than half (at 0.7, by about four points). Measured on the same model and sentences: beam-4 scores 30.17 BLEU, temperature 1.0 sampling scores 14.34 — a large drop in BLEU, not a measured halving of translation quality.

Translation admits many valid renderings, but you are not asking the system for variety here — you want the sequence the model rates highest, which is a statement about model probability rather than about quality, and beam search is a heuristic for finding it rather than an exact search. Sampling deliberately introduces variation, and across a sentence the chances of drawing a worse token accumulate. The setting that makes a chat reply feel natural cost about 16 BLEU on this model and this test set — evaluate decoding on the task you actually intend to deploy.

Note3. Is a wider beam worth it?

Almost never past 4. On this model, going from beam 1 to beam 2 buys +0.61 BLEU; going from beam 2 all the way to beam 64 buys +0.57 more, for seven times the compute.

And check the folklore before repeating it: the “curse of beam search”, where quality degrades at large widths, does not appear here at all — BLEU rises monotonically to 64 and the length ratio stays flat. The most likely reason is the decoder, not the model: generate() normalises a beam’s score by its length by default — length_penalty=1.0 — which is one of the standard corrections for the length bias behind the curse that Murray and Chiang (2018) analyse. Turn it off (length_penalty=0.0) and sweep again if you actually want to look for the disease. And note that the rises are small enough that none of them was tested for significance, so “rises monotonically” describes the point estimates and nothing more.

Note4. Your BLEU went up 1.2 points after you tuned the length penalty. Did the system improve?

Probably not. Sweeping the length penalty from 0.2 to 2.0 moves BLEU by 1.26 points and chrF by 0.09, while the output length ratio moves from 1.007 to 1.041.

BLEU is responding to how many words you produced, through its brevity penalty and its precision denominator; in this sweep the ratio stays above 1, so the brevity penalty is inactive and only the precision denominator moves. Tuning against BLEU pulls output length toward the reference, which for this system, whose output runs long, means slightly shorter output. If a change moves BLEU and leaves chrF flat, suspect a metric artefact before you call it an improvement, and read the outputs to find out which it is.

Note5. A 9B LLM or a 237M dedicated model?

Measured, on identical sentences: the dedicated model wins by 3 BLEU into English and 6 into Turkish, at 41–164× the speed. On sentence-level news translation, under this evaluation setup, the dedicated model is the better choice on every axis measured.

Reach for the LLM when the benchmark cannot express what you need: document context for pronouns and register, a glossary in the prompt, a language pair nobody publishes a model for. Those are real and this table cannot show them — WMT is a list of isolated sentences.

Note6. Translate-train or translate-test?

Translate-test pays the translation cost on every request forever and turns every translation error into a wrong prediction. Translate-train pays once, offline, and the noise at least arrives consistently so the model can partly learn around it.

Two things decide the rest: which direction your MT is actually better in — measure, they are not symmetric — and whether your task depends on surface form. Span extraction does not survive translate-test without an extra alignment step, because the offsets do not transfer. Document classification survives it comparatively well.

15.10 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s measurements make answerable.

Testing whether you understand the task

Q: Why was attention in its modern form proposed for translation and not for something else?

Because the encoder-decoder RNN compressed the entire source into one fixed-length vector, and translation makes that bottleneck obvious: quality collapsed as sentences got longer, in the specific pattern of translating the start and then drifting. Attention removed the compression by keeping one vector per source token. Say also why it fit: translation has output-to-input correspondence, so a weighted average that can concentrate on one source position is the right interface — which is why early attention maps looked like word alignments and why the same reading of a deep language model cannot be taken for granted.

Q: What does BLEU actually compute?

Modified n-gram precision for n = 1 to 4, geometric mean, times a brevity penalty. Be able to work an example, and say that on a test set the counts are summed over sentences before the formula is applied. Then say what it cannot see: any correct paraphrase, the difference between a legitimate reordering and a wrong one, and — by default — anything that differs in case. A correct English translation that moves one phrase scores 50.

Q: When would you not report BLEU?

Never — report it, because comparability matters and its absence looks evasive. The question is what you report with it. For a morphologically rich target, chrF, because it gives partial credit for a right stem with a wrong suffix. And a neural metric, because it is the only one of the family that sees meaning. Ranking systems on BLEU alone is the failure.

Testing whether you have shipped one

Q: Your MT quality dropped after a model upgrade, but only for one customer. Debug it.

Almost certainly terminology, not quality. Their house vocabulary changed under them — the new model translates their term differently — and a corpus-level reference metric will not surface it, because both words are valid translations and the difference is a rounding error across the test set. It is not invisible in principle: a term-focused test set built from their glossary, or MQM with a terminology error category, sees it directly. What will not see it is the aggregate BLEU you were already running. Measure it directly: take their glossary and count adherence. Then fix it at decoding or in a prompt, not by retraining.

Q: You need translation for 40 language pairs and have a budget.

Route. A dedicated model per pair where one exists and volume justifies it, an LLM for the long tail and for documents needing context or terminology control. The number that decides the split is latency-per-sentence: measured here, 0.024 seconds against 1.088 is the difference between one machine and forty-five. Say also that you would measure both directions separately, because they are not symmetric — 32.33 out of Turkish against 23.13 into it, on the same model family and corpus.

Q: How would you build MT for a domain with no parallel data?

Back-translation, and be able to say why it works: train a rough reverse-direction system, use it to translate your monolingual target text into the source, and train on those pairs with the synthetic side as input. The noisy half is on the input, where robustness is expected, and the clean human half is on the output, where the model is being taught what to write. Then say what you would try first, which is finding a few thousand genuine in-domain pairs in a translation memory — they usually beat a large amount of synthetic data.

15.11 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below.

The stack these numbers came from. Python 3.14.5, Apple Silicon

library	version used	what it is for here
sacrebleu	2.6.0	BLEU, chrF, and — as much to the point — downloading and identifying the test set
transformers	5.15.1	the Marian model, and every decoding setting in Section 15.4
torch	2.13.0	the MPS backend it ran on
ollama	0.24.0	serving qwen3.5:9b at Q4_K_M, its default quantisation, for the head-to-head

Worth knowing about, not used above:

The wider ecosystem, versions as of writing

library	current version	when you would reach for it
unbabel-comet	2.2.7	the neural metric the WMT community now leads with; wants a GPU (on a CPU it runs, slowly) and a model download, and is worth both
ctranslate2	4.8.1	serving Marian and similar models 2–4× faster than PyTorch eager mode, with int8 quantisation — the obvious next step after the latency table above
sentencepiece	0.2.2	the subword vocabulary these models are built on (Section 8.3)
sacremoses	0.2.0	the Moses tokenizer and truecaser, still needed when reproducing older MT results
stanza	1.14.0	sentence splitting and morphological analysis for Turkish, when your input arrives as documents rather than sentences
nltk	3.10.3	has its own BLEU, which does not match sacrebleu’s — a live example of why Post (2018) had to be written

15.12 Lab: build and evaluate a Turkish–English pipeline

Pick a domain you have text for — your own documents, a public corpus in your field, product descriptions, support tickets. You need about 100 source sentences and, for the evaluation half, 30 with human translations. Producing those 30 yourself is part of the lab.

Step 1 — name the test set, before anything else

Write down exactly which sentences you are evaluating on and where they came from. If you use a public set, name it the way sacrebleu does. Half the irreproducible MT results in the world are irreproducible because this step was skipped.

Step 2 — translate with a dedicated model, both directions

opus-mt-tc-big-tr-en and -en-tr, beam 4. Record seconds per sentence. Report BLEU with its signature and chrF. The two directions will not match; note the gap before you read Section 15.5’s explanation of how much of it is the metric.

Step 3 — translate with an LLM, same sentences

Put your glossary in the prompt, if you have one. Record seconds and tokens per sentence, because this is where the cost argument gets made.

Step 4 — the control

Score the untranslated source against the reference. It should be near zero. If it is not, your metric is picking up something other than translation quality — shared proper nouns, numbers, or a language pair closer than you thought — and every number above needs re-reading.

Step 5 — sweep the decoder

Beam 1, 4, 16, and sampling at temperature 0.7. Plot BLEU against seconds per sentence. You are reproducing Section 15.4’s table on your own data, and the question is where your curve flattens.

Step 6 — read twenty outputs

Properly, side by side with the source. Rank the two systems yourself before looking at the scores again. Then find one sentence where your ranking and BLEU’s disagree, and write down which of the two was wrong and why.

Step 7 — measure terminology, if you have a glossary

For each glossary term, count how often each system used your word rather than a synonym. This is the number that decides a real localisation project and it appears in none of the metrics above.

What to hand in

One table — systems against BLEU, chrF, seconds per sentence, terminology adherence — plus the twenty-sentence reading, the disagreement you found, and a paragraph on which system you would ship and at what volume the answer changes.

15.13 Exercises

1. BLEU by hand. Compute BLEU for one sentence pair with a calculator, then check against sacrebleu. Now change one inflectional suffix in the candidate and recompute. Report how much BLEU moved, how much chrF moved, and whether the meaning moved as much as either.

2. The reordering tax. Take a correct translation and move one phrase to a different legal position. Compute BLEU before and after. Then do it in Turkish, where reordering is more freely available, and compare the two penalties.

3. Find the curse, or fail to. Sweep beam width from 1 to 128 on a model of your choice and plot BLEU and output length. If quality degrades, say at what width; if it does not, find out whether your model normalises by length during search.

4. Length-penalty artefact. Tune the length penalty to maximise BLEU on a development set. Report the BLEU gain, the chrF gain, and the change in output length ratio. Decide whether you would ship the tuned value.

5. Direction asymmetry. Measure the same model family in both directions on the same corpus. Explain how much of the gap is difficulty and how much is Section 15.5’s metric effect, using the word and character ratios of your own data.

6. Back-translate something. Take 2,000 monolingual sentences in your domain, translate them into the source language with an off-the-shelf model, and fine-tune on the synthetic pairs. Report the change on a genuine held-out set — and check whether it helped on the domain and hurt off it.

7. Translate-test error propagation. Take a classification task, translate the test inputs, and measure accuracy. Then hand-check the ten misclassified documents: how many are model errors and how many are translation errors? That ratio is the argument for or against the pipeline.

15.14 Where to go deeper

First, print your BLEU signature. If you cannot say which tokenizer, casing and smoothing produced a number, that number cannot be compared to anyone else’s — which is the entire argument of Post (2018).

Beyond the papers

resource	what it is
sacrebleu	the reference implementation and the signature mechanism; also the easiest way to get chrF and TER right
OPUS-MT	the dedicated models that beat a 9B LLM in this chapter, with training recipes published
Stanford CS224N	the machine-translation lectures, including attention derived from the MT problem it was invented for
The Annotated Transformer	the architecture as runnable code, written around a translation task
ACL Anthology	every WMT findings paper, free — the shared-task reports are the honest state of the field each year
Coding a Transformer from scratch in PyTorch — Umar Jamil	builds encoder–decoder translation end to end, which is the architecture this chapter’s dedicated models still use

The primary sources

topic	source
the metric, and how to report it	Papineni et al. (2002); Post (2018)
character n-grams instead	Popović (2015)
a learned metric	Rei et al. (2020)
what human evaluation actually measures	Freitag et al. (2021)
how to read a WMT result without over-claiming	Kocmi et al. (2024)
where neural MT began	Sutskever et al. (2014); Bahdanau et al. (2015)
and what it replaced	Brown et al. (1993); Koehn et al. (2003)
the failures that survived the neural turn	Koehn and Knowles (2017)
back-translation, still the strongest data trick	Sennrich et al. (2016)
prompting an LLM to translate	Vilar et al. (2023)

15.15 What to remember

• Attention was a repair to a translation bug. The encoder-decoder RNN squeezed a whole sentence into one fixed vector, quality collapsed on long sentences, and keeping one vector per source token fixed it. The architecture that now runs everything started there.

• BLEU gives a correct translation 50.27 when it moves one phrase and differs in case on two words. It is modified n-gram precision with a brevity penalty and nothing more; it cannot see paraphrase or meaning, and cannot tell a legitimate reordering from a wrong one. Report it, never rank on it alone, and always report its signature.

• In this corpus the English/Turkish word-token ratio is 1.29. The same content needs 1.29× as many word tokens in English but only 1.02× as many characters, so a word n-gram metric has 22.5% fewer slots in Turkish (\(1 - 1/1.29\)) and each error costs proportionally more. In an exploratory test that puts a comparable error into each language, on different sentence subsets, BLEU charges 14.90 in Turkish against 12.65 in English, while chrF charges 3.05 against 5.34. The two metrics disagree 4.89× in Turkish and 2.37× in English — which is why the metric you choose can decide which system you ship.

• Never compare BLEU across languages. “Turkish scores 23 and German 34” is not a statement about difficulty; the metric is measuring different fractions of each sentence.

• Decoding parameters are task settings, not model settings. Beam 4 scores 30.17; temperature 1.0 sampling scores 14.34 on the same model and sentences. The setting that makes chat feel natural took 52.5% off the score here.

• Check folklore before repeating it. The “curse of beam search” did not appear here at all — BLEU rose monotonically to beam 64 with a flat length ratio. What did appear is brutal diminishing returns: beam 1→2 buys +0.61, beam 2→64 buys +0.57 for seven times the compute.

• A BLEU gain with a flat chrF is a reason to suspect a metric artefact. Sweeping the length penalty moves BLEU 1.26 points and chrF 0.09, tracking output length; confirm by reading outputs before you believe the gain. Tuning against BLEU pulls output length toward the reference; for a system whose output runs long, as here, that shortens it.

• A 237M dedicated model beat a 9B LLM in both directions, by 3 BLEU into English and 6 into Turkish, at 41–164× the speed. Reach for the LLM when you need what the benchmark cannot express: document context, glossary adherence, register, or a pair nobody publishes a model for.

• Measure both directions. 32.33 out of Turkish, 23.13 into it, same model family and corpus. Never assume MT quality is symmetric; the asymmetry decides pipeline design.

• Translate-test pays forever and translate-train pays once. And translate-test breaks anything that depends on surface form — span offsets do not survive translation without an extra alignment step.

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. “Neural Machine Translation by Jointly Learning to Align and Translate.” International Conference on Learning Representations.

Brown, Peter F., Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. “The Mathematics of Statistical Machine Translation: Parameter Estimation.” Computational Linguistics 19 (2): 263–311.

Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. “Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.” Transactions of the Association for Computational Linguistics 9: 1460–74.

Kocmi, Tom, Vilém Zouhar, Christian Federmann, and Matt Post. 2024. “Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies.” Proceedings of ACL.

Koehn, Philipp, and Rebecca Knowles. 2017. “Six Challenges for Neural Machine Translation.” Proceedings of the First Workshop on Neural Machine Translation.

Koehn, Philipp, Franz Josef Och, and Daniel Marcu. 2003. “Statistical Phrase-Based Translation.” Proceedings of NAACL-HLT.

Meister, Clara, Ryan Cotterell, and Tim Vieira. 2020. “If Beam Search Is the Answer, What Was the Question?” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2173–85.

Murray, Kenton, and David Chiang. 2018. “Correcting Length Bias in Neural Machine Translation.” Proceedings of the Third Conference on Machine Translation, 212–23.

Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “BLEU: A Method for Automatic Evaluation of Machine Translation.” Proceedings of ACL.

Popović, Maja. 2015. “chrF: Character n-Gram f-Score for Automatic MT Evaluation.” Proceedings of the Tenth Workshop on Statistical Machine Translation.

Post, Matt. 2018. “A Call for Clarity in Reporting BLEU Scores.” Proceedings of the Third Conference on Machine Translation.

Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. “COMET: A Neural Framework for MT Evaluation.” Proceedings of EMNLP.

Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. “Improving Neural Machine Translation Models with Monolingual Data.” Proceedings of ACL.

Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. “Sequence to Sequence Learning with Neural Networks.” Advances in Neural Information Processing Systems 27.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30.

Vilar, David, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. “Prompting PaLM for Translation: Assessing Strategies and Performance.” Proceedings of ACL.