# Translation page sources ## Recorded model output The page's SIE outputs were recorded through SIE Serverless, the production API at `https://api.superlinked.com`, on 2026-10-25: the page cases and the study's Hy-MT2 arm. Both repeat earlier runs on direct sie-servers on Modal, with the same request bodies byte for byte. SIE's staging environment could not place either translation model on an L4 on the days of those runs, and the registration asks for a run through SIE's cloud after a direct one. The direct runs stay the record. - Page cases (`cases.json`): through Serverless, 2026-10-25. MADLAD ran all 12 cases; Hy-MT2 ran 10. The direct run: one direct sie-server 0.10.0 on one Modal L4, 2026-10-13. - Study (WMT24++, below): the Hy-MT2 arm through Serverless, 2026-10-25, all 1,488 segments. The direct runs: two direct sie-servers on two Modal L4s (4 CPU, 16 GiB each), one per model, 2026-10-14; the study files do not record the server version. The study's MADLAD arm ran through Serverless on 2026-10-26, all 1,488 segments; it is not the page's row. - Runner: `run.py` (Python standard library only), which sends the same request bodies to Serverless and to a direct sie-server - Model: `google/madlad400-3b-mt` - Hugging Face revision in the served model config: `fa184c675da0b5c9e1c8694fccd4e12e2d422094` - Recorded with the float32 CTranslate2 artifact `superlinked/madlad400-3b-mt-ctranslate2-float32` at `2cac811471ffb01ea6502ab4d317d5a7e9cb6f7d`. Serverless serves `superlinked/madlad400-3b-mt-ctranslate2-bfloat16` at `3e2bd4703cc3f17c08baecfe771f3e30f539d071`, which did not load in the Modal sandbox on the days of the direct runs. The page cases ran again through Serverless, on the artifact it serves, and so did the study's MADLAD arm, on 2026-10-26 (MADLAD-400 3B through Serverless, below). - On 2026-10-20 the study's MADLAD arm ran again with that bfloat16 artifact, outside the registration: the same 1,488 requests, on one direct sie-server 0.10.1 on a Modal L4. - Model card: - Paper: - Endpoint: `https://api.superlinked.com/v1/generate/google__madlad400-3b-mt` for the page cases and the study's run of 2026-10-26; the direct runs used `http://127.0.0.1:8080/v1/generate/google__madlad400-3b-mt` - The response carries no revision. - Run manifest SHA-256: page cases `bba78609e668863e67f3ccbfbf9e9e1f7a7ad0df3805511792353fbe25d248dd` (direct run `e73c3e1eeb45150e1ed93686a49a0627f46a5d8039f2925fcd583693463be6e0`), study `505b161a1269ea8d9df96e6e53e248476cac9b7209893ba7581ff3cb804b3392` (direct) and `b9d3924a3878e44b9484d0bc0d336d7c26372f0a68122d7fa8b79cb4fe291a42` (through Serverless, 2026-10-26) - Model: `tencent/Hy-MT2-1.8B` - Hugging Face revision in the served model config: `9a341cd1b679d3efd23b46e847b01745a71ed792`, on SGLang in bfloat16 - Model card: - Endpoint: `https://api.superlinked.com/v1/generate/tencent__Hy-MT2-1.8B`; the direct runs used `http://127.0.0.1:8080/v1/generate/tencent__Hy-MT2-1.8B` - The response carries no revision. - Run manifest SHA-256: page cases `c44640770e9206aedf5098b7ae6b63a9e4004c9a80fb429f5b43e739df6bebfa` (direct run `f6da35baa757bd8bd6d4f1cf1158dfe2cafc212bfe0b55c841261ce1c037bbf6`), study `41fff7d6b8ab5d22d526c46fc1d7fc34a95f7f4683e34b13c5f97cf5bd936b9e` (direct run `8595ba43435ab59c8dca3375d4204e5b6085d579cfcd672166d24b9a6cd788b7`) - Not sent: the two Swedish (`sv_SE`) page cases. Swedish is not in Hy-MT2's served language table, so Hy-MT2 has no output for them. The served model configs that name these revisions are `packages/sie_server/models/google__madlad400-3b-mt.yaml` and `packages/sie_server/models/tencent__Hy-MT2-1.8B.yaml` in the public superlinked/sie repository. Every English input on the page is verbatim source text, and every human reference is the translation the same publisher shipped for the same text. The request renders the prompt with the Serverless translation contract of each model. MADLAD gets the target-language tag, one space, then the English text. Hy-MT2 gets a one-line instruction that names the target language, then the English text, inside the model's user and assistant markers. `max_new_tokens` comes from the input-length bucket and stops at the model's output cap: 512 tokens for MADLAD, 1024 for Hy-MT2. `temperature` is 0, as in the site snippet. The page shows the raw `text` field of each response. It does not edit, reflow or correct a translation. Fine-print marks connect the same fact in the source, the model output and the human reference: a number, a time, a phone number, an email address or a negation. They are a display alignment chosen by a person reading the texts, not model attribution. The website does not serve the raw model files. The runner, the cases file, the page cases' recorded requests and responses, and the study's run manifests (each response's SHA-256 and token usage) are kept for verification outside this published directory, and site CI compares the page data with them. The study is public on Hugging Face (Translation study, below): the sample, the outputs and token usage of every arm it publishes, the scores and the scorer. Its raw response bodies are not published; they are available on request in that dataset's Community tab. ## What the page shows - Hero and API: the Android notification-permission message (below) on `tencent/Hy-MT2-1.8B`, the model SIE Serverless picks for Japanese, into Japanese (`ja_JP`, with Hy-MT2's instruction naming Japanese), 256 output tokens at most, temperature 0, recorded through Serverless on 2026-10-25. The response's `finish_reason` is `stop`; it used 50 input and 51 output tokens. The direct run of 2026-10-13 returned the same text. The mark is "Starting in Android 13" in the English, "Android 13以降" in SIE's output and "Android 13 以降" in Android's own Japanese. The output has no slip: no grammar error, and standard terms. Android's Japanese calls the permission "権限", Android's word for an app permission; Hy-MT2 writes "許可", the everyday word for permission. The page shows the output as returned. - Headline: SIE's row (below) costs $3.1472 per 1M source characters on the study workload; Google Cloud Translation lists $20 per 1M characters. That is 84.3% cheaper, written "84% cheaper" (rounded down). - Proof: SIE's row against Google Cloud Translation on the study (below). Quality is chrF++ over the 1,488 segments: SIE 48.1 (48.15, rounded down) and Google Cloud Translation 51.8 (51.78). Google scores higher, and the page makes no quality claim. Cost per 1M characters: SIE $3.15 ($3.1472, rounded up) and Google $20.00. The proof's heading, "84% cheaper than Google Cloud Translation", comes from those two bars: 84.25%, rounded down. GPT-6 Luna is left off the chart by an editorial choice; its results are reported here. **Run this eval** opens the study's README on Hugging Face. - Pain: one Google Cloud Translation output from the study, picked by a rule frozen before any output was read, beside Google's bill for a billion characters at its list price, and two published reports of translation changing a number (Pain chapter, below). Nothing in the chapter is SIE's. The other page cases were recorded and are not displayed. Hy-MT2's German air passenger notice has two slips: a stray article in "den Text mit den Ihren Rechten", and "Abfertigungsgang" for the boarding gate, where the official German has "Flugsteig"; "Abfertigungsgang" is not a standard German term. Hy-MT2 translated the Norco seatstay as a seat ("siège"), wrote "ne inclut" for "n'inclut" in both French Android cases, spelled "Widerruffrist" for "Widerrufsfrist" in the German withdrawal notice and gave "umowa" the wrong gender ("z tego umowy") in the Polish one. In the Muc-Off recall it wrote "e-mail" and "8:30 à 17:00", where the official French has "courriel" and "8h30 à 17h00": Canadian French writes "courriel" for email, and an "h" in clock times. In the Brazilian Portuguese Android case it dropped the article and capitalised the feature name: "Use Desbloqueio por Impressão Digital", where Android has "Use o Desbloqueio por impressão digital". The Mexican Spanish Firefox case has no slip, but its one checkable fact is a negation; the page shows the Japanese case, whose fact is a version number that reads the same in all three texts. The two Swedish cases are outside the study's 31 locales. ## Translation study: WMT24++ The study's files are at [superlinked/sie-task-evidence @580893f2, translation/wmt24pp-study-2026-10](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/580893f2c7c7cac20ac4765e5cb7a17e6260e377/translation/wmt24pp-study-2026-10). The folder holds the sample with its human references, every published arm's outputs as returned, the three score files, the registered scorer, a script that recomputes all three byte for byte, and the registration with its five amendments (internal references redacted). It includes MADLAD-400 3B's run through Serverless (`data/outputs/madlad400-3b-mt-cloud.jsonl`, scored in `scores/scores-madlad-cloud.json`) and the pain chapter's rule files in `study/pain-v1/`: the frozen rule in `study/pain-v1/frozen/` (`RULE.md`, `detector.py`, `FROZEN.json`), the detector's flags in `study/pain-v1/run/` and the arm-blind confirmation in `study/pain-v1/confirm/`. MADLAD-400 3B's float32 and bfloat16 runs and the registered row are not in it, for the reason under Results. The page's **Run this eval** link is the folder's [README](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/580893f2c7c7cac20ac4765e5cb7a17e6260e377/translation/wmt24pp-study-2026-10/README.md); its "Reproduce the scores" steps run that script with one command. Four arms keep their study names there: `google-nmt`, `gpt-6-luna-effort-none`, `gpt-6-luna-effort-high` and `hy-mt2-1.8b` (the direct run). The Serverless recording, SIE's row, which the registration calls `hy-mt2-1.8b-production`, is published there as `hy-mt2-1.8b-cloud`, and MADLAD's Serverless run, `madlad400-3b-mt-production` in the registration, as `madlad400-3b-mt-cloud`. Registered before any study request was sent. Five dated amendments followed the runs. On 2026-10-20 the page began to chart Google Cloud Translation alone (below). On 2026-10-25 the second amendment changed two things: the page's row became the per-language choice SIE Serverless served that day, and SIE is priced at its Serverless list prices on that date instead of its list prices on the run date (both below). The third, the same day, changed only which recording of that row the page reads: the Hy-MT2 arm's run through Serverless, the run the registration asks for after a direct one (below). The fourth, on 2026-10-28, adds the pain chapter's rule (Pain chapter, below). The fifth, on 2026-10-29, publishes the results of MADLAD-400 3B's run through Serverless, which is under the flag gate, and keeps its two earlier runs withheld (MADLAD-400 3B through Serverless, below). Nothing else in the registration changed. - Question: on one frozen WMT24++ sample, how do SIE's two translation models compare with Google Cloud Translation (NMT) and GPT-6 Luna on chrF++, and what does each cost per 1M source characters? - Dataset: Google, WMT24++ (Deutsch et al., 2025), dataset revision `fd7405c06494bc66a57b25f55d217a72f96e60dc`, English source, the post-edited `target` as the reference - URL: - Paper: - Licence: Apache License 2.0. The texts belong to their publishers. The page shows one study row, the pain case, under this licence with attribution (Pain chapter, below: Display rights), and no social row. - Sample: 48 segments in each of 31 locales, the same 48 English sources in every locale: 1,488 segments and 341,589 source characters, counted in code points. Each locale has 12 segments per domain (literary, news, social and speech) from 30 documents in 28 bootstrap clusters. Excluded before drawing: the canary row, 38 rows the dataset marks as bad sources, the 72 rows the registered per-language model choice below was decided on, 8 sentinel rows, the page's own WMT24++ case (row 367), 25 rows whose reference equals the source in a study locale and 1 source carrying an HTML entity. - Metric: chrF++, sacrebleu 2.6.0 `CHRF(word_order=2)`, signature `nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.6.0`; a corpus score per locale, and overall the unweighted mean of the 31 locale scores. - Interval: paired bootstrap over clusters, 4,000 draws, seed 799, 95% percentile interval. - Verdicts, with a margin of 1.0 chrF++: better when the interval is above zero and the difference at least 1.0, worse for the mirror case, level when neither holds and the interval lies inside -2.0 to +2.0, and inconclusive otherwise. ### SIE's row The page shows one SIE row: the model SIE Serverless picked for each study locale on 2026-10-25. Serverless offers Translation on Hy-MT2 1.8B and MADLAD-400 3B. That day it picked Hy-MT2 in every locale both models serve, which covers all 31 study locales, and MADLAD in the 24 locales Hy-MT2 does not serve, none of them in this sample. So the page's row is Hy-MT2 1.8B in all 31 locales: the Hy-MT2 arm as recorded through Serverless on 2026-10-25, output for output, with that recording's scores and intervals. Since 2026-10-29 Serverless picks MADLAD in three study locales, fr_CA, it_IT and fr_FR (MADLAD-400 3B through Serverless, below); the page's row has not changed. Serverless made that choice after the study ran, from this sample's own scores, so the row is not independent of the sample. It became the page's row on 2026-10-25, by the second amendment to the registration. A caller names the model in each request; nothing on the page says SIE routes requests by language. The registered row stays the record: SIE's per-language model choice, fixed before the study ran. In each locale it takes Hy-MT2's output where Hy-MT2 led MADLAD by more than 1.0 chrF++ in an earlier comparison, on rows this sample excludes, and MADLAD's everywhere else, ties included. - Hy-MT2 1.8B, 19 locales: fa_IR, fil_PH, gu_IN, hi_IN, id_ID, ja_JP, ko_KR, mr_IN, nl_NL, pt_BR, pt_PT, ru_RU, ta_IN, te_IN, th_TH, tr_TR, ur_PK, vi_VN, zh_CN - MADLAD-400 3B, 12 locales: ar_EG, ar_SA, bn_IN, cs_CZ, de_DE, es_MX, fr_CA, fr_FR, he_IL, it_IT, pl_PL, uk_UA The registered row sends no requests of its own: it is built from the two SIE arms of the direct runs, locale by locale. ### Recorded again through Serverless The third amendment points the row at the Hy-MT2 arm's run through Serverless on 2026-10-25, which sent the direct run's 1,488 request bodies byte for byte; the direct run of 2026-10-14 stays the record. Scored beside it, the Serverless run reads 48.15 against the direct run's 48.08: +0.07 (+0.03 to +0.12), level, and 31 level per locale. It flags the same 2 of 1,488 cases. It used 115,543 input tokens, as the direct run did, and 167,976 output tokens against the direct run's 167,897, so it costs $3.1472 per 1M characters against $3.1459 at the same prices. SIE's verdict against each rival is unchanged: the direct run read -3.70 (-4.39 to -3.08) against Google Cloud Translation, -2.33 (-3.29 to -1.67) against GPT-6 Luna with no reasoning and -2.34 (-3.31 to -1.54) with high reasoning, each worse. ### Results | Arm | chrF++ | Literary | News | Social | Speech | Flagged | |---|---|---|---|---|---|---| | SIE's row (Hy-MT2 1.8B, Serverless) | 48.15 | 43.79 | 51.19 | 43.41 | 49.38 | 2 of 1,488 (0.13%) | | Hy-MT2 1.8B, direct run (the record) | 48.08 | 43.72 | 51.13 | 43.27 | 49.30 | 2 of 1,488 (0.13%) | Per-domain scores are secondary and never a headline. Integrity flags are raised and never rescored: an empty output, a wrong script, a copy of the source, an HTML entity absent from source and reference, two sentences joined with no space, an output cut at its token cap, or one over three times the reference's length. - SIE's row (Hy-MT2 1.8B): 2 wrong scripts, the same two cases as the direct run. The registration says "An arm with more than 5% flagged cases gets a manual read before results go anywhere". MADLAD-400 3B's registered float32 run is over that gate, at 75 of 1,488 (5.04%), and so is its bfloat16 re-run, at 77 of 1,488 (5.17%), with none of their flagged outputs read by hand yet, so this file withholds the results of those two runs, and the registered row's, which take the float32 outputs in 12 locales, until that read. MADLAD's run through Serverless is under the gate, and the fifth amendment publishes its results (next section). ### MADLAD-400 3B through Serverless On 2026-10-26 the study's MADLAD arm ran through Serverless, at `https://api.superlinked.com/v1/generate/google__madlad400-3b-mt`: the direct run's 1,488 request bodies, byte for byte, on the bfloat16 artifact Serverless serves. That artifact's output now has the HTML-entity escaping undone (superlinked/sie pull request 819), which the two earlier runs lacked. All 1,488 requests returned HTTP 200 on the first try. The run flags 34 of 1,488 (2.28%), under the gate, and no output has an HTML entity. It is not the page's row, and the page charts none of its results. | Arm | chrF++ | Literary | News | Social | Speech | Flagged | |---|---|---|---|---|---|---| | MADLAD-400 3B, Serverless | 44.62 | 40.93 | 48.95 | 38.57 | 44.49 | 34 of 1,488 (2.28%) | - Flags: 23 sentences joined with no space, 6 outputs cut at the token cap, 3 wrong scripts, 1 copy of the source and 1 empty output. - SIE's row (Hy-MT2 1.8B, Serverless) minus MADLAD-400 3B: +3.53 (+2.56 to +4.76), better. Per locale it is 17 better, 1 worse and 13 inconclusive. The worse one is fr_CA, -3.64 (-6.62 to -0.96), where MADLAD scores higher. - Tokens: 96,565 input and 107,423 output. At MADLAD's Serverless prices (below) that is $1.8517 per 1M source characters. Since 2026-10-29 Serverless picks MADLAD-400 3B in fr_CA, it_IT and fr_FR, and Hy-MT2 1.8B in the other 28 study locales. That choice rests on larger exploratory sets, not on this sample: in each of those locales, every eligible WMT24++ row, 858 in all, scored with the registered scorer. They are the sample's 48, as recorded above, and 810 more, sent to both models through Serverless on 2026-10-26. On those rows Hy-MT2 minus MADLAD is -2.78 (-3.66 to -2.00) in fr_CA, -2.19 (-3.28 to -1.18) in it_IT and -1.55 (-2.60 to -0.50) in fr_FR. On the sample's own 48 rows only fr_CA is worse; it_IT, at -2.38 (-4.81 to +0.16), and fr_FR, at -2.00 (-5.19 to +0.68), are inconclusive. cs_CZ, checked the same way, is level at -0.22 (-0.82 to +0.49) and stays on Hy-MT2. The exploratory sets are not part of the registration, and the page claims nothing from them. ### Cost per 1M source characters SIE's row: its recorded input and output tokens at SIE's Serverless prices, over the sample's 341,589 source characters. | Arm | Input tokens | Output tokens | Per 1M characters | |---|---|---|---| | SIE's row (Hy-MT2 1.8B, Serverless) | 115,543 | 167,976 | $3.1472 | | Hy-MT2 1.8B, direct run (the record) | 115,543 | 167,897 | $3.1459 | Serverless bills each request in whole credits of $0.00001, rounding the exact price up. For the study's 1,488 requests it charged 108,228 credits, which is $3.1684 per 1M characters and 84.2% cheaper than Google Cloud Translation's list price. The page prices the recorded tokens at the list price, as the registration does. Serverless reported 41,149 of the input tokens as cached and charged them at the full input price; so does the cost above. SIE's Serverless prices, USD per 1M tokens, stated exactly, as Serverless has charged them since 2026-10-22: - `tencent/Hy-MT2-1.8B`: input 1.1252040404, output 5.62602020202 - `google/madlad400-3b-mt`: input 0.99816487456, output 4.99082437276, priced on the bfloat16 artifact Serverless serves ### Rivals - Google Cloud Translation: Translation API v3 `translateText` with the NMT model (`general/nmt`), `text/plain`, one segment per request. List price $20 per 1M characters, read 2026-10-13 at . The first 500,000 characters a month are free, which would cover this whole 341,589-character workload; the page uses the list price. Google also sells a Translation LLM under the same product name at a different price; the page compares with NMT. - GPT-6 Luna: Chat Completions at no reasoning and at high reasoning, with a fixed translation instruction and a 4,096-token output allowance. List price $0.10 per 1M input and $0.50 per 1M output tokens, read 2026-09-27. Both rivals ran on the sample on 2026-10-19, 1,488 of 1,488 segments each. Each rival's cost is its recorded usage at its list price, over the sample's 341,589 source characters. Google Cloud Translation billed 341,589 characters. GPT-6 Luna used 188,743 input tokens at each setting, and 116,030 output tokens with no reasoning against 561,139 with high reasoning (reasoning tokens included). | Rival | chrF++ | SIE's row minus the rival | SIE's verdict | Per 1M characters | Flagged | |---|---|---|---|---|---| | Google Cloud Translation | 51.78 | -3.63 (-4.31 to -3.02) | worse | $20.0000 | 4 of 1,488 | | GPT-6 Luna (no reasoning) | 50.41 | -2.26 (-3.21 to -1.61) | worse | $0.2251 | 1 of 1,488 | | GPT-6 Luna (high reasoning) | 50.42 | -2.26 (-3.23 to -1.47) | worse | $0.8766 | 1 of 1,488 | SIE's row scores 48.15 and costs $3.1472 per 1M characters. Per locale, SIE's row against Google Cloud Translation is 23 worse and 8 inconclusive; against GPT-6 Luna with no reasoning, 1 better, 16 worse and 14 inconclusive; with high reasoning, 1 better, 12 worse and 18 inconclusive. The page's chart shows SIE's row against Google Cloud Translation only. That is an editorial choice, made on 2026-10-20 after every arm was scored, and recorded as a dated amendment to the registration. Both GPT-6 Luna settings ran as registered, and the two GPT-6 Luna rows above report them. The chart shows Google Cloud Translation ahead of SIE's row on chrF++, and the page claims price alone; it states no comparison with GPT-6 Luna. Known bias, in the rivals' favour: Google built WMT24++, and language models trained after February 2025 may have seen it. ## Pain chapter: a changed number The pain chapter shows one recorded segment in which Google Cloud Translation changed a number, beside what Google bills for a billion characters, and two published reports of translation changing a number. A rule picked the segment. We froze the rule before opening any arm's output, and the fourth amendment to the registration records it, on 2026-10-28. ### The rule - Name: `pain-case-rule-v1`, frozen at 2026-10-28T11:51:24Z. We wrote it from the study sample's sources and human references, the registration, the public folder's checksums and the README's Data structure section (lines 112-181). No output or score file of any arm was opened before the freeze. - SHA-256: the rule text `8b9a55a8e1460e646be75d6a6f8ad841ffa1bda70c07fc142f0fc3c4cbb2941b`, its detector `a2224e7dd81cff0c7d4b0054e0fd866153c23e1099a73df9082c936956f40d5d` and the freeze record that holds both digests and the time `fbaf907ff12dd2f4b3d3f3116a1fd7761576a6faa3b56fa87cf3062597d37fe6`. The rule pinned the public folder's `data/sample.jsonl` and each arm's outputs file by the digests in the folder's `SHA256SUMS`, and the run checked them. It ran on Python 3.14.5 with Unicode 16.0.0. - A fact is a number written in digits in the English source. It counts when the human reference keeps it: 718 facts in 509 of the 1,488 segments. The detector flags a counted fact the output drops or changes, and a number in the output that neither the source nor the reference states. Locale number formats and the scale words in its table (100万 for a million) raise no flag. Units, currencies and which value belongs to what are not compared. - One reader confirms each flag from the source, the reference and the output, on a sheet with no arm name and no segment id. The reader was an agent, Claude Opus 5.5. A flag it does not confirm gets an exclusion code: a number in words, the same value in another form, digits used as text, range shorthand, an idiom, text outside the translation, or undecidable. The decisions file, SHA-256 `b4e283f3b57a8a351e89abd7a0707ae20e084a5a97ed355c93101bbcb6a68176`, was hashed before the arm map was read. Blinding was partial: the reader's prompt also held the detector's run report, which listed each arm's flagged locales and segment ids, so a row's arm could be worked out from its locale and segment. The rule asks for an arm-blind read, and Amendment 4 records the departure. Every decision applies one test, a number written in words or a different value, the same way on every row. - After the result was written up, an auditor re-decided all 18 flags on a reshuffled sheet with no arm, segment id or sheet key, and made the same decision on every flag. The auditor had seen the first decisions, so this is a re-check, not an independent read. A fresh reader who has seen neither the run report nor the results would remove both caveats. - An arm's count is its segments with at least one confirmed flag, out of 1,488, with a Wilson 95% interval. Each rival is paired with SIE's row on the same segments by an exact two-sided McNemar test, Holm-adjusted over the three rivals. - Decision: a rival whose count is below SIE's row gets no case. Otherwise its case is the first segment in sample order where it dropped or changed a counted fact, SIE's row has no confirmed flag, the domain is not social and its output is not empty. The page may say the rival changed that fact. It may say the rival changes facts more often than SIE only when its rate is higher and p is below 0.05. ### Result The detector raised 18 flags on 15 distinct outputs. The reader confirmed 7, all Google Cloud Translation's, on 4 segments; the other 11 are numbers written in words. | Arm | Raw flags | Confirmed by the rule | Counted | Wilson 95% | |---|---|---|---|---| | SIE's row (Hy-MT2 1.8B, Serverless) | 0 | 0 of 1,488 | 1 of 1,488 | 0.01% to 0.38% | | Hy-MT2 1.8B, direct run (the record) | 0 | 0 of 1,488 | 1 of 1,488 | 0.01% to 0.38% | | Google Cloud Translation | 12 | 4 of 1,488 | 4 of 1,488 | 0.10% to 0.69% | | GPT-6 Luna (no reasoning) | 2 | 0 of 1,488 | 0 of 1,488 | 0.00% to 0.26% | | GPT-6 Luna (high reasoning) | 4 | 0 of 1,488 | 0 of 1,488 | 0.00% to 0.26% | The page prints none of these counts. "Counted" adds the case in the limitation note below. Google's four are segment 782 in bn_IN ("30 bills" became 300 dollars), mr_IN (thirty thousand) and vi_VN (300 dollars), and segment 794 in tr_TR (the year 2543 became 2023). GPT-6 Luna's six raw flags are "1970s" written as "seventies". Against SIE's row as the rule froze it, Google Cloud Translation is 4 to 0, p = 0.125 (Holm 0.375). With SIE's counted segment it is 4 to 1, p = 0.375 (Holm 1.0). Each GPT-6 Luna setting has p = 1.0. No difference is significant, so the page compares no rates. The pick is Google Cloud Translation's `wmt24pp-782-bn_IN`, a speech segment and the first that qualifies in sample order. The next are segment 782 in mr_IN, 794 in tr_TR and 782 in vi_VN. Neither GPT-6 Luna setting has a qualifying segment, so GPT-6 Luna has no case. ### Limitation note, 2026-10-28 An audit after the result found a fact change the frozen detector cannot see. The rule is unchanged. In segment 782 in ta_IN, SIE's row and the direct run both write "30 பில்லியன்" (30 billion) where the source and the reference have 30. The detector keeps every reading of an output number, and a scale word after it only adds the scaled reading, so the bare 30 still reads as the fact kept. A scan of all five arms for a scale word after a kept number finds this segment, in both SIE arms, and nothing else. We count it: SIE's row has 1 of 1,488 in the table above, and the paired test uses it. ### What the pain chapter shows - The English source of `wmt24pp-782-bn_IN` in full, credited to WMT24++ with a link to the dataset; Google Cloud Translation's Bengali as returned and tagged `bn-IN-x-mtfrom-en` (machine translated from English), with Google's "powered by Google Translate" badge beside it; and the WMT24++ human reference. The marks are on the number alone: "30" in the source and "৩০০" in Google's output, the detector's two flagged spans, and "৩০" in the reference, the counted fact it keeps. No flag sits on the reference; the rule counts the fact because the reference keeps it. The words beside each number stay unmarked, as the rule compares numbers and not units. Each Bengali text has our English gloss of the phrase, "300 dollars" and "30 bills", with the number marked. - The bill: Google Cloud Translation billed 341,589 characters for the study, $6.8318 at its list price. The card prices a buyer's workload instead: 1,000,000,000 characters at that price is $20,000. Google's pricing page, read on the date under Rivals, makes the first 500,000 characters a month free; that takes $10 off a month that holds the whole billion, and the card leaves it out. The card shows Google alone. SIE's price is on the proof's cost bar: for the study's text, Serverless charged SIE's row 108,228 credits, $1.0823 for the study's 1,488 requests. - Two reports, read at the source on 2026-10-28, each headline written from its finding: - NBC New York, 2018-02-06, "15,000 Eggs Delivered to Norwegian Olympic Team After Google Translate Error": Norway's Olympic chefs drafted an order of 1,500 eggs with Google Translate, and "a translation error converted their order of 1,500 eggs to 15,000 eggs". - Wang, Xu, Guzman, El-Kishky, Rubinstein and Cohn, "As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation", Findings of ACL 2021, first posted 2021-07-18. Its behavioural tests find number errors in major commercial systems and research models, for high- and low-resource languages. The paper names its commercial systems only as System A and System B, so the headline names none. - Not shown: every arm's count and interval, which RULE.md section 5.3 asks the page to print beside the case. The page shows no SIE result in its pain chapter, and SIE's count is the one in the limitation note, so all counts are here, in the table above. Amendment 4 records the departure. - Not shown: SIE's row on the same segment, which RULE.md section 5.3 also asks for. SIE's row writes "৩০টি টাকা" there: it keeps 30 and writes taka for bills, a unit the rule does not compare. The task-page guide asks for pain cases from two providers. The page charts Google Cloud Translation alone (Amendment 1), so both pain cards are Google's: the segment and the bill. The page owner approved this exception on 2026-10-28. Nothing of SIE's is shown in the pain chapter, no output, count or bill; SIE's ta_IN slip is counted and stated here. ### Display rights, decided 2026-10-29 The segment is speech, with no social handle. By an operator decision of 2026-10-29, recorded in Amendment 4, we show it under the licence the WMT24++ dataset declares for its files, Apache-2.0, with attribution: the card credits WMT24++ and links the dataset, and the dataset entry above cites it. That is the basis on which the whole sample, this segment included, is already published in the study's public folder on Hugging Face. The WMT24 task asks that uses other than research consult the original owners of the texts; for this segment we rely on the dataset's Apache-2.0 licence. The rule's per-row display-rights check passes, so no row is skipped. Google's output carries Google's "powered by Google Translate" badge beside it, linked to , as Google's attribution requirements ask (). The badge is Google's brand feature, used under those requirements; it is not ours and no licence of ours covers it. It is the file `png/color-regular.png` from Google's badge archive (), unaltered, the same bytes as the study folder's `assets/powered-by-google-translate.png`, shown at its natural 176 by 16 pixels on a white plate, since it is Google's colour badge for light backgrounds and the card is dark. SHA-256 of the image: `1ce4cf9db4cd6c8ae6f6d49542264f5334f40ac79139d51c145a4e039d26323d`. Google Translate and the Google logo are trademarks of Google LLC. Google's Cloud Translation terms also ask a site that shows Google's translations to carry this disclaimer: THIS SERVICE MAY CONTAIN TRANSLATIONS POWERED BY GOOGLE. GOOGLE DISCLAIMS ALL WARRANTIES RELATED TO THE TRANSLATIONS, EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OF ACCURACY, RELIABILITY, AND ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. ## Primary sources ### EU consumer rights withdrawal notice - Source: European Union, Directive 2011/83/EU on consumer rights, Annex I(A), Model instructions on withdrawal - CELEX: `32011L0083` - URLs: - English: - German: - Swedish: - Polish: - Licence: EUR-Lex legal notice; legal documents published in EUR-Lex may be re-used for commercial or non-commercial purposes () - Downloaded file SHA-256: - EN `2fe50fe87781643f1945bc89be05617f58fdb09b3078bbc2cba7b7e28553c4ad` - DE `a8d748b8e47f5d1a5a694aab2572a210fe5252987e24fd9b27e508c5f7aba073` - SV `177c4012120baa767164295cecba4b59002c06fd06e6ce0b34e6eb6ecd29e652` - PL `d80f7e54b54ebe9ee01f140a674e3a690ae0c827f55d284cffea1e3216e893f3` - Excerpt: the first sentence under Right of withdrawal and the withdrawal-deadline sentence, in each language version - Extraction: visible text with whitespace collapsed; footnote markers removed; the two sentences joined by one space ### EU air passenger check-in notice - Source: European Union, Regulation (EC) No 261/2004, Article 14(1) - CELEX: `32004R0261` - URLs: - English: - German: - Licence: EUR-Lex legal notice, as above - Downloaded file SHA-256: - EN `bcf5496a63827a938d4a11526c0e9debe5cb599c00cba55eab3f1ddc00cbaa30` - DE `02684a6932ddf3911a8257f701ca08c717cb0006f11a4bd7c9b47eda04369efe` - Excerpt: the quoted notice text an operating carrier must display at check-in - Extraction: visible text with whitespace collapsed; surrounding quotation marks removed ### Health Canada consumer product recalls - Source: Government of Canada, Recalls and Safety Alerts open data - Dataset: - Licence: Open Government Licence - Canada (). Contains information licensed under the Open Government Licence - Canada. - Downloaded file SHA-256: - `HCRSAMOpenData.json` `ba157e6f0422f7b0c58f07c3cbd1524cb0a28cd261c93d8ddd112b7cc43947e4` - `SCRSAMDonneesOuvertes.json` `9393ac3b9361f0a4f980ee558da93615f5385ff49a4237edc96c4236865c475a` - Records: - NID `77805`, MY25 Norco Sight VLT CX e-bicycle, last updated 2025-08-05: - NID `81504`, Muc-Off Disc Brake Cleaner, last updated 2026-01-20: - Excerpt: the first two paragraphs of What you should do and of Ce que vous devriez faire - Extraction: HTML entities decoded; the export joins paragraphs with no separator, so they are split at that seam and joined by one space ### Android system messages (one shown on the page) - Source: The Android Open Source Project, `frameworks/base`, `core/res/res/*/strings.xml`, tag `android-15.0.0_r1` - URL: - Licence: Apache License 2.0 - Downloaded file SHA-256: - `values` `5034fb28babbf5c50a249004a8c535fd0c714e5044321f400dec2016888cc5de` - `values-fr-rCA` `b77119bf03c2444d0801d50086df9e13bda41157bdad9fb658f54dd7a704301c` - `values-fr` `e7cf915bd5dd8e34b58861759e7eec118556e525afffa549ef7dec98c8b36bf6` - `values-pt-rBR` `db94c02e9e6a1674a75921819d9ba4d43a9850df1067c06de9e0a5624b25e7c1` - `values-ja` `2ef375e690d1011686295f67f79a91e06ff29c5486af7bbd4c0e84204dc6d54d` - Keys: `deprecated_target_sdk_message`, `alternative_fp_setup_notification_content`, `review_notification_settings_text` - Shown on the page: `review_notification_settings_text`, the English `values` string and the Japanese `values-ja` one - Extraction: Android resource quoting and apostrophe escapes removed ### Firefox private browsing promotion - Source: Mozilla, `aboutPrivateBrowsing.ftl`, message `about-private-browsing-pin-promo-title` - English: - Mexican Spanish: - Licence: Mozilla Public License 2.0 - Downloaded file SHA-256: - en-US `8d89279790d145f0e37073edd042c58e44fe4bccc52f3f68e21b72a9600e309e` - es-MX `74599ca84b9add00146fb19bc6747b1cb04429f23b2ca8f570d0a1852d3bbaf9` - Extraction: the message value, unchanged ### WMT24++ page case (recorded, not displayed) - Source: Google, WMT24++ (Deutsch et al., 2025), `en-sv_SE.jsonl`, dataset revision `fd7405c06494bc66a57b25f55d217a72f96e60dc` - URL: - Paper: - Licence: Apache License 2.0 - Downloaded file SHA-256: `9a04d3ae733a59f907e141ab253489407b9dd92a73ba3894869867dc81a0d37b` - Row: `segment_id` 367, social domain. The page shows no social row, and the study sample excludes this one. - Reference: the post-edited `target` field, which the dataset card recommends as the default reference