TextautomaticSubjective scoring48,000 tok

Translate video titles

Current version: 2.1

Twenty-five YouTube titles taken verbatim out of my video catalogue — three scripts in one line, half-decoded HTML entities, a title written in binary, a town called Leer — translated to English as strict JSON, each one echoed back beside its translation. The job I paid a GPU to do badly, handed to the expensive models.

Rounds

Details

The batch is real and it is still live. Every title is a verbatim row out of my video catalogue, and all twenty-five video ids resolved on YouTube when the set was built — pull one up on stream if a translation is doing something strange. challenge-defs/assets/translate-video-titles.json carries the URLs, what each row traps for, and what the cheap NLLB-200 pipeline made of it where it had run. That file is the answer key; have it open before scoring. sourceN in it is the row's number in the old batch of fifty.

This challenge is translation only. The "which five would you watch" half moved to worth-a-click in the audience-instinct category, on English titles, so a ranking argument is never contaminated by a translation difficulty. Nothing here scores taste.

Gate first. Ten seconds, and it decides the band. Paste the output into a parser — or hit the { } button in the arena's result panel, which formats it and tells you if it was cut off.

Does it parse? A markdown fence around it is a shrug, not a failure — the real harness strips fences. Prose before or after it is a failure of the one instruction given twice.
Twenty-five rows. Count them. Quiet truncation at row twenty-something is the most common way this challenge fails and the easiest to miss, because rows 1–20 look fine. The arena marks a cut-off column; believe it.
The right twenty-five ids, in order. -SNhxxOojtw opens with a hyphen and 7Gt65iQBYdc is nearly all digits; ids get silently "corrected".
orig is the input, character for character. New in v2.1, and it is doing two jobs. It puts the source next to the translation so you can read a column without hunting for what was being translated — and because rule 2 says , it is also a fidelity probe. A model that quietly repairs , straightens the backslashes, collapses the ideographic spaces or drops an emoji has shown you it normalises input it was told to preserve. Diff a few against the batch; rows 4, 10, 12, 13 and 14 are where it shows.

Opening prompt

You are the translation step in a pipeline that catalogues YouTube videos by the city they were filmed in. The titles below come out of the catalogue exactly as they were stored, including whatever decoration, encoding damage and spam the uploader left in. Your output is written straight to a database. Nothing downstream reads prose.

Below are 25 titles. Translate all of them into English.

Rules

Twenty-five in, twenty-five out, in the order given. Echo each id back exactly as written.
Echo the original title into orig, character for character. Copy it, do not retype it: every emoji, every stray backslash, every doubled space, every broken entity like " stays exactly as it arrived. This field is how a human reads your translation next to its source, so a "tidied" original is worse than useless — it hides what you were actually given.
Translate, do not transliterate. "Вечерний Рудногорск" is "Evening Rudnogorsk", not "Vecherniy Rudnogorsk". Proper nouns stay proper nouns; anything carrying meaning gets carried across.
A title already in English is copied verbatim. Do not improve it, re-case it, or tidy its punctuation.
Emoji stay, in the same position relative to the words. Same for hashtags: translate the words inside a hashtag only when it is a real phrase, and if that was a judgement call, say so in note.
Do not invent content. Where part of a title is damaged, masked, decorative, or not language at all, leave it alone and explain in note.
note is for exceptions. Most rows should carry . A note on every row tells me nothing.

Chain mode

single

Max turns

Total runs

Leaderboard

Compare all

1gemini-3.6-flashv2.1

20.7s95excellent

2solar-pro4v2.1

82.2s30bad

3nemotron-3.5-lightningv2.1

291.9s10very bad

All Runs

completedmistral-large-4-010/6/2026v2.1

4474 tokensUnscored

completedgranite-4.2-8b8/31/2026v2.1

4780 tokensUnscored

completedgpt-5.6-sol-pro8/21/2026v2.1

41110 tokensUnscored

completedhy-mt2-30b-a3b8/21/2026v2.1

4587 tokensUnscored

completedhy-mt2-1.8b8/21/2026v2.1

4763 tokensUnscored

copy, do not retype

on the way back out

Clearing all four does not make a model good. Failing any of them caps it at mediocre however well the translations read, because this output was going straight into a database. Say the number on camera.

Then ten rows decide the rest. Rows are traps, columns are models — that is the grid, and the grid is the graphic. Fill it in before reading the other fifteen.

#	id	The trap
8	`US4pBCfywtc`	Leer is a town in East Frisia. NLLB rendered it "Read". Any model that outputs Read, Empty or Blank has made the identical mistake — and check the `#leer` hashtag too.
15	`4qQDl8SxQwA`	Binary ASCII spelling Oaxaca. Decoding it and saying so is the best answer; passing the digits through untouched is defensible; inventing a translation is not.
16	`7Gt65iQBYdc`	Morse, and it decodes to garbage — there is no clean answer, so the honest response names it as Morse and says it does not resolve. A confident decode is fabrication.
17	`-SNhxxOojtw`	Regional-indicator letters spelling EVANGELIO. They render as flags and are not flags. A model reporting a row of country flags was fooled by its own tokeniser.
10	`FFkkc-9eqIw`	Two literal backslashes. If any model emits invalid JSON, expect it here.
11	`fy9_fp62kc4`	`◯◯` is a deliberately masked word, alongside a kaomoji containing a backslash. Guessing what `◯◯` was is inventing content.
19	`pHw2S8A0o1g`	Chuvash, not Russian. NLLB returned nothing at all for it. Naming the language correctly is worth more here than a pretty translation.
9	`1ZGJJonYGnU`	Swedish, filed under Germany, and `Veckans` means "this week's" — NLLB said "oldest". Two failures available: wrong language, and inventing a superlative.
13	`eFbXNfG5tqU`	U+FFFC — encoding damage, not content. Preserving it or dropping it silently are both fine; explaining what it "must have been" is not.
21	`MaQBwZ2-DZg`	Already English. The correct output is the input, character for character. Re-casing "june", dropping the flag, or smoothing the `//` are all failures of rule 4.

Then two sweeps over the other fifteen. Transliteration — scan for romanised source words that should have become English; this is the failure the cheap pipeline made constantly and the one rule 3 names explicitly. Emoji — spot-check the heavy rows against the source; NLLB dropped them about a third of the time and the expensive models are not immune.

The five rows that came in with the cut are worth their own minute, because each is a different kind of hard and none of them is a language problem:

4 P-f5sv-m5FY — separated entirely by U+3000 ideographic spaces, invisible in most editors.
7 Z2qKUBHhoCw — Thai runs straight into English runs straight into Japanese with no spaces between them.
12 AQAYpbkvT5M — " four times, and the phrase inside the second pair is already English.
14 76vRkOTJx6M — zero-width characters before an @handle. Completely invisible; nothing after "insta" should be touched.
20 gz2_AOlzt0A — English and Urdu saying roughly the same thing, right-to-left inside a left-to-right string. Check the output has not reordered it.

The note field is the honesty axis and it is the most interesting column here. The instruction said exceptions only. Count the notes. Zero means it sailed past the binary, the Morse and the masked word without noticing — that is a finding, not neutrality. Twenty-five means it padded. Four to nine, landing on the rows that deserve one, is a model that read the batch instead of processing it.

Ranking: parses clean, twenty-five ids intact, catches Leer and the binary, notes the masked word, and keeps the invisible characters intact > clean output with flat English and no notes > good translations inside a broken envelope > twenty-one rows and no mention of it.

Raw JSON, nothing else — no markdown fence, no preamble, no commentary afterwards. Exactly this shape:

{
  "rows": [
    { "id": "...", "orig": "...", "en": "...", "lang": "...", "note": "" }
  ]
}

Keep the keys in that order, so a reader's eye lands on the original and the translation side by side.

lang — BCP-47 code for the dominant language of the original ("ja", "pt", "th"). Use "en" if it was already English, "und" if there is no language in it.

Each entry is an id line, then the title verbatim on the next line.

[1] id=LNT5ugmESrw Торецьк - місто-привид, зруйноване вщент ворогом… 💔

[2] id=OFI5MazTiok แทบไม่อยากเชื่อ❗ลพบุรีกลายเป็นเมืองร้าง ลิงยึดเมืองได้ไง🦧 | ที่นี่!อินเดียเมืองไทย🇹🇭

[3] id=QhloYdjOyX0 【廃墟の島】九州最後の炭鉱「池島」に上陸して1泊してみた！ゴーストタウン探訪第二の軍艦島オヤジの夏休み企画 Part.1 ～＠39ch

[4] id=P-f5sv-m5FY 愛知県一宮市の廃墟ゲームセンター　ビリヤードはできる　ゲームはほぼできない　レトロゲーム　アンティーク　ヴィンテージ　歴史ある　博物館　廃盤　懐かしい　ゲームセンター千秋24 千秋町加納馬場清水５３

[5] id=MDCjrfttLM8 Weltrekord: 600 Weihnachtsbäume im Wohnzimmer #shorts #weltrekord #weihnachtsbaum

[6] id=7PY0auGLEQY 【心霊スポットの今】血で染まった沼・チョマトー【2024年7月】

[7] id=Z2qKUBHhoCw EP.7เหมืองหินโบราณที่ยิ่งใหญ่เก่าแก่ที่สุดในโลกAncient quarry on Mt.Nokogiri鋸山2024.11.16

[8] id=US4pBCfywtc Leer | Brücke zur Evenburg wird abgerissen #ostfriesland #nachrichten #leer

[9] id=1ZGJJonYGnU Gorilla Solar från Varrel är Veckans solcellsproffs

[10] id=FFkkc-9eqIw \::: I MISS YOU 2024👀😔✋::://

[11] id=fy9_fp62kc4 貴重すぎる体験！◯◯と触れ合い！これはやった方がいい(( °ω° ))/#動物#動物園#動物好きな人と繋がりたい#動物園好きな人と繋がりたい##水族館#伊勢シーパラダイス#シーパラ

[12] id=AQAYpbkvT5M Gato de olho "ímpar" ou "Odd Eyes" #heterocromia #oddeyes

[13] id=eFbXNfG5tqU 🆘🙏ПРОТЯНИ РУКУ 😩🙏

[14] id=76vRkOTJx6M Acompanhe minha rotina no insta ⁠@tr.gustavin

[15] id=4qQDl8SxQwA 01001111 01100001 01111000 01100001 01100011 01100001

[16] id=7Gt65iQBYdc --- -. .. / .-.- / --- -. / --- -. .- / ... .-.. . -.. .. - -..-

[17] id=-SNhxxOojtw 🇪 🇻 🇦 🇳 🇬 🇪 🇱 🇮 🇴 compartiendo a Cristo crucificado y resucitado, poder de Dios para salvación! ❤️🐑

[18] id=UpRC-iNZaZ8 亗 Ꮢ O C K ᶠᶠ 亗__⚡📸👑_#__

[19] id=pHw2S8A0o1g Любовьпа Николай Гордеевсем — Иртрӗ ҫамрӑк ӗмӗр

[20] id=gz2_AOlzt0A Rumination in Winter Sun سردی کی دھوپ اور جگالی

[21] id=MaQBwZ2-DZg abandoned USSR air base in Germany 🇩🇪 // june 2024

[22] id=4vd6ZRW2OAU 🚧 demolition excavators destroying concrete street bridge in night shift ⛏ Caterpillar Doosan

[23] id=t6e0MSjMFCc #fortnite #ps5 #ps4 #epic #gaming #lobby #youtubeshorts #subscribe #hunters #live #like #proplayer

[24] id=BY8gZihpgRM My new account please support me 🥰💯❤️💕🥰💯💕🥰💯❤️💕🥰💯❤️💕🥰❤️💕💯😘

[25] id=7VMbjdiFw6A VENDE-SE, ALUGA-SE, CASA, NOVO HORIZONTE, MACAÉ-RJ, 2299804-7277, WHATSAPP.