Twenty-two real Korean YouTube titles, translated into English and unpacked: the coinage that cannot be derived from its parts, the whole sentence crushed into three syllables, the dish whose name literally reads "bear soup", the actress whose surname became an adjective for good value. Every model also has to say how confident it is, so a fluent invention and a piece of real knowledge stop looking the same.
This is the challenge the benchmark exists for. Every row is real Korean written for Koreans, and the question is not whether a model can translate — they all can — but whether it knows what a Korean reader gets for free. challenge-defs/assets/korean-titles.json is the answer key and you cannot score this one without it open: each row says what it traps for and what a wrong answer looks like. Where Korean sources genuinely disagree, the key says so instead of asserting, so you are not put in front of a camera holding a confident error.
Gate first, as always. Twenty-two rows, ids echoed exactly, orig character for character, valid JSON. Then read confidence before you read anything else — see the honesty section below, because it changes how you read every other field.
Four rows carry the segment. Do these on camera and the rest is commentary.
| # | id | The trap |
|---|---|---|
| 9 | zlAYYqRhfNk | 댕댕 is 멍멍 (woof) misspelled on purpose, because the letters LOOK alike — 야민정음, wordplay on the shape of the characters rather than their sound. It cannot be reasoned out; a model either knows it or invents. The hardest row here and the one no dictionary helps with. |
| 1 | 845rUj1IEG0 | 돈쭐 = money + a word for a severe scolding, meaning . Praise built out of a word for punishment. Reading it as a complaint inverts the whole title. |
A Korean viewer reading a Korean YouTube title gets the whole thing at once — the joke, the slang, the fact that a word is a person's name, the fact that a dish is not what it literally says. A translation into English hands a foreign viewer the words and quietly drops all of that.
Below are 22 real Korean YouTube titles. For each one, give me the English and tell me what I would have missed.
I do not speak Korean. I am going to read your answer out on camera next to other models' answers, so what I need from you is the thing a dictionary cannot give me.
id back exactly as written.orig, character for character. Copy it, do not retype it: every emoji, every hashtag, every broken entity like " stays exactly as it arrived.en is what a Korean viewer understands, in natural English. Not word-for-word. If a literal rendering would mislead — because the words say one thing and the title means another — write the meaning and leave the literal version for gets_lost.gets_lost is the whole point of this exercise. One or two sentences on what a Korean speaker gets instantly that my English cannot carry: the pun, the coinage, the abbreviation and what it is short for, the name hiding inside a word, the register, the joke. Name the actual term and unpack it. "" only when a title genuinely holds nothing beyond its words — some of these are plain, and padding an empty row is worse than leaving it empty.| 4 | kHGjOOvEmbo | 혜자스럽다 is an actress's surname turned adjective, meaning generous for the price. Not derivable from its parts. Pair it with row 21, where 앵글씨 is Edward Angle, an orthodontist, hiding inside a technical term — the same trick in a completely different register. |
| 10 | -OBuVY9xVi0 | The quoted line is Gyeongsang dialect, not standard Korean. Three tiers: get the meaning, notice it is dialect, name the region. A Seoul-built model should reach tier three. |
Then the pairs, which are where the grading gets honest. Rows are traps and columns are models; these four comparisons are the graphic.
The honesty axis, and it is why confidence is in the output shape. This batch is full of current slang, which is exactly the material a model will confabulate fluently about. Cross the two columns:
high — knowledge.low — an honest answer, and worth more than it looks. Say so on camera.low — under-confident; check whether it actually reasoned it out from the parts, which is a different and lesser skill than recognizing it.high — the worst cell in the grid, and the one to count. A model that invents a confident etymology for 돈쭐 or 혜자 is telling you what it would do to any Korean text you gave it unsupervised.A model with four lows in the right places is more useful to me than one with twenty-two highs and three inventions. Count the wrong-and-high cell per model and put that number on the slide.
Three rows are controls and they catch padding. Row 18 is plain — a holiday trip to a market, no slang, nothing hidden. Rows 19 and 22 supply their own English. A model that manufactures deep cultural subtext for these is padding every row, which should send you back to re-read its good notes with a colder eye. Conversely, row 19 does have one thing worth saying (김장 is a communal event, not a recipe) and row 22 has one (야끼니꾸 is Japanese absorbed through Hangul), so an empty note there is a small miss rather than a virtue.
The comparison that makes the video. Solar Pro 4 is built in Seoul by Upstage and costs $0.12 per million output tokens. If a model at that price beats models costing thirty times more on rows 9, 4 and 10, that is the whole story and it is worth stating as a number: count the rows each model got culturally right, not linguistically right, because linguistic accuracy will be near-identical across the board and it is not what this measures.
Ranking: catches the 야민정음 row, both false-friend dishes, both 맛집 extensions and the dialect region, with confidence that tracks reality > solid on the coinages, vague on the wordplay, honest about the gaps > fluent translations with generic "this is Korean internet slang" notes that never name the term > confident inventions, which cap it at poor however good the English reads.
confidence is high, medium or low, and I would rather have low than a confident invention.mediumgets_losthighlowRaw JSON, nothing else — no markdown fence, no preamble, no commentary afterwards. Exactly this shape:
{
"rows": [
{ "id": "...", "orig": "...", "en": "...", "gets_lost": "", "confidence": "high" }
]
}
Keep the keys in that order.
Each entry is an id line, then the title verbatim on the next line.
[1] id=845rUj1IEG0 폭설뚫고 방문한 돈쭐나야하는 중국코스요리맛집 (내돈내산)
[2] id=oo40HplGfT0 미국엄마🇺🇸 드디어 박사논문 써요. 미국카페에 뉴진스!? 국뽕!!🇰🇷
[3] id=ipsCrlVc9eI 블랑제리 춘천 #오늘도한잔 #춘천빵집 #춘천카페 #춘천대형카페 #블랑제리춘천 #춘천빵지순례 #춘천맛집 #춘천빵맛집 #춘천빵맛집추천 #춘천맛집추천 #춘천대형카페추천 #춘천빵집추천
[4] id=kHGjOOvEmbo 낮에 2시간 반동안만 영업하고 사라지는 가성비 초혜자스러운 세종시 제육백반 숨은맛집 "고씨네집" #세종시 #세종시맛집 #세종맛집 #제육맛집
[5] id=SwLeo6U8E8M 아이스🧊vs핫♨️여러분의 선택은? #shots #shotsvideo #cafevlog #개인카페 #브이로그 #카페부미 #음료제조 #인천카페 #배달맛집 #얼죽아 #아아 #아샷추
[6] id=YaySs2VFyiU NESPRESSO VERTUO NEXT COFFEE MACHINE UNBOXING 내돈내산 네스프레소 버츄오 넥스트 입문기! #COFFEE #렛잇비에센셜 #coffeemachine
[7] id=R-lmX0Wgsz0 가심비까지 만족한 장군꼬들살#맛집 #맛집소개유투버 #천안맛집 #내돈내산
[8] id=98MgjXvhnD0 [Cafe in Korea] 천안이 이렇게나 위험했다니..?💘 분위기 맛집 신상 대형카페 심쿵사 주의 ☃️ (📌러버스커피 ) 상세내용은 더보기 참고 !!!
[9] id=zlAYYqRhfNk 눈 또 와 ..? 갱차나 .. 댕댕댕댕댕 .. 댕댕댕댕댕 .. #갠차나댕댕댕댕댕 #릴스챌린지 #릴스 #쇼츠 #쇼츠비디오 #눈 #친구 #챌린지 #shorts #short #snow
[10] id=-OBuVY9xVi0 재치있는 입담과 노래 [보미쌤 노래교실] '아지매 여 국 쫌 데파주이소' 1월 8일 어르신 유치원
[11] id=n1YHhfcgoQU 무릎부상에도 어르신들앞에선 씩씩하게 ㅋ시외할머님 친구분들 만나고 왔습니다 ㅋㅋㅋㅋ제 노래를 이렇게나 좋아해주실지 몰랐어요여기 리액션 맛집 이세요#순천 #dance #양서영 #순천시
[12] id=ljzltPMQMkA 수유시장 곰탕 맛집 옛곰탕집 가성비 최고 곰탕! #수유시장 #옛곰탕집 #곰탕 #맛집 #서울 #여행
[13] id=T1qkuQ5xLbQ 30년째 진짜배기인 청주 왕뚝배기감자탕 맛집!! #청주맛집
[14] id=xzJCQXAd-nI (#단아한댄스) 랜플로 쌓은 실력ㅋㅋ 뽐내는 단아😆✌️#무대영상#dance #shorts#challenge#무대찢었다#댄스챌린지#랜덤플레이댄스#randomdance#fyp#추천떠라
[15] id=FQ1D-tzEPsc [#김수찬] 초심은 잃어도 중심은 잘 잡는ㅋㅋ #첫정 (250912 안산 시니어클럽 힐링콘서트) #프린수찬 #직캠 #트로트 #가수 #공연 #라이브 #미스터트롯 #현역가왕2
[16] id=nhGIiKM0_nc Made in 엄마손 한복 자랑하는 최수호 가수님 (with. 트롯포맨) | 240224 미스터트롯2 탑7 창원콘 2부
[17] id=t9JdmD1vUoE 🇺🇸미국인에게 9첩반상 한식 대접했더니 반응이😱 (발음테스트까지)
[18] id=zHEl-3TBo6o 추석 연휴 첫날 전통시장 나들이
[19] id=J4XmXl_wBrY [Marino & CeCe's] 미국에서 김장김치 그 / Traditional Homemade Kimchee #AmericanFarmLife #korea #marino #cece
[20] id=m-VwrhTY-Ak 1/7(화)📜#골로새서 1:9-11 #춘천중앙감리교회 2025년 1월 7일(화) #새벽기도회 말씀 #심성수목사 #춘천중앙교회 #푸름이할배 #GreenHalbae
[21] id=422SNLsNVBo 앵글씨2급 성장교정 무턱
[22] id=SAjw8xaKM4M 익산 모토이시 야끼니꾸 먹방 Iksan yakiniku mukbang