The bandwidth of language, measured

the human-to-machine channel (plumbing)

• The tokenizer tax (TIR) - Same meaning (UDHR Article 1), different token cost. Under GPT-3.5/4's cl100k, Hindi costs...
• The tax is a fixable design choice - GPT-4o's newer o200k tokenizer roughly thirds Hindi's tax (5.5x -> 1.6x) and halves most n...
• Subword fertility - Tokens per character - a register-comparable efficiency measure. Casual text raises it ~15...
• Register: the tax does NOT worsen on clean casual text - We expected casual text to tax non-English worse. It does not: casual raises absolute cost...
• ...but it breaks on emoji + code-switch - The frontier the register null left: emoji-dense English is ~3.8x clean fertility, non-Lat...

the language itself

• The ~39 bits/sec convergence - Across 17 languages, speech converges on ~39 bits/sec: dense languages (Mandarin) are spok...
• Morphological packing - Agglutinative languages pack more morphemes per word: Finnish and Turkish top morphemes/wo...

the machine's meaning (geometry)

• Does meaning survive the crossing into the machine? - Embed one fixed meaning (UDHR Article 1) in 10 languages and measure how tightly it cluste...
• Plumbing + geometry - Non-English is disadvantaged in BOTH halves of the pipeline: the tokenizer fragments it (p...
• ...but the two halves are largely decoupled - Across 44 languages the tokenizer tax and the geometry penalty correlate only weakly - and...
• Cost and utility move together - The tax measures what you pay (tokens); the model's bits-per-character measures how well i...
• How legible is syntax inside a small LLM? - A Hewitt-Manning structural probe recovers dependency-tree structure from Qwen2.5-0.5B's m...
• AI doesn't reinvent Zipf's law of abbreviation - Two neural agents trained to invent a code in a referential game (no efficiency pressure) ...

history and philosophy it rhymes with

• Shannon's guessing game - Shannon had people guess the next letter of English text; the guesses revealed it carries ...
• Cracking a code with no key - Champollion broke hieroglyphs (1822) with the Rosetta Stone's trilingual crib. Ventris bro...
• Where weak Whorf survives - Languages name colors in a near-universal order (Berlin & Kay 1969), denting strong relati...
• Languages without numbers - The Piraha, with words only for ~one/two/many, struggle to match sets beyond ~3 (Gordon 20...
• 'Untranslatable' = expensive, not impossible - Saudade, hygge, mamihlapinatapai - nothing is truly untranslatable; it can always be parap...
• Three channels in one set of marks - The Masoretes' Hebrew cantillation marks (te'amim, 9th-10th c.) carry THREE channels at on...
• Damin and Ithkuil: the poles of packing - Damin (a Lardil ceremonial register) re-expressed the WHOLE language in ~150-200 abstract ...
• A tongue poured into a narrow channel - Silbo Gomero whistles Spanish across ravines; West African talking drums beat tonal langua...
• Decipherment in reverse - The inverse of cracking a lost code: designing a message understandable with NO shared con...
• Babies point to ask questions - Before a child can say a word, it points (~9-12 months) - and pointing even serves an inte...
• A parent's face as one load-bearing bit - At a glass-covered apparent drop (the visual cliff), 1-year-olds look to the mother's face...
• The face is older than our species - ChimpFACS found chimpanzees share most of our facial action units, on nearly the same musc...
• Watching grammar appear - When Nicaragua opened deaf schools c. 1980, children converged on a sign language in the p...
• The long fight over zero - Positional notation needs a symbol for nothing. Babylon left a gap, the Maya carved a shel...
• Reading used to be out loud - Greek and Latin were written in scriptio continua - NOSPACESALLCAPS - so costly to parse t...
• Inventing writing from nothing - Most scripts evolve over centuries; the Cherokee syllabary was invented by one non-literat...

what other species do (non-human communication)

The atlas beyond humans. The honest throughline: these are statistical and combinatorial STRUCTURE, not demonstrated meaning - cite "structured," never "translated."

• Birdsong vs language on the same axes - We ran our Shannon machinery on Bengalese finch song (12,637 syllables) vs UDHR. Birdsong:...
• Sperm whale codas on the same axes - We ran our entropy/Zipf/MI pipeline on the real Sharma 2024 coda data (8,719 codas). Sperm...
• A sperm whale phonetic alphabet - Across 8,719 sperm whale codas, Sharma et al. found four combinable features - rhythm, tem...
• Dolphins have names - Bottlenose dolphins develop individually distinctive signature whistles and COPY each othe...
• The dance that points to nowhere visible - Von Frisch decoded the honeybee waggle dance: the run encodes direction (relative to the s...
• The vervet alphabet of fear - Vervet monkeys give distinct alarm calls for eagle, leopard, and snake, each triggering a ...
• Decoding without a Rosetta stone - ML now models whale codas (CETI, WhAM) and birdsong (FinchGPT), and human-speech-pretraine...

key references