In 2002 Benedetto, Caglioti and Loreto published a result in Physical Review Letters that still reads as a provocation: you can recover the family tree of the Indo-European languages using nothing but a general-purpose file compressor. No linguistics, no dictionaries, no parsers. Just gzip, the same DEFLATE you use to shrink a tarball.
The corpus was the Universal Declaration of Human Rights, chosen because it is the most translated document in existence. Over fifty translations went in. A tree matching the accepted Romance, Germanic, Slavic, Celtic, Baltic and Ugro-Finnic groupings came out, with Basque and Maltese correctly stranded as outliers.
The mechanism is three lines long. Take document A. Append a short snippet b from another document. Compress. Compare against compressing A alone.
┌───────────────────────────┐ A │ gzip builds a dictionary │ C(A) ┌─────────────┐ │ of A's repeated │ = 2628 bytes │ 7000 chars │──┤ substrings │ │ of English │ └───────────────────────────┘ └─────────────┘ ┌─────────────┬────────────┐ │ 7000 chars │ 1500 chars │──▶ gzip ──▶ C(A + b) │ of English │ ??? │ = 2628 + Δ └─────────────┴────────────┘ ▲ └── the snippet under test. it is encoded using a dictionary that was built for A, not for it. ──────────────────────────────────────────────────────────── Δ_A(b) = C(A+b) − C(A) bytes spent describing b under A's model xent(A,b) = 8 · Δ_A(b) / |b| bits per character ──────────────────────────────────────────────────────────── measured on our own run (section 09): b = English under A = English → 2.651 bits/char b = English under A = French → 3.312 bits/char b = English under A = Finnish → 3.637 bits/char ▲ └── the gap IS the distance
| Detail | 2002 paper | Why it matters |
|---|---|---|
| Corpus | UDHR, 50+ languages | Same content, different distributions. The ideal controlled experiment. |
Snippet b | 1 to 15 KB | Results were robust across that whole range |
Reference A | 32 to 64 KB | DEFLATE's window is 32 KB. Past that, A's head is invisible to b. |
| Language ID floor | down to 20 characters | Distributional signal is dense in text |
| Authorship task | 93.3% on 90 Italian texts | Same metric, different question. It generalises. |
The method drew a sharp rebuttal from Joshua Goodman, who pointed out that a plain character n-gram model plus naive Bayes does the same job faster and better, and that framing it as "zipping" obscures the fact that compressors are just clumsy probability models. He was right on the engineering. The physicists were pointing at something else, and it is the thing worth keeping: compression ratio and predictive accuracy are the same measurement wearing different clothes.
Strip the problem down. You are sending instructions to a robot on a distant rock: up, down, left, right. The mission profile is a biased random walk, so the symbols are not equally likely. Call that distribution q.
Shannon named this the information content of an event. It is worth reading the negative log as a question about halving: how many times do I chop the space in half to isolate this outcome? A symbol with probability 1/8 needs three halvings, so three bits. A symbol with probability 1/2 needs one.
distribution q optimal code cost ───────────────────────────────────────────────────────── up 1/2 ──▶ 0 1 bit −log₂(1/2) = 1 down 1/4 ──▶ 10 2 bits −log₂(1/4) = 2 left 1/8 ──▶ 110 3 bits −log₂(1/8) = 3 right 1/8 ──▶ 111 3 bits −log₂(1/8) = 3 prefix-free: no codeword is a prefix of another, so the stream needs no delimiters and decodes greedily. 0 1 1 0 1 0 1 1 1 0 │ └─┬─┘ └┬┘ └─┬─┘ │ up left down right up the tree that generates it. every leaf is a symbol, every edge is one bit, depth is the codeword length: · 0╱ ╲1 ╱ ╲ up · depth 1 · p = 1/2 0╱ ╲1 ╱ ╲ down · depth 2 · p = 1/4 0╱ ╲1 ╱ ╲ left right depth 3 · p = 1/8 depth = −log₂ p. the code is the distribution, drawn.
-log2(q) is not a whole number and no symbol-to-bitstring table can be optimal. That is not a problem with the theory, it is a problem with the encoding scheme. The information content of a whole message is the sum of its symbols' information contents, and an optimal encoder gets arbitrarily close to that total. Arithmetic coding is the construction that actually achieves it: it never emits a codeword per symbol at all, it narrows a single interval and writes the result once.
If you know the distribution and you build the matched code, the average cost per symbol is the weighted sum of the information contents. That average is the entropy.
Read it as a picture and it stops being an abstraction. Lay out one bar per symbol. Bar width is the probability. Bar height is the information content. Total width is 1 because the probabilities sum to one, so the total area is the average bits per symbol.
For the robot, that is 1.75 bits per instruction. A naive fixed-width encoding would spend 2 bits on four symbols. The bias in the distribution is worth a quarter of a bit per symbol, and entropy is the exact statement of how much.
Now the mission changes. Command reorients the walk to head rightward: up and down drop to 1/8 each, left goes to 1/4, right takes half. Call the new reality p. Your encoder is still hard-wired for q, because it shipped eighteen months ago and it is on a rock.
The codeword lengths do not change. What changes is how often each one gets used.
That asymmetry is the entire idea. p sets the bar widths because it decides how often each symbol shows up. q sets the bar heights because it decides what each symbol costs. Drag the slider and watch the two come apart.
At the default settings the encoder spends 2.625 bits per instruction against a floor of 1.75. It is burning 0.875 bits on every symbol it sends, forever, because its model of the world is stale.
Fix p. Treat q as the free variable and plot H(p,q) as a function of it. Restricting to two outcomes makes this drawable: each distribution has exactly one degree of freedom.
Two facts, and they are the payload of the whole argument:
Now run the logic backwards, because this is where it gets interesting. Suppose you did not know what loss function to use. You know you want some decreasing function F(q) that punishes low probability assigned to things that happen. Infinitely many functions have that shape. Which one?
Add one requirement: the average loss, weighted by the true frequencies, must be minimised only when the model matches the data. Constrain Σq = 1, set up the Lagrange multiplier, and the condition falls out as dF/dq = c/q. Only logarithms satisfy that.
you want: argmin Σ p·F(q) = p subject to Σq = 1 q stationarity of the Lagrangian L = Σ p·F(q) − λ(Σq − 1) : ∂L/∂qᵢ = 0 ⟹ pᵢ · F′(qᵢ) = λ this must hold for every i at once, and by assumption at qᵢ = pᵢ, so qᵢ · F′(qᵢ) = λ for all i ⟹ F′(q) = λ ⁄ q integrate: F(q) = λ·ln q + c and F must decrease, so λ < 0 ───────────────────────────────────────────────────────────── F(q) = −log q . the logarithm is not a design choice. it is the only family that satisfies the requirement.
A language model is a function from a token sequence to a probability distribution over the next token. Training needs a loss. Here is the standard description, which is correct and slightly unsatisfying: for each position, take the probability the model assigned to the token that actually came next, take the negative log, average over the corpus.
context p(true next token) −log₂ p ────────────────────────────────────────────────────────────── The ▁ 0.031 5.01 The capital ▁ 0.412 1.28 The capital of ▁ 0.897 0.16 The capital of France ▁ 0.964 0.05 The capital of France is ▁ 0.991 0.01 ──────── mean 1.30 bits a confused model spreads its mass thin, so −log p is large everywhere. a model that follows the text is rarely surprised. the loss IS average surprise, in bits, from the model's own point of view.
Why the log though? The standard answer is that you are computing cross-entropy between the model's output and a one-hot distribution with all the mass on the true token, and the zeros cancel everything else out. That is technically true and explains nothing. If it all evaporates, why not call it log loss?
Here is the version that actually explains it. Take a pattern common enough to appear many times in the corpus, like my name is ___. Common names appear often, rare names rarely. The model emits one distribution q over all possible continuations. The total loss contributed by every instance of that pattern is:
The one-hot framing is what a single training example looks like. Cross-entropy is what the corpus looks like in aggregate. The empirical distribution never appears in any one gradient step, but it is what the sum is converging against, and it is why the loss bottoms out when the model matches the statistics of the data rather than memorising individual tokens.
| Convention | Value | Note |
|---|---|---|
| natural log | nats | what frameworks actually use. cleaner derivatives. |
| log base 2 | bits | divide nats by ln 2 = 0.6931. same loss, readable units. |
| bits per character | BPC | total bits / character count. tokeniser-independent, so it compares across model families. |
| perplexity | 2H or eH | effective branching factor. same number, exponentiated. |
Pretraining wastes most of the signal available at each position. The teacher forcing target is one token. Everything the model believed about the other fifty thousand candidates is discarded.
Distillation fixes that. Run a large model over the same corpus. At each position, instead of comparing the small model's distribution against a one-hot spike, compare it against the large model's full distribution. The loss is cross-entropy between two real distributions, both dense.
PRETRAINING DISTILLATION ─────────── ──────────── target = one-hot target = teacher's distribution Anna █ Anna ████████ John · John ██████ Ravi · Ravi ███ Mei · Mei ███ Olu · Olu ██ ... · ... ▌ one token of signal the whole shape, every step per position ───────────────────────────────────────────────────────────── you need millions of my name is ▁ examples before the empirical p is well estimated. the teacher hands you a usable p on the first one.
The chess analogy is exact: learning by watching a game, versus having a stronger player talk through every candidate move and how much they like each one. Same position, vastly more signal.
Cross-entropy has a floor, and the floor depends on p. That makes raw cross-entropy awkward to compare across datasets: a low number might mean a good model, or it might mean easy data. Subtract the floor and you get the part that is your fault.
Zero when the distributions match, positive otherwise, and never negative. It is not a metric: it violates symmetry and the triangle inequality. That asymmetry is not a defect to be patched, it is a knob.
FORWARD D(p ‖ q) ───────────────────────────── Σ runs over the points where p is large. if q→0 somewhere p>0, that term blows up. the model is punished for ruling out something that happens. ⟹ mass-covering it hedges. it would rather be vague than caught out. used by: pretraining, distillation, MLE generally
REVERSE D(q ‖ p) ───────────────────────────── Σ runs over the points where q is large. terms where q→0 vanish no matter what p does there. the model can ignore entire regions for free. ⟹ mode-seeking it picks one peak and commits. sharp, and lossy. used by: variational inference, some RL objectives
The cleanest numerical demonstration uses two outcomes. Let a = (0.9, 0.1) and b = (0.5, 0.5). All four quantities are different:
| Quantity | Bits | Reading |
|---|---|---|
| H(a) | 0.469 | a is skewed, so it is cheap to encode |
| H(b) | 1.000 | a fair coin costs exactly one bit |
| H(a, b) | 1.000 | a b-code on a-data. every symbol costs 1 bit regardless. |
| H(b, a) | 1.737 | an a-code on b-data. the rare branch costs 3.32 bits and now fires half the time. |
| D(a ‖ b) | 0.531 | |
| D(b ‖ a) | 0.737 | same pair, different number. order is load-bearing. |
Σ p log(p/q), rather than as a difference of two entropies. It is the same thing: split the log of the quotient and the two halves are exactly H(p,q) and −H(p). The compact form is shorter but it hides the interpretation, which is that you are measuring an excess over a floor.
Everything above is verifiable on a laptop in under a minute. The numbers on this page come from rebuilding the 2002 experiment on the same corpus (UDHR, via NLTK's udhr2 package), then rerunning it with an explicit probability model to see what changes.
Setup: 32 languages, 7000 characters of reference per language, a disjoint 1500-character snippet as the probe. Distance is symmetrised KL, and the tree is UPGMA over that matrix. Nearest-neighbour accuracy is scored against ground-truth family labels. The 2002 paper used a normalised distance and Fitch-Margoliash from PHYLIP; the simpler pair here is easier to read and lands in the same place.
| Method | Self BPC (English) | Self BPC (Italian) | Correct family NN |
|---|---|---|---|
| gzip, level 9 | 2.651 | 2.405 | 24 / 32 = 75.0% |
| order-5 char model | 2.366 | 2.103 | 24 / 32 = 75.0% |
The explicit model compresses better on every language, which is the expected result: it emits calibrated probabilities where DEFLATE only finds repeated substrings. On this corpus it does not cluster better, because 7 KB is not much to train on and seven of the eight misses are the same in both runs: English, Basque, Maltese, Turkish, and the Celtic three. The eighth differs. gzip loses Lithuanian to Slovene; the n-gram model loses Hungarian to Icelandic.
The stable failures split into two kinds. Basque, Maltese and Turkish have no relative at all in the sample, so a nearest neighbour of any kind is a miss by construction. Irish, Welsh and Breton do have each other, but Goidelic and Brythonic orthography share almost nothing at the character level, so the family never coheres.
English drifting toward Romance in both runs is not an error either. A character-level model sees orthography, not descent, and English spelling carries an enormous Norman and Latin loan vocabulary.
bits per character of English, one axis, six rungs ─────────────────────────────────────────────────────────────── uniform over 256 bytes ████████████████████████████████ 8.00 letter frequencies only █████████████████ 4.18 gzip on 7 KB ███████████ 2.65 order-5 model on 7 KB █████████ 2.37 Shannon 1951, humans ████ 1.10 a good modern LLM ███ 0.70 every rung is the same measurement: −log₂ of the probability the model assigned to what actually came next. the units do not change anywhere along the ladder, which is the whole reason gzip and a 70B transformer are comparable at all.
The other side of the bridge is Delétang et al. (2023), who ran it in the opposite direction: use a frozen Chinchilla 70B as the probability model inside an arithmetic coder and compress things it was never trained on. It reaches 43.4% on ImageNet patches against PNG's 58.5%, and 16.4% on LibriSpeech audio against FLAC's 30.3%. A text model beating domain-specific codecs on images and audio is a strong statement that what it learned is general structure, not English.