Compression Formats Deep Dive

The Landscape

Twelve formats, three jobs. An archive bundles files and metadata into one stream but compresses nothing. A compressor shrinks one data stream but knows nothing about files. Some containers do both at once. Keeping the two jobs straight is most of the map - the rest is picking a point on the ratio/speed curve.

Generations

The Format Matrix

Who is who. Algorithms first: nearly everything is LZ77 plus an entropy coder - the differences are the window size, the coder (Huffman vs range coding vs finite state entropy), and how much CPU the format is willing to spend.

Layering and Pipelines

TAR carries no compression of its own; it is the archive layer that bundles files and metadata, then a separate codec compresses the resulting stream. That separation is why the .tar.* combinations exist at all - pick the archive once, swap the compressor to taste.

  the two-layer model

  files / dirs ──► TAR (archive: files + metadata, 1:1) ──► CODEC ──► one file

      tar + GZIP   =  .tar.gz
      tar + XZ     =  .tar.xz
      tar + ZSTD   =  .tar.zst

  integrated containers (archive + codec fused):

      ZIP = .zip       7-ZIP = .7z       RAR = .rar

the same idea in commands# archive layer stays tar; only the codec flag changes
tar -czf site.tar.gz  site/          # gzip: everywhere, fast enough
tar -cJf site.tar.xz  site/          # xz: max ratio, slow
tar --zstd -cf site.tar.zst site/    # zstd: the modern default

# integrated: archive + codec in one step
zip -r site.zip site/
7z a -mx=9 site.7z site/

# bare stream compressors
zstd -19 --long=27 big.log           # long-window mode
lz4 -1 assets.bin

Ratio vs Speed

Speeds are normalized to zstd level 3 = 1.00x, single core, modern x86-64; higher is faster. Decompression is always faster than compression for the same format. Ratios are typical / best-case on mixed real-world data.

  denser ▲                        compress speed, log scale (zstd-3 = 1x)

  8:1 |  XZ  7-ZIP
      |    LZIP
  6:1 |      BROTLI
  5:1 |          RAR
  4:1 |             BZIP2
  3:1 |             ZLIB ZIP GZIP      ZSTD
  2:1 |                                              LZ4
      +──────────────────────────────────────────────────────
       0.1x        0.3x        0.6x     1x           4x+
      slow + dense corner                 fast + loose corner

Format Capabilities

The flags that decide integration: can it stream through a pipe, can you seek into the middle, does it compress files as one solid block, will it use your cores.

¹ pigz: parallel gzip implementation (compression multi-threading). ² zlib-ng: drop-in zlib replacement with SIMD; enables faster decode.

Window / dictionary range

The window is how far back a format can reference. A 32 KB DEFLATE window cannot see repetition 1 MB apart; a 128 MB zstd window can. Log scale:

  ZIP      32 KB    █
  GZIP     32 KB    █
  ZLIB     32 KB    █
  LZ4      64 KB    ██
  BZIP2    900 KB   ██████          (block size)
  BROTLI   16 MB    ███████████     → 1 GB large-window ext
  XZ       64 MB    █████████████   → ~4 GB max
  7-ZIP    64 MB    █████████████   → ~4 GB max
  ZSTD     128 MB   ██████████████  → ≤2 GB (trained dict ≤2 GB)
  LZIP     1 GB     ██████████████████
  RAR      1.3 GB   ███████████████████

Error Recovery

What happens when bits flip. Three tiers: formats that can repair or route around damage, formats that resync at the next block boundary and lose only one block, and streams where corruption kills everything after it.

Choosing a Format

• Start at zstdNear-gzip ratios at several times the speed, seekable frames, strong integrity, threads well. It is the default for a reason.
• Swap only on a constraintNeed every last byte: XZ or 7-ZIP. Need wire speed: LZ4. Serving text to browsers: BROTLI. Surviving bit rot: RAR or LZIP.
• Pick the archive layer separatelyTAR for unix pipelines and backups, ZIP when the receiver is unknown - it opens everywhere with nothing installed.