What Is Zstandard (zstd)? Fast Compression Explained
Zstandard (zstd) is a lossless compression algorithm that pairs gzip-beating ratios with very fast decompression. How it works and when to use it.
Zstandard, usually written zstd, is a lossless compression algorithm designed to be fast at both ends: it compresses quickly, decompresses very quickly, and typically achieves better compression ratios than gzip at comparable speeds. Created by Yann Collet at Facebook and open-sourced in 2016, it has become a default choice for file systems, package managers, databases, and increasingly for HTTP responses.
Its defining feature is range. A single algorithm covers everything from “barely slower than copying memory” to “squeeze out every byte, however long it takes”, selected with one numeric level, while decompression stays fast regardless of the level used to compress.
How Zstandard works
Like gzip and Brotli, zstd combines two classic stages: finding repetition, then encoding what is left efficiently.
Stage one: match finding. The compressor scans the input for byte sequences it has already seen and replaces later occurrences with short references of the form “copy N bytes from M bytes back”. This is the LZ77 family of techniques. Zstd’s match finders range from a simple hash lookup at low levels to elaborate search strategies at high levels that consider many candidate matches and pick the cheapest overall encoding.
Stage two: entropy coding. The output of stage one is a stream of literals (bytes that did not match anything) and sequences (match lengths and distances). Zstd encodes these using two methods: Huffman coding for literals and Finite State Entropy (FSE), an implementation of asymmetric numeral systems, for the sequence data. FSE achieves compression close to arithmetic coding while being fast enough to decode at high throughput on ordinary CPUs.
That second stage is a large part of why zstd outperforms gzip. Gzip’s DEFLATE format uses Huffman coding alone, which cannot represent symbol probabilities as precisely, and it is limited to a 32 KB window for finding matches. Zstd supports far larger windows, so it can exploit repetition across much longer distances in the input.
Compression levels
Zstd exposes levels from 1 to 19 for everyday use, with higher “ultra” levels available at the cost of much more memory. It also supports negative levels, which trade ratio for even more speed.
- Low levels (1 to 3) aim for throughput. They are well suited to compressing data on the fly, such as logs, network traffic, or database pages, where CPU time matters more than the last few percent of size.
- Middle levels balance ratio and speed for general file storage.
- High levels spend much more CPU time searching for matches. They make sense for data compressed once and decompressed many times, such as software packages or static assets.
The important asymmetry: decompression speed barely changes with the compression level. A file compressed at level 19 decompresses about as quickly as one compressed at level 3. That makes high levels attractive for anything distributed widely, because the expensive work is paid once by the publisher.
Dictionary compression
Compression algorithms learn from the data they have already seen. With small inputs, such as a 1 KB JSON response, there is not enough data to learn from, so ratios are poor.
Zstd addresses this with trained dictionaries. You feed the trainer a representative sample of your small messages, and it produces a dictionary of common substrings and statistics. Both the compressor and decompressor load the same dictionary, so even the first bytes of a tiny message can reference patterns from it. For workloads made up of many small, similar records, such as API payloads, cache entries, or telemetry events, dictionaries can improve ratios dramatically.
The catch is operational: both sides must have the exact same dictionary, so you need a way to version and distribute it.
Zstandard vs gzip vs Brotli
| gzip | Brotli | Zstandard | |
|---|---|---|---|
| Entropy coding | Huffman | Huffman with context modelling | Huffman and FSE |
| Match window | 32 KB | Up to 16 MB | Configurable, very large at high levels |
| Built-in dictionary | None | Static web-oriented dictionary | None; supports trained dictionaries |
| Fast-level speed | Good | Moderate | Very fast |
| Max-level ratio | Lowest | Very high | Very high |
| Decompression speed | Fast | Fast | Very fast |
| Typical home | Legacy HTTP, archives | Static web assets | File systems, packages, databases, dynamic HTTP |
Brotli’s strength on the web comes partly from its built-in dictionary of common HTML, CSS, and JavaScript fragments, which helps on small text files. For a deeper look at those two, see gzip vs Brotli compression. Zstd tends to win when compression must happen in real time, because its fast levels deliver good ratios at very high throughput.
Zstandard on the web
HTTP supports zstd through the standard content negotiation mechanism. A browser that understands it includes zstd in its Accept-Encoding request header; a server that supports it compresses the response and labels it with Content-Encoding: zstd. Clients that do not advertise zstd receive Brotli or gzip instead, so adding it is safe.
The strongest case is dynamic responses: server-rendered HTML, API JSON, and anything else generated per request. Compressing those at a high Brotli level on every request is expensive, while zstd’s low levels compress them quickly with good ratios. That saves server CPU and can shave time off time to first byte. For static assets that can be precompressed at build time, Brotli at its highest level remains a strong option.
If you serve through a CDN, check whether it handles zstd negotiation and caching for you. Because the compressed bytes differ by encoding, responses need a Vary: Accept-Encoding header so caches store each variant separately, as covered in HTTP caching headers.
Beyond HTTP
Most zstd usage happens out of sight:
- File systems such as Btrfs and ZFS can compress blocks transparently with zstd.
- Linux package formats in several distributions use zstd so packages download small and install fast.
- Databases and data formats use it for on-disk pages and columnar files, where fast decompression speeds up queries.
- Streaming and logging pipelines compress data in flight at low levels to cut bandwidth and storage.
The takeaway
Zstandard combines LZ77-style match finding with modern entropy coding to deliver better ratios than gzip and much faster decompression, with one level setting that spans real-time to archival use. Pick low levels for data compressed on the fly, high levels for data compressed once and read many times, and trained dictionaries for streams of small, similar messages. On the web, offer it alongside Brotli and gzip through Accept-Encoding, and let negotiation pick the best format each client supports.
Keep reading
The Lycoris Team · · 5 min read Huge Pages Explained: Bigger Memory Pages, Fewer TLB Misses
Huge pages map memory in 2 MB or 1 GB chunks instead of 4 KB, cutting TLB misses for memory-heavy workloads. How they work and when to use them.
Takina · · 5 min read HTTP 103 Early Hints: Faster Page Loads Explained
HTTP 103 Early Hints lets a server tell the browser which assets to fetch before the final response is ready. How it works, when it helps, and its limits.
The Lycoris Team · · 5 min read What Is a Flame Graph? Reading CPU Profiles Visually
A flame graph visualizes sampled stack traces so the widest bars show where a program spends its time. How to read one, build one, and avoid common traps.