UTF-8 Explained: How Unicode Text Becomes Bytes

UTF-8 is a character encoding that represents every Unicode character as a sequence of one to four bytes. Plain English letters take a single byte, identical to ASCII, while accented letters, other scripts, symbols, and emoji take two, three, or four bytes. That combination of ASCII compatibility and full Unicode coverage made UTF-8 the default encoding of the web, of most modern operating systems and programming languages, and of nearly every text-based file format and protocol in use today.

Characters, code points, and encodings

Text handling gets confusing because three different ideas are often called “characters.”

• Unicode is a catalog. It assigns each character a number called a code point, written like U+0041 for A or U+00E9 for é. The range runs from U+0000 to U+10FFFF, which leaves room for over a million code points.
• An encoding is a rule for storing those numbers as bytes. UTF-8, UTF-16, and UTF-32 are all encodings of the same Unicode code points.
• What a reader sees as one character, such as a flag or an accented letter, may be several code points combined. Those user-perceived characters are called grapheme clusters, and how they affect string handling in code is covered in JavaScript strings, code points, and graphemes.

Before Unicode, there were hundreds of incompatible encodings, each covering a handful of languages. Opening a file in the wrong one produced garbled text. Unicode fixed the catalog problem; UTF-8 made the storage problem practical.

How UTF-8 encodes a code point

UTF-8 uses the high bits of each byte to say how long the sequence is. The rest of the bits carry the code point.

Code point range	Bytes	Byte pattern
U+0000 to U+007F	1	0xxxxxxx
U+0080 to U+07FF	2	110xxxxx 10xxxxxx
U+0800 to U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Take é, code point U+00E9, which is 11101001 in binary. It falls in the two-byte range, so its 11 payload bits, 00011 101001, are slotted into 110xxxxx 10xxxxxx, producing 11000011 10101001, or C3 A9 in hex.

The euro sign € is U+20AC and needs three bytes: E2 82 AC. Most emoji live above U+FFFF and need four bytes.

Why the design is so clever

The bit patterns are not arbitrary. They give UTF-8 several properties that matter in practice.

• ASCII compatibility. Any valid ASCII file is already valid UTF-8, byte for byte. Decades of tools, protocols, and source code kept working.
• No stray ASCII bytes. Every byte in a multi-byte sequence has its high bit set. A byte like 0x2F (/) or 0x00 can never appear in the middle of another character, so code that scans for delimiters, path separators, or null terminators does not need to understand UTF-8 at all.
• Self-synchronizing. Continuation bytes always start with 10, and leading bytes never do. If you land in the middle of a stream, you can skip forward to the next character boundary without decoding from the start.
• No byte-order problem. UTF-8 is defined as a sequence of single bytes, so the big-endian vs little-endian question that affects UTF-16 and UTF-32 does not arise.
• Sort order preserved. Sorting UTF-8 strings byte by byte gives the same order as sorting by code point.

UTF-8 vs UTF-16 vs UTF-32

	UTF-8	UTF-16	UTF-32
Unit size	1 byte	2 bytes	4 bytes
Bytes per code point	1 to 4	2 or 4	Always 4
ASCII compatible	Yes	No	No
Byte order issues	None	Yes (BOM or labeled variant)	Yes
Size for English text	Smallest	About double	About quadruple
Size for many East Asian scripts	3 bytes per character	2 bytes per character	4 bytes per character
Common in	Web, files, Linux, APIs	JavaScript strings, Java, Windows APIs	Some internal processing

UTF-16 uses one 16-bit unit for most common characters and a pair of units, called a surrogate pair, for code points above U+FFFF. It is the internal string representation of JavaScript, Java, and the Windows API for historical reasons: they adopted Unicode when it was expected to fit in 16 bits. UTF-32 is simple, one unit per code point, but wasteful, so it is mostly used internally where fixed-width indexing helps.

Even where UTF-16 is used in memory, data almost always crosses the network and lands on disk as UTF-8. Web pages declare <meta charset="utf-8">, JSON is specified as UTF-8 in practice, and most databases default to it.

Common UTF-8 pitfalls

Mojibake. When UTF-8 bytes are decoded as a legacy single-byte encoding, é (C3 A9) shows up as é. Seeing Ã followed by another odd character is a strong sign of a UTF-8 versus Latin-1 mix-up somewhere in the pipeline.

Byte length is not character count. "café" is four characters but five bytes in UTF-8. Database column limits, protocol length fields, and truncation logic must be clear about which they mean. Truncating at a byte boundary can split a multi-byte character and produce invalid output.

Database “utf8” that is not full UTF-8. MySQL’s older utf8 character set stores at most three bytes per character, so it cannot hold emoji or other four-byte code points. The full encoding there is utf8mb4. This is one of the encoding details worth knowing when comparing PostgreSQL and MySQL.

The byte order mark. Some editors write the bytes EF BB BF at the start of UTF-8 files. UTF-8 does not need a BOM, and it can break shell scripts, CSV parsers, and concatenated files that do not expect it.

Invalid sequences as an attack vector. Overlong encodings, such as representing / with two bytes instead of one, were historically used to slip past filters that checked raw bytes. Modern decoders reject them, but it is one reason to decode and validate input before inspecting it, a principle also covered in input validation vs sanitization vs escaping.

Encoding vs compression vs Base64

UTF-8 is often confused with other byte transformations. It only maps code points to bytes. Making text smaller is the job of compression such as gzip or Brotli, which work on top of UTF-8 bytes. Base64 is a third, unrelated step that turns arbitrary bytes into ASCII-safe text for transport, as in the header and payload of a token you can inspect with the JWT decoder.

The takeaway

UTF-8 encodes each Unicode code point in one to four bytes, using the high bits of each byte to mark sequence length. It is backward compatible with ASCII, free of byte-order issues, safe for byte-oriented tools, and compact for most text, which is why it became the universal default. Declare it explicitly, keep byte length and character count separate in your head, use utf8mb4 in MySQL, and decode before you validate.