UTF-8 vs ASCII

The short version: ASCII is a character set with 128 entries. UTF-8 is a way of storing the roughly 150,000 characters of Unicode, and its first 128 entries are byte-for-byte identical to ASCII.

That last clause is not a detail. It is the reason UTF-8 won, and it is why a question that sounds like a comparison of two rivals is really a question about how one of them absorbed the other.

What ASCII was

The American Standard Code for Information Interchange was standardized in 1963 and finalized in 1967. It uses seven bits, which gives 128 possible values: 95 printable characters and 33 control codes.

The printable set is the English keyboard — uppercase A to Z, lowercase a to z, the digits, and common punctuation. The control codes are the giveaway about what ASCII was for. Codes like carriage return, line feed, bell and form feed are instructions to a teleprinter, a machine that physically returned a print head to the left margin and advanced the paper. Those two separate operations are why Windows still ends lines with two characters and Unix with one, sixty years after the last teleprinter was unplugged.

Seven bits was a deliberate economy in an era when a bit of storage cost real money. Every byte had a spare bit, often used for parity checking on noisy lines.

What went wrong with it

ASCII had no å, no é, no ü, no ñ. It had no Greek, no Cyrillic, no Arabic, no Hebrew, and nothing at all for the tens of thousands of characters used in Chinese, Japanese and Korean. It was an American standard and it said so in the name.

The obvious fix was the spare eighth bit, which doubles the range to 256. Everyone did this, and everyone did it differently. Latin-1 filled the upper half with Western European letters. Windows-1252 did nearly the same thing but not quite. KOI8-R used it for Cyrillic, ISO 8859-7 for Greek, and East Asian scripts needed multi-byte schemes of their own such as Shift-JIS and Big5.

The result was that byte 229 meant å on one machine, х on another, and ε on a third. The file could not say which one it meant. Text moved between systems and turned to nonsense, a phenomenon common enough to acquire a Japanese name that English borrowed: mojibake.

There was no way to write a document containing both Swedish and Greek, and no way for a program to know what it was reading. The eighth bit had bought a second alphabet, not a solution.

What UTF-8 did

Unicode answered the first half of the problem by assigning a unique number, a code point, to every character in every script. What Unicode is covers that catalogue in detail.

Assigning numbers leaves the second half open: how do those numbers become bytes? UTF-8 is one answer, designed by Ken Thompson and Rob Pike in 1992, reportedly sketched out over dinner on a placemat. It is variable-length, using one to four bytes per character:

Code points	Bytes	Covers
U+0000 to U+007F	1	ASCII
U+0080 to U+07FF	2	Latin with accents, Greek, Cyrillic, Hebrew, Arabic
U+0800 to U+FFFF	3	Chinese, Japanese, Korean, most remaining scripts
U+10000 and above	4	Emoji, historic scripts, rare characters

The design has three properties that matter more than the compression.

Why backward compatibility decided it

Every ASCII file is already a valid UTF-8 file. The first 128 code points encode as a single byte with the top bit clear, which is exactly what ASCII was. No conversion, no migration, no flag day. A system could declare itself UTF-8 on a Tuesday and every existing document, filename, config file and protocol header kept working unchanged.

The competing approach, UCS-2 and later UTF-16, made the opposite choice: fixed-width two-byte characters, which meant every ASCII file had to be converted, every C string routine rewritten, and every file format redefined. Windows and Java adopted it early and have been carrying the consequences ever since, including the need for a byte order mark to say which end of each pair comes first. UTF-8 has no byte order, so that entire class of problem does not exist for it.

No byte of a multi-byte character can be mistaken for ASCII. Continuation bytes always have the top bit set. A C library scanning for a / or a NUL terminator cannot accidentally match the middle of a Chinese character. That single property let Unix filesystems and network protocols carry Unicode without being rewritten.

It is self-synchronizing. Land anywhere in a UTF-8 stream and you can tell from the byte itself whether you are at the start of a character or inside one, then walk backwards a byte or two to find the boundary. A corrupted or truncated stream loses one character, not everything after it.

Being smaller for English text was a bonus, not the argument. The argument was that nobody had to change anything.

What this means in practice

“Plain ASCII” is no longer a meaningful category. If a system asks for ASCII, it almost certainly accepts UTF-8, because the two are indistinguishable for text that stays inside the 128 characters. The distinction only becomes real when a character outside that range shows up.

Byte count and character count are different numbers. In UTF-8, å is two bytes, a Chinese character is three, and most emoji are four. A database column declared as 20 characters and a limit of 20 bytes are not the same limit, and the difference surfaces at the worst moment: when a name with an accent in it is silently truncated halfway through a character.

Mojibake still happens, and it always has the same cause. Something wrote UTF-8 and something else read it as a single-byte code page. å becoming Ã¥ is the signature. It is the most common complaint about CSV files opened in Excel, and you can confirm it in seconds by opening the file in a viewer that reports the encoding it detected rather than guessing silently.

Use UTF-8 for everything new. There is no remaining case for choosing a legacy code page, and the web has settled: well over 98 percent of pages declare UTF-8. The only live decision left is whether to put a byte order mark on the front, and the answer is almost always no.