What is Unicode?
Unicode assigns a number to every character in every writing system. UTF-8 decides how those numbers become bytes. Confusing the two causes most text bugs.
Unicode is a catalogue. It assigns a unique number — a code point — to every character
in every writing system humans use, plus a great many symbols that are not writing at all.
The letter A is code point 65. The letter å is 229. The waving hand emoji is 128,075.
That is all Unicode does. It says which number means which character. It says nothing about how those numbers are stored in a file, and that second question is where nearly every text bug lives.
Unicode is not UTF-8
This distinction is the one worth internalising.
Unicode is the mapping from characters to numbers. UTF-8 is one way of turning those numbers into bytes. UTF-16 is another. They describe the same characters and produce completely different files.
UTF-8 is variable-width: plain English letters take one byte each, most European accented letters take two, most Asian characters take three, and emoji take four. That design is why UTF-8 won — an English text file in UTF-8 is byte-identical to the same file in ASCII, so thirty years of existing files kept working.
It is also why å becomes Ã¥ when a file is read with the wrong assumption. Those are
its two bytes, interpreted one at a time.
Code points are not characters, either
Here the catalogue gets subtle in ways that break code.
A family emoji looks like one character. It is four people plus three invisible joining
characters — seven code points that a font renders as a single picture. A flag is two
letters from an alphabet that exists only to make flags. An é might be one code point,
or it might be e followed by a combining accent, and both look identical on screen while
comparing as different strings.
What a reader perceives as one character is called a grapheme cluster. Counting those correctly is why a properly built character counter reports a family emoji as one character while a naive one reports five — and why removing emoji has to match whole sequences rather than individual code points.
The invisible characters
Unicode includes characters that render as nothing at all: zero-width spaces, non-breaking spaces, soft hyphens, byte order marks, and directional marks. Every one of them has a legitimate use, and every one of them causes confusion when it appears where it was not expected.
A zero-width space pasted into a CSV file breaks the column silently. A non-breaking space in code looks exactly like a normal space and fails to compile. These are not exotic edge cases — they arrive routinely from word processors and web pages.
Homoglyphs
Different code points can look identical. Latin а and Cyrillic а are different
characters that most fonts render the same way. This is the mechanism behind homoglyph
phishing: a domain name that reads as apple.com but is not.
It is also a routine nuisance in ordinary data, where a name copied from one system will not match the same name in another because one of them contains a Cyrillic letter nobody typed deliberately.