Lion Cipher Lab

Write a strict UTF-8 decoder from scratch, and catch overlong and surrogate encodings

• Foundations
• Advanced
• ~75 min
• Free · no account required

Why it matters

Every program that reads text from outside first turns BYTES into CHARACTERS. That step is decoding, and it is a security boundary that almost nobody looks at. UTF-8 has one rule that matters more than the rest: every character has exactly ONE correct spelling in bytes. A decoder that also accepts a second, longer spelling (an OVERLONG encoding) or a piece of UTF-16 that is not a character (a SURROGATE) gives an attacker a way to write '/' or '.' that does not look like '/' or '.' to anything checking the raw bytes. Around 2000 that exact gap let attackers read files outside the web root of…

What you'll learn

• explaining the difference between BYTES and CODE POINTS, and reading a UTF-8 byte sequence by hand from its bit layout (lead byte, continuation bytes, the payload bits)
• writing a UTF-8 encoder that round-trips all 1,112,064 Unicode scalar values byte-for-byte against Python's own (0 mismatches) and refuses all 2,050 surrogates and out-of-range values
• implementing a strict decoder straight from RFC 3629 / Unicode Table 3-7 that agreed with Python's codec on every one of 1.7 million test inputs, including the exact error offset and the U+FFFD 'maximal subpart' replacement
• differential testing against an oracle: exhaustive sweeps of every 1-, 2- and E0..EF 3-byte sequence plus seeded random 4-byte samples, so the claim 'my decoder is correct' rests on a count, not a feeling
• measuring how dangerous a 'just mask the bits' decoder is: it accepted 6,272 of 47,232 invalid 2-byte inputs and 100% of invalid 3-byte inputs with a valid-looking lead, turning C0 AF, E0 80 AF and F0 80 80 AF into '/'
• reproducing the check-then-decode path traversal shape behind CVE-2000-0884 on a local toy server (4 of 11 payloads read the secret, 5 of 11 with Windows path rules) and fixing it so 0 of 11 do
• choosing an error handler with numbers: errors='ignore' turned .<FF>./x into ../x, errors='replace' collapsed 47,232 distinct two-byte inputs into shared outputs, surrogateescape round-trips but raises on a plain encode
• knowing where strictness stops: valid fullwidth dots still became ../ after NFKC, and a pure-Python decoder ran 53 to 235 times slower than the built-in one

Steps in this lab

1. Bytes are not characters: what a decoder does, and the one rule UTF-8 adds
2. Set up: a terminal, Python, one folder
3. Write utf8.py: an encoder, a naive decoder, and a strict decoder from Table 3-7
4. The classic malformed inputs, one row each
5. Differential testing: prove your decoder right on 1.7 million inputs, and count the naive one's holes
6. Why it matters: a toy file server, a byte-level check, and a naive decode
7. Honest negative 1: strict decoding does not stop look-alike characters
8. Honest negative 2: every error handler costs something -- measured
9. Honest negative 3: your decoder is correct, and far too slow to ship
10. Optional: run Markus Kuhn's public stress test through both decoders
11. Do it better: validate at the boundary, decide on the final form, then clean up

Open the full interactive lab →