Lion Cipher Lab

Break and defend an identity system with Unicode normalization and confusables

• Foundations
• Advanced
• ~135 min
• Free · no account required

Why it matters

Every access-control decision your company makes ends in a string comparison. Is the person logging in the same person who registered? Is this the account the password-reset email should go to? Is this username on the allowlist? Somewhere underneath all of that sits a function that turns what a human typed into a key, and that function is the real security boundary -- not the login form, not the session cookie, not the MFA prompt. Unicode is what that function has to survive, and almost nobody in security has actually measured what it does to their identity namespace. This lab makes you…

What you'll learn

• what canonicalization actually is and why it is the security control under every username, email local part, filename, domain and access-control decision -- the function that answers 'are these two strings the same principal', which almost no team has ever measured
• the four Unicode normalization forms measured in BOTH directions on a real corpus: how many visually distinct pairs each one wrongly MERGES (NFKC: 6 of 7,021) and how many visually identical pairs it still MISSES (NFKC: 91), because normalization both creates collisions and fails to prevent them
• the ORDER bug that CWE-180 is named for -- check-uniqueness-then-canonicalize versus canonicalize-then-check-uniqueness -- run live against one registration stream with the same key function, producing 26 silent account overwrites versus 0
• str.lower() versus str.casefold() measured against each other on the German sharp s, the capital sharp s, the Kelvin sign, the long s, the final sigma and the Turkish dotted and dotless i, plus a measurement (not an assertion) of whether Python's lower() honours a locale
• writing a UTS #39 style skeleton() from scratch with NO data file: a name-derived pass over unicodedata.name() covering 1,430 codepoints, a hand table for the homoglyphs whose names do not mention a Latin letter, and the honest finding that the name heuristic maps U+0443 backwards
• reading a confusable result as a FLOOR and not a ceiling, and knowing that a skeleton is a hunting index for review queues rather than a primary key -- with the over-merge cost measured on the same corpus
• detecting single-script versus mixed-script identifiers from a hand-written block table, and measuring the FALSE POSITIVES a restricted profile imposes on legitimate Cyrillic, Greek, Japanese and Korean users instead of pretending the restriction is free
• measuring how many invisible and directional carriers survive strip(), NFKC, NFKD, a unicodedata.category filter and a real policy -- and why stripping category Mn is a name mangler rather than a security control
• the length-limit-before-normalization vector: finding the codepoint that expands 1 character to 18 (3 utf-8 bytes to 33) and chaining a passing len() check to a silently truncating database column into an account takeover
• reading what the standard library idna codec really implements by measuring what it accepts and transforms, rather than trusting the module name -- including the sharp s, the final sigma and a zero width joiner that it deletes outright

Steps in this lab

1. The one function that decides who you are
2. [0] Set up: one folder, three files, nothing to install
3. Create idsys.py -- the identity system and every primitive it needs
4. Create corpus.py -- 119 usernames and TWO separate ground truths
5. Create harness.py -- the measurement rig, then run it once
6. [0] Read the environment block first, because these numbers move
7. [1] The default system: a clean namespace, and 42 duplicates it cannot see
8. [2] What the four normalization forms actually do
9. [2] The table: what each form buys, and what it breaks
10. [2] The six false merges NFKC creates, named
11. [2b] NFKC can make a string LONGER, by a factor of eighteen
12. [3] The ORDER bug: two lines apart, 26 takeovers apart
13. [3] Reading the overwrite log, and the fix
14. [4] lower() is for looking at. casefold() is for comparing.
15. [4] Which comparisons actually collide, and the corpus-wide cost
16. [4] Is Python's lower() locale sensitive? Measure it, do not assert it
17. [5] Building a confusable skeleton from scratch, with no data file
18. [5] Where the heuristic is WRONG, and why that is the important result
19. [5] What the skeleton catches that NFKC and casefold missed
20. [5] What the skeleton gets WRONG -- it is an index, not an equality test
21. [6] Twelve invisible carriers against six defences
22. [6] Why the naive category filter is a name mangler
23. [6] The RIGHT-TO-LEFT OVERRIDE, modelled honestly
24. [7] Script mixing, and the false positives nobody publishes
25. [7] The locked-out users, by name
26. [8] IDNA through the standard library: measure it, do not trust the name
27. [8] Punycode collisions and the round trip that is not one
28. [9] End to end, BEFORE: the password reset that is not yours
29. [9] BEFORE, the other path: the attacker's credential on the admin row
30. [9] AFTER: zero, and exactly which control did it
31. [9] The bill: 13 correct merges, 12 real people locked out
32. [9] The same fix with the knob turned down
33. [10a] Failure mode one: normalize here, but not there
34. [10b] Failure mode two: normalize for display, key on raw
35. [10c] Failure mode three: a length limit before normalization
36. [10d] Failure mode four: comparing canonical against raw
37. [11] The scoreboard: three tables, and why the third one is the real one
38. When NOT to do any of this
39. The way a professional does it differently
40. Take this to work on Monday: the review checklist

Open the full interactive lab →