Lion Cipher Lab

Decode and normalize encoded prompt injection before your input filter, and measure what still slips through

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Most LLM apps put a filter in front of the model: a list of phrases like 'ignore previous instructions' checked against everything a user types and every document the app retrieves. It feels like a defence, and against the plain phrase it is. But the filter matches characters, and the model reads meaning. Write the same instruction in base64, hex, ROT13 or leetspeak and the filter sees noise while a capable model can still decode and follow it - the 2023 Jailbroken paper by Wei, Haghtalab and Steinhardt documents exactly this gap. In this lab the naive filter caught every plain canary…

What you'll learn

• explaining why an encoded instruction beats a keyword filter: the filter matches characters, while a capable model can read base64, hex, ROT13 and leetspeak (Wei et al. call it mismatched generalization)
• building a labelled test corpus that includes LEGITIMATELY encoded benign content - data URIs, hashes, hex dumps, base64 log fields, URL queries, JWTs - because that is where a decoder's false positives come from
• measuring a naive regex filter per encoding: 1.000 on plain canary injections, 0.000 on every one of 13 encoded variants
• writing a bounded decode-and-normalize layer (base64, hex, percent-encoding, ROT13, leet folding, reversal) that generates candidate views breadth-first with depth, view-count and input-size limits plus deduplication, and runs the filter on every view
• re-measuring detection (0.089 -> 0.756 overall) AND false positives (0.032 -> 0.063, all from quoted payloads in security discussion) and reading which decode path flagged each item
• showing with numbers why each bound exists: 5424 views at depth 8 without dedup against 36 with it, 19.6 million characters processed without a size limit against 520 thousand with one, and the payload the size limit itself misses
• naming what still slips through - a cipher the layer does not know, a payload split across fields, nesting deeper than the bound - and pricing each patch in false positives and work before deciding where a filter belongs: one leaky layer in front of privilege separation

Steps in this lab

1. What an input filter is, and why an encoded instruction walks past it
2. Open a terminal, check Python, and make a scratch folder
3. Build the labelled corpus: canary attacks in 14 encodings, and benign text that is legitimately encoded
4. The naive filter: six regex patterns, and what it sees
5. Build the decode-and-normalize layer: bounded views, one filter call per view
6. Measure detection per encoding: naive against careful against aggressive
7. Read the false-positive table: what decoding does to benign traffic
8. Measure why every bound exists - and the blind spot the size limit creates
9. What still slips through, and what each obvious patch costs
10. The judgment: where a decode-and-filter layer belongs, and how a pro deploys it
11. Recap, and one experiment: raise the depth and watch what moves

Open the full interactive lab →