Lion Cipher Lab

Train a BPE tokenizer and measure how it fragments IOCs in your context budget

• AI Security
• Advanced
• ~80 min
• Free · no account required

Why it matters

Every language model reads and writes TOKENS, not characters, and you pay for, wait for and run out of context in tokens. The tokenizer that turns text into tokens was trained on ordinary text, and security text is not ordinary: IP addresses, hashes, Windows paths, base64 and defanged URLs break into many small pieces. In this lab you build the tokenizer yourself and measure it. A tokenizer trained on ordinary prose spent 119 tokens per log line, so only 68 log lines fit in an 8,192-token window; one trained on logs fitted 128 - and then needed almost three times as many tokens for a plain…

What you'll learn

• writing a byte-level byte-pair-encoding (BPE) tokenizer from scratch in about 120 lines of standard-library Python: a pre-tokenisation regex, the merge-training loop and the encoder
• training four tokenizers with the SAME 600-merge budget (ordinary prose, security logs only, half and half, and half and half with IOC-aware pre-tokenisation) so that only the data and the splitting rule differ
• measuring characters per token on held-out text and tokens per indicator for ten IOC types, and turning that into a context budget (68 versus 128 log lines in 8,192 tokens) and a bill
• measuring the trade-off nobody puts on the slide: training on logs took logs from 1.20 to 2.26 characters per token and took ordinary prose from 5.01 down to 1.65
• quantifying split instability: an IOC tokenised alone and the same IOC inside a log line got the same tokens in as few as 25% of cases, and a one-character change rewrote about 1 token, not the whole split
• fixing context instability with an IOC-aware pre-tokeniser (1.00 in every IOC type) and measuring what it cost: prose 3.42 to 3.17 characters per token, and a one-digit change now rewrites 2.41 of an IPv4's 5.3 tokens
• proving the hard floor on hashes: even a tokenizer trained ONLY on SHA-256 hashes with 8,256 entries reached 2.41 characters per token on new hashes, and log2(V)/4 says no vocabulary can do much better

Steps in this lab

1. The words first: tokens, byte-pair encoding, merges, pre-tokenisation, characters per token, IOCs - and what is real here and what is generated
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build the corpora: ordinary prose, security logs and report lines, and 300 held-out indicators of each of ten types
4. Write the tokenizer: a pre-tokenisation regex, the merge-training loop and the encoder (bpe.py)
5. Train four tokenizers on the same 600-merge budget, so only the data and the splitting rule differ
6. Measure: characters per token, tokens per IOC type, and how many log lines fit in a context window
7. Read the budget: a general tokenizer fits half as many log lines, and that is a bill and a blind spot
8. Honest negative 1: training on in-domain logs helped logs and wrecked ordinary prose
9. Measure split instability: the same IOC in a different place, after a one-character change, and after defanging (stability.py)
10. Read the instability: the space in front matters more than the typo, and the folk claim is half wrong
11. The fix, measured: IOC-aware pre-tokenisation - what it bought and what it cost
12. Honest negative 3: hashes cannot be compressed, and one line of maths says why (hash_ceiling.py)
13. Why this, not that: domain tokenizer, IOC-aware pre-split, or keep indicators out of the prompt
14. When NOT to bother with any of this
15. The way a pro does it differently, and what to ship
16. OPTIONAL appendix: the same measurements with a production tokenizer (tiktoken)

Open the full interactive lab →