Lion Cipher Lab

Implement speculative decoding with two tiny language models, and prove it changes nothing but speed

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Large language models write one token at a time, and every token needs a full pass through the big model. That makes generation slow and expensive, and it is why every serious inference stack (the software that runs models for you) now offers speculative decoding: a small, cheap model guesses a few tokens ahead and the big model checks all the guesses in a single pass. The promise is unusual. It is not 'faster but a bit worse'. The maths says the text comes out with exactly the same probabilities as if the big model had written it alone. If you work on AI systems, including AI tools in a SOC,…

What you'll learn

• training two character n-gram language models from scratch on a seeded synthetic corpus of SOC log lines, a slow order-7 target and cheap order-3 and unigram drafts, with Witten-Bell interpolation so no character ever has probability zero
• implementing speculative sampling exactly as Leviathan et al. and Chen et al. describe it: draft k tokens, verify them in one target call, accept with probability min(1, p/q), resample from the normalised residual max(0, p - q), and take a free bonus token when all k are accepted
• proving exactness against the EXACT target distribution, not by eye: speculative sampling scored a total-variation distance of 0.0092 (chi-square p = 0.62) against a plain-sampling noise floor of 0.0067, while the draft model on its own scored 0.7998
• showing greedy speculative decoding is byte-identical to greedy target decoding (same sha256 over 200 prompts x 120 characters, 0 differences) while cutting target calls from 24,000 to 9,614
• measuring acceptance rate, tokens per target call and modelled speedup for k = 1..8, and checking them against the paper's formula (good draft: 1.61x measured and 1.61x predicted at k = 3)
• predicting when speculative decoding loses before you build it: a draft whose overlap a (0.07) is below its relative cost c (0.1) ran at 0.97x, and 0.60x at k = 8
• catching a plausible shortcut that silently changes the output: 'keep the draft token if it equals the target's top choice' gave a TV distance of 0.0954, ten times the floor
• choosing k from measurements: when verification is not free (+8 ms per scored position) the best k fell from 3 to 2, and k = 6 and above was slower than not speculating at all

Steps in this lab

1. The words first: target, draft, decode step, speculative decoding - and the fact that both models here are tiny
2. Set up: a terminal, Python, numpy and a lab folder
3. Make the synthetic corpus: SOC log lines, runbook sentences, and some recipes
4. The two models and the meter: a slow, better target and three cheap drafts
5. Read the algorithm: draft, verify once, accept with min(1, p/q), resample from the residual
6. Prove it changes nothing: compare against the EXACT target distribution
7. Greedy: byte-identical output, fewer target calls
8. Measure the speedup for k = 1..8, and check it against the paper's formula
9. Honest negative 1: a bad draft makes it slower than not speculating, and you could have predicted it
10. Honest negative 2: the 'accept if it matches the target's top choice' shortcut silently changes the output
11. Honest negative 3: large k wastes work, and the best k moves when verifying is not free
12. When to use it, when not to, and how a pro switches it on

Open the full interactive lab →