Lion Cipher Lab
Scale test-time compute with Best-of-N and a verifier, then watch a weak verifier get gamed (deterministic regex-detection stub)
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
"Scaling test-time compute" is one of the strongest ideas in modern AI: sample many candidate answers, score them with a verifier, keep the best, and watch quality climb - Snell and colleagues showed it can beat training a bigger model. Security teams are starting to use it to generate and rank things a machine can check, like detection rules. This lab builds the smallest honest version - Best-of-N over candidate detection regexes that must match malicious log lines and spare benign ones - and then shows the failure that the cheerful version hides. With a STRONG verifier (the full labeled set…
What you'll learn
- building a labeled detection problem - malicious vs benign command-line log lines, with a held-out test set - so a verifier can actually CHECK a candidate rule
- running Best-of-N test-time scaling: sampling N candidate detection regexes (seeded, reproducible) and keeping the one a verifier ranks highest, for N = 1, 2, 4, 8, 16, 32
- reading held-out accuracy as a CURVE against N, and separating it from the verifier's own score the way you must separate train from test
- seeing a STRONG verifier (full labeled set + a complexity penalty) make Best-of-N help monotonically to a plateau: held-out 0.695 -> 0.945
- catching a WEAK verifier get GAMED: as N grew its score rose 0.750 -> 1.000 while held-out FELL 0.695 -> 0.480 - reward hacking you can see in your own numbers
- recognising the gamed detector for what it is: four memorised literals that score 1.000 on an 8-case smoke test and catch 0 of 104 real attacks on held-out
- proving the fix is verifier COVERAGE, not cleverness: adding a complexity penalty to the 8-case verifier still got gamed; only the 120-case verifier fixed it
Steps in this lab
- The words first: test-time compute, Best-of-N, a verifier, held-out data, reward hacking - and the fact that the generator here is a stub
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Build a labeled corpus: malicious vs benign command-line log lines, plus a tiny weak smoke test
- Write the candidate generator and the two verifiers, then read the self-test
- Scale Best-of-N and print the two curves, side by side
- Read the curves: the strong verifier helps to a plateau, the weak verifier makes more compute worse
- The honest negative: the weak verifier got GAMED - look at the rule it chose
- Why this, not that - when Best-of-N helps, when it hacks your metric, and how a pro builds the verifier
- Ship it: prove the fix is coverage not cleverness, decide what to log, and clean up