Lion Cipher Lab
Detect when a security benchmark saturates across model generations, and rebuild its headroom (synthetic benchmark, STUB models)
- AI Security
- Intermediate
- ~60 min
- Free · no account required
Why it matters
New frontier models now arrive every few months, and every release comes with a table of benchmark scores. The trouble is that public benchmarks fill up fast: the 2025 Stanford AI Index reports scores rising by 18.8, 48.9 and 67.3 percentage points in a single year on three benchmarks introduced in 2023 to be hard. Once the top models pile up near the ceiling - and the ceiling is below 100%, because some answer keys are wrong - the gaps between them are smaller than the noise, and a security team choosing a model for alert triage by that table is choosing on noise. This lab makes the whole…
What you'll learn
- simulating a 400-item security benchmark with a known 5% of wrong gold labels, answered by nine synthetic frontier generations of stub models released every four months
- reading a saturation table per generation: top score, room under the ceiling, gap to the previous generation with a paired-bootstrap 95% CI, dead and unsolved item shares, and split-half Kendall tau
- finding a benchmark's useful life - here 5 generations, about 20 months - with a rule that survives one noisy result (two NOs in a row, or less than 0.010 of room)
- recognising that the items no frontier model solves on a saturated benchmark are mostly wrong gold labels (20 of 20 here), so the ceiling is 0.950, not 1.000
- rebuilding headroom by re-grading flagged items, dropping dead items and adding a harder held-out split, then proving it worked: separable at gens 7, 8 and 9 on the new 143-item set, none on the old 400
- naming the costs with numbers: wider intervals on a small split, a regression on the basics the rebuilt set misses, and a projected second saturation 20 months later
Steps in this lab
- The words first: benchmark, gold label, frontier generation, saturation, ceiling, headroom, dead items - and the fact that everything here is synthetic
- Open a terminal, check Python 3 and make a scratch folder (Windows, macOS, Linux)
- Write the synthetic benchmark and the stub generations (sim.py)
- Watch it saturate: one row per generation (saturate.py)
- Find the moment it stopped discriminating - its useful life - and why the leftover items are label errors
- Rebuild the headroom: re-grade, drop dead items, add a hard held-out split (rebuild.py)
- What the rebuild cost: wider intervals, a regression it cannot see, and a second saturation
- When to run this check, why it works, and when not to trust it
- The way a pro does it differently - and an experiment where the naive alarm misfires