Lion Cipher Lab
Shrink a security benchmark with item response theory and measure what the subset gets wrong
- AI Security
- Advanced
- ~90 min
- Free · no account required
Why it matters
A good security benchmark - a thousand CVE descriptions each keyed to the right CWE weakness, say - is expensive to run on every model release, and the tempting shortcut is to run 50 or 100 of the items instead. Item response theory (IRT), the statistics behind adaptive exams, says you can do better than picking those items at random: learn how hard and how sharp each item is from models you have already run, and keep the items that carry the most information. tinyBenchmarks showed this working on public LLM leaderboards with 100 items. This lab rebuilds the idea from nothing, with stub…
What you'll learn
- writing down a 2-parameter logistic (2PL) item response model - difficulty b, discrimination a, ability theta - and generating a 61-model x 1,000-item answer matrix from it with a fixed seed
- fitting 2PL item parameters back from answers alone in about 60 lines of standard-library Python (joint maximum likelihood with Fisher-scoring steps and gentle priors), and checking the fit against the hidden truth
- shrinking a benchmark three ways - random items, random items plus an IRT score estimate, and items picked by Fisher information - and grading each on held-out models by worst score error and Kendall tau-b
- reading a compression result as a distribution, not a single number: the 50-item information subset beat 100% of 200 random draws after a 48-model fit, but only 44% after a 12-model fit
- catching the failure that matters: a new architecture whose strengths the fitted population never varied in was ranked 10th instead of 7th by the information subset at every size, because that subset was 96% one skill
- deciding when a compressed eval is a safe regression screen and when the full benchmark has to be re-run before anyone repeats the number
Steps in this lab
- The words first: benchmark, item, item response theory, the 2PL, Fisher information, Kendall tau - and the fact that every model here is a stub
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Build the 1,000-item benchmark and 61 STUB models from a written-down 2PL
- Fit the 2PL back from the answers alone, on 12 past models
- Write the grader: three ways to shrink, graded on 12 held-out releases and on nova
- Read the 12-model result: below 100 items the clever subset is a coin toss, and small subsets are wide
- Refit on all 48 past models: the information subset now beats every random draw at 50 items
- The honest negative: the new architecture gets misranked, and the better fit made it worse
- Why this, not that - and when NOT to trust a compressed eval
- The way a pro does it differently - and one experiment that proves where nova's error came from