Lion Cipher Lab
Tell a real eval win from noise: bootstrap a confidence interval and a paired McNemar test on a 50-item security eval, and find you can't ship the difference
- AI Security
- Intermediate
- ~90 min
- Free · no account required
Why it matters
Two models score 71% and 76% on your 50-item security eval and someone wants to ship the second one. The 5-point gap looks decisive and it is nothing. This lab builds the tools that show why, from Python's standard library and a planted answer key, so every claim is a number you generated rather than a chart you were handed. First the point scores: model A gets 36 of 50 and model B gets 38 of 50, a 4-point measured gap. Then a bootstrap - resample the 50 items with replacement 20000 times and read the percentiles - which puts each model's accuracy in an interval 24 points wide and puts the…
What you'll learn
- reading an eval leaderboard the way it should be read - never a bare point score, always a point score with an interval and a paired test attached, because a two-model gap of 4 points on 50 items (0.7200 vs 0.7600 here) is a number you cannot ship
- writing a percentile BOOTSTRAP from nothing but random.Random: resample the 50 items with replacement 20000 times, seeded, and read the 2.5th and 97.5th percentiles - the accuracy interval comes out 24 points wide ([+0.6000, +0.8400] for A) and the DIFFERENCE interval is [-0.1000, +0.1800], straddling zero, with 34% of resamples showing B not ahead at all
- building McNemar's paired test by hand from the 2x2 discordant counts (b = 8, c = 6) and computing the exact two-sided binomial p-value (0.7905) instead of trusting a library, plus the continuity-corrected chi-square (0.0714)
- seeing WHY the paired test is the correct one and the unpaired two-proportion z-test (p = 0.6484 here) is not - the unpaired test pretends 100 independent observations where there were 50 paired ones, mis-states its variance, and cannot see that the same items were scored twice
- computing a required sample size and a minimum detectable effect from the normal approximation with statistics.NormalDist - detecting a true 5-point gap at 80% power needs ~1220 items per model (50 is 24x too few), and 50 items can only detect a gap of about 25 points
- measuring the power of your test by Monte Carlo rather than asserting it: simulate the 50-item eval 20000 times under a true 5-point gap and the paired McNemar test calls it significant 5.41% of the time - indistinguishable from the false-alarm rate, i.e. blind
- carrying one rule out of the lab: a leaderboard delta whose confidence interval includes zero is not a result, and a model swap justified by such a delta is a coin flip dressed as a decision
Steps in this lab
- The words first: eval, accuracy, sampling noise, confidence interval, bootstrap, paired test, McNemar, power - and the fact that the two models here are planted vectors
- Open a terminal and make a scratch folder (all three operating systems)
- Build the 50-item eval and score two models on it (point accuracy first)
- Bootstrap a 95% confidence interval for each model - and for the difference
- The paired test done by hand: McNemar from the 2x2 table, and why the unpaired test is wrong
- The calculation that should have come first: power, minimum detectable effect, required n
- How to read an eval delta, and when a difference is actually worth a model swap