Lion Cipher Lab
Rank security models from pairwise battles with Bradley-Terry, and put a confidence interval on every rank
- AI Security
- Advanced
- ~100 min
- Free · no account required
Why it matters
Every week a new model 'tops the leaderboard', and somebody on your team wants to swap your phishing-triage or alert-triage model for it. Most of those leaderboards are built the way Chatbot Arena is built: show two anonymous models the same prompt, let a person (or a judge model) pick the better answer or call a tie, collect thousands of these pairwise battles, and turn them into a ranking. The turning is where it goes wrong, and this lab does it four ways against an answer key so you can see exactly how. Six deterministic stub models, whose hidden true strengths are written in the script,…
What you'll learn
- simulating a Chatbot-Arena-style preference arena from pure standard-library Python - six stub models, 2,000 seeded battles, a simulated analyst with a written rule, a 12% tie rate and a small slot-A position bias - so every leaderboard method can be graded against a hidden answer key
- showing that RAW WIN RATE grades the schedule, not the model: on an uneven opponent mix it got 4 of 6 rank positions wrong, crowning delta (true #2) and lifting echo (true #4, average opponent -0.16) above alpha (true #3, average opponent +0.78)
- implementing online ELO and measuring its order-dependence honestly: the same 2,000 battles replayed in 22 orders gave 11 distinct rankings at K=32 and still 3 at K=4, with four different models taking #1
- fitting BRADLEY-TERRY by maximum likelihood with the MM (Zermelo / Hunter) iteration in about twenty lines, proving it order-free (0.0 change when the battles are reversed) and recovering the true order with every score within 27 points of the hidden truth
- knowing what ties do to a Bradley-Terry fit: counting a tie as half a win shrank the score spread from 295 to 259 points without changing the order, and dropping ties restored it to 310
- bootstrapping battles to put a 95% confidence interval on every score AND every rank - bravo and delta both 1..2, P(#1) 0.882 vs 0.118, gap +19 with CI -11..+51 - and saying out loud that '#1 on the leaderboard' is not a claim this data supports
- estimating how many battles it takes to separate two close models by re-running the whole arena: a 12-point true gap was claimable in 3/30 arenas at 2,000 battles and 30/30 only at 64,000
- deciding when a leaderboard rank is meaningful for YOUR task: rank intervals over point ranks, tied tiers over ordinal positions, balanced slot assignment, and your own in-domain battles over any public aggregate
Steps in this lab
- The words first: pairwise battle, arena, win rate, Elo, Bradley-Terry, bootstrap, rank interval - and the fact that the models here are stubs
- Open a terminal and make a scratch folder (all three operating systems)
- Build the arena: six stub models, 2,000 seeded battles, and the answer key in plain sight
- The models are STUBS - here is their entire behaviour, and what that does and does not let you conclude
- Rank by raw win rate - and watch the opponent mix lie
- Online Elo: same battles, different order, different leaderboard
- Fit Bradley-Terry by maximum likelihood with the MM iteration - order-free, and it recovers the answer key
- Bootstrap the battles: a confidence interval on every score and every rank
- How many battles would it take to separate the top two?
- The judgment: when a leaderboard rank means something, and when it does not
- The way a pro does it differently - and what to ship