Lion Cipher Lab

Catch benchmark contamination inflating your security eval: prove the model only aced it because the answers leaked, then detect the leak and report the corrected score

• AI Security
• Intermediate
• ~90 min
• Free · no account required

Why it matters

A number on a leaderboard is the single most persuasive object in machine learning, and it is also the easiest one to fake without meaning to. The mechanism is benchmark contamination: the test items of an eval, or close paraphrases of them, end up in a model's training data - through a scraped web page, a leaked answer key, a teammate who fine-tuned on the eval to 'see if it helps' - and from then on a high score measures memory rather than ability. In security this is not academic. Teams pick detection models, triage assistants and vuln-scoring copilots off benchmark tables, and a model…

What you'll learn

• building a small security eval with a PLANTED answer key (60 items: 30 ATT&CK-technique-id questions and 30 CVE-severity questions) and two deterministic model stubs - a CONTAMINATED one that memorised the exact test items and a CLEAN one that is genuinely, imperfectly capable - so every claim below is a measured number and not an anecdote
• seeing contamination inflate a leaderboard in one run: the contaminated model scores 0.9667 on the original eval while the more capable clean model scores 0.7333, so the naive ranking would ship the weaker model because it 'won' by 23.3 points
• running the OVERLAP / CANARY detector and knowing its limits: 58 of 60 test questions found verbatim in the training corpus, mean 13-gram overlap 0.9778, a planted canary the contaminated model regurgitates and the clean one never emits - and the two honest floors (items too short for an n-gram, and shared boilerplate keeping overlap above zero) that mean overlap can miss and can never fully clear a model
• running the PERTURBATION GAP detector, which needs no access to training data at all: reword each item and shuffle its options, and the memorised model collapses 0.9667 -> 0.4500 (gap 0.5167, with 0 of 60 perturbed items still matching its memory) while the capable model holds 0.7333 -> 0.6833 (gap 0.0500)
• reading the CORRECTED score off the perturbed run and watching the leaderboard flip: clean 0.6833 now beats contaminated 0.4500, which is the honest ranking the inflated one hid
• telling a signal from noise with a threshold: the clean model's 0.0500 gap is re-rolled-guess noise, the contaminated model's 0.5167 gap is contamination, and a 0.10 flag sits between them - so you can state a rule instead of eyeballing it
• carrying out a rule for vetting any benchmark result before you trust it, including why a 99% on a hard eval where strong clean models get ~70% is a red flag and not a trophy, and the one case (paraphrase-augmented leakage) where the perturbation gap can be fooled

Steps in this lab

1. The words first: eval, contamination, leakage, canary, overlap, perturbation gap, corrected accuracy - and the fact that both models here are stubs
2. Open a terminal and make a scratch folder (all three operating systems)
3. Build the eval: 60 security questions whose correct answers you already know, plus a canary
4. The two models are STUBS, and here is exactly how each one behaves
5. Score both models on the original eval and watch the leaderboard lie
6. Detector 1: exact-match and n-gram overlap, plus the canary - and where this detector goes blind
7. Detector 2: the perturbation gap - the cheap tell that needs no training-data access
8. The corrected scorecard and a rule for vetting any benchmark result
9. Judgment: when a high score is real, when it is memorised, and where the perturbation gap can be fooled
10. Ship it: what to run, what to report, and the last honest word about the stub

Open the full interactive lab →