Lion Cipher Lab

Run a capability-uplift trial on a cyber task and watch the choice of control arm change the answer (synthetic participants, STUB assistant)

• AI Security
• Advanced
• ~2 hours
• Free · no account required

Why it matters

When a frontier AI company says a model does or does not cross a dangerous-capability threshold, part of that claim usually rests on an UPLIFT study: put people on a task with the model and without it, and see whether the model makes them meaningfully better. For a defender, that is the threat-model question that matters most about cyber - not 'can the model pass a test' but 'how much better does it make a novice, and how much better does it make an expert'. Those studies have one design choice that quietly decides their answer: what the people WITHOUT the model get to use. Compare against…

What you'll learn

• reading a frontier lab's uplift claim the way its authors have to defend it: uplift is always uplift OVER SOMETHING, and the something decides the verdict - here the same assistant measures +0.1635 over a no-tools control and only +0.0615 over a search-engine control
• writing a pre-registration (threshold 0.10, primary contrast, pooling rule, three-way decision rule) to a file BEFORE any data exists, and fingerprinting it with SHA-256 so nobody, including you, can quietly move the line after seeing the results
• building a randomized, stratified uplift trial from Python's standard library: 16 checker-graded defensive log-analysis tasks, synthetic novice, intermediate and expert participants, block randomization into no-tools, search and assistant arms, and a STUB assistant whose entire behaviour is four written-down rules
• computing per-stratum and pooled uplift with a participant-level bootstrap interval and applying a pre-registered CROSSES / BELOW / INCONCLUSIVE rule mechanically, without rounding an interval up
• spotting a ceiling effect (experts already at 0.8531 with no tools, so success uplift over search is +0.0250 and its interval includes zero) and catching the uplift it hides in time-to-solve (expert minutes per task 8.1 with search, 5.8 with the assistant)
• seeing why a 27-person pilot cannot decide anything: its point estimate over search is +0.1319, above the threshold, and the full 180-person trial lands BELOW it
• knowing when an uplift trial is the right instrument for a defender's threat model, when a benchmark score is enough, and what a professional evaluation team does differently from this simulation

Steps in this lab

1. The words first: frontier lab, capability threshold, uplift, arms, control, strata, ceiling, pre-registration - and what is fake here
2. Open a terminal, check Python 3 and make a scratch folder (Windows, macOS, Linux)
3. Pre-register BEFORE any data exists: write the threshold, the control, the rule, and fingerprint the file
4. Build the trial: 16 checker-graded defensive tasks, three expertise strata, three arms and a four-rule STUB assistant
5. Analyse exactly as pre-registered: success and time by stratum and arm, then uplift with intervals
6. Same data, two verdicts: the control arm decides whether the assistant crosses the line
7. Stratify: novices gain the most, and their interval straddles the threshold the pooled number hides
8. The ceiling hides expert uplift, and time-to-solve tells a different story from success rate
9. Recap the interval, then sweep sample size: a 27-person pilot points the wrong way
10. When to run an uplift trial, when not to, and what a professional evaluation team does differently
11. Prove the plan was not edited, then clean up

Open the full interactive lab →