Lion Cipher Lab

Measure an AI agent's 50% task-time horizon the METR way on a synthetic cyber task suite, and forecast it honestly for your threat model

• AI Security
• Advanced
• ~100 min
• Free · no account required

Why it matters

Every security team is now asked some version of the same question: when will an AI agent be able to do, on its own, the kind of attack our red team does? The loudest answers are hype or dismissal. The most useful public method is METR's: measure how long tasks take skilled humans, see which tasks an agent completes, and summarise each model by the task length at which it succeeds half the time - its 50% time horizon - then watch how that number grows across model releases. METR's own paper reports that on their software task suite this horizon has been doubling roughly every seven months…

What you'll learn

• fitting a logistic regression of task success against log2(human-expert minutes) by Newton's method in pure standard-library Python, and reading off the task length at which predicted success is 50% - the time horizon - and 80%
• keeping the 80% horizon next to the 50% one: on this suite it is 4.7x to 7.7x shorter for every generation (gen-F: 1.52 h at 50%, 18.1 min at 80%), so the 'reliable' capability lags the headline one by more than a year of trend
• turning six horizons into a doubling time by fitting log2(horizon) against release date (5.0 months here) and bootstrapping the tasks for a 95% confidence interval (4.2 to 6.0 months)
• catching what that bootstrap cannot see: the planted truth is 6 months, one outlier generation pulls the fit, and leaving a single generation out moves the doubling time from 5.0 to 5.9 or 4.2 months - more than the task CI admits
• showing a forecast's uncertainty grow with distance: the 95% band on the 50% horizon is 2.4x wide at the last release and 13.4x wide three years out
• measuring suite-composition bias: dropping the 45 messy, low-context tasks inflates every horizon 1.2x to 2.1x and pulls the forecast 4 months earlier
• translating a trend into a threat-model planning input - 'when might an autonomous agent plausibly complete a task class that takes our red team X hours' - with both the 50% and 80% dates and an explicit CI, tied to specific controls rather than headlines

Steps in this lab

1. The words first: time horizon, 50% and 80%, doubling time, bootstrap, messy tasks - and the fact that every number here is SYNTHETIC
2. Open a terminal and make a scratch folder (all three operating systems)
3. Generate the SYNTHETIC suite: 150 cyber tasks from 1 minute to 16 hours, six fictional generations
4. The data is SYNTHETIC and the agent is a STUB - here is every planted truth, in writing
5. Fit success against log2(human minutes) per generation, and read off the 50% and 80% horizons
6. Fit the doubling time, bootstrap it, and forecast when the trend reaches your threat model
7. The honest negatives, each with a number: the 80% lag, messy-task inflation, and the one generation that moves everything
8. Map it onto YOUR threat model: edit the task classes, then tie each date to a control
9. Judgment: a planning input, not a prophecy - when to use it, when not to, and how a pro does it differently
10. Ship it: re-run the evidence, write the one-page forecast, and clean up

Open the full interactive lab →