Lion Cipher Lab

Measure the elicitation gap between a plain-prompt eval and a scaffolded run before you trust a capability claim (deterministic STUB model, defensive analysis tasks)

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Every few months a lab report or a system card says something like 'the model solved 12% of our cyber tasks'. Defenders use numbers like that to decide how worried to be. But the people who run those evals - METR, the UK AI Security Institute - say plainly that a bare prompt under-states what a determined user gets once the model is given tools, more steps, retries and a way to check its work. For your threat model the number that matters is the ELICITED capability, not the headline. In this lab you measure that gap yourself, on a stub you can see inside, and you learn to read every future…

What you'll learn

• building a keyed capability eval: 40 defensive analysis tasks (auth-log counting, base64 config decoding, single-byte XOR URLs, process-tree walks, a packed config) with an exact-match checker, so every answer is graded by a machine, not by eye
• running a tool self-test first (40/40 solved by their tools) so a broken task can never masquerade as a model that cannot do the work - the 'spurious failure' METR tells evaluators to fix
• climbing a cumulative ELICITATION LADDER - plain prompt, + tools, + step budget, + retries with a perfect checker, + task hints - and reporting the solve rate at every rung: 3.9% -> 20.6% -> 57.0% -> 86.3% -> 92.4%
• estimating pass@k with the unbiased estimator from the Codex paper (n = 16 attempts per task), and reading the pass@k curve for diminishing returns: 73.1% at k=1, 92.4% at k=8, 92.5% at k=16
• running a BUDGET-MATCHED comparison: the plain prompt given the same 132 model calls as the scaffold (pass@3) reaches only 11.2% against the scaffold's 57.0%, so the gap is not just 'more tries'
• showing why an elicited capability number is a LOWER BOUND: at the retries rung four tasks were never solved, and one of them (hidden success chance 0.15) fell as soon as hints were added
• proving that retries with an IMPERFECT checker inflate the claim: a format-only checker reported 40/40 solved while only 27/40 were really correct (+32.5 points)
• writing a capability claim the way a professional does: the number, the elicitation level it was measured at, the budget, the checker, and an explicit 'floor, not ceiling' label

Steps in this lab

1. The words first: capability eval, elicitation, scaffold, pass@k, checker, lower bound - and the fact that the model here is a stub
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build 40 defensive analysis tasks, each with one answer a machine can check
4. Write the real tools and the two checkers, then prove no task is broken
5. Write the STUB model and read every knob before you trust anything it does
6. Climb the elicitation ladder and print the solve rate at every rung
7. Read the curve: the gap is huge, the returns diminish, and the gap is not just 'more tries'
8. The honest negative: an elicited number is a LOWER bound, and you cannot easily find the ceiling
9. Ablation: retries with an imperfect checker inflate the claim
10. Why this, not that - how a professional reads and writes a capability number
11. Ship it: write one capability claim with its elicitation level attached, and clean up

Open the full interactive lab →