Lion Cipher Lab
Measure a stub triage agent's reliability with pass^k, not pass@k, and see the number a demo hides
- AI Security
- Advanced
- ~90 min
- Free · no account required
Why it matters
Every agent demo is a pass@k number in disguise. Somebody ran the agent, maybe a few times, and showed you the run that worked. That is a fair measure of what the agent CAN do. It says nothing about what it WILL do on the next alert, and for a SOC agent that isolates hosts and disables accounts on its own, the next alert is the only one that matters. This lab puts both numbers on one table. Two stub containment agents, A and B, run 16 times each on 200 tasks with a keyed correct action. Their pass@1 - the chance one attempt is right - is nearly a tie, 0.4547 against 0.4834. Their pass@8 - the…
What you'll learn
- implementing the unbiased pass@k estimator, 1 - C(n-c,k)/C(n,k), and the pass^k estimator, C(c,k)/C(n,k), in a few lines of standard-library Python with math.comb, and checking them against hand-worked cases (c=15 of n=16 gives pass^8 exactly 0.5000, not the plug-in's 0.5967)
- running two stub SOC containment agents 16 times on each of 200 answer-keyed tasks and reading the three numbers that disagree: pass@1 nearly tied (0.4547 vs 0.4834), pass@8 favouring A (0.8523 vs 0.6394), pass^8 favouring B (0.0697 vs 0.2874)
- reporting the per-task breakdown - right on every attempt, on some, on none (A: 4/182/14, B: 38/103/59) - because the average hides whether an agent is reliably right, reliably wrong, or a coin flip
- measuring the temperature tradeoff instead of assuming it: raising the stub's temperature from 0.7 to 2.5 lifted agent B's pass@8 from 0.5780 to 0.8752 and cut its pass^8 from 0.4137 to 0.0687 while pass@1 barely moved
- quantifying estimator noise over 100 full reruns: naive run-k-once sd 0.0198, unbiased from n=16 sd 0.0151, n=64 sd 0.0069, a plug-in (c/n)^k that is biased +0.0238, and the 100x error you get extrapolating pass^8 from pass@1 alone (0.0033 vs a true 0.3202)
- choosing the metric from the deployment: pass@k when a verifier or a human picks the best of k candidates, pass^k when the agent acts on its own and every run has to be right, and both, with the per-task table, in any report
Steps in this lab
- The words first: attempt, answer key, pass@1, pass@k, pass^k, n, c, k, unbiased estimator - and the fact that the agents here are stubs
- Open a terminal and make a scratch folder (all three operating systems)
- The tasks: 200 containment decisions with the correct action already known
- The agent is a STUB: one probability, one temperature knob, one hashed coin flip per attempt
- Write the two estimators with math.comb, and test them against cases you can work by hand
- Run both agents 16 times per task and print the two curves side by side
- The number a demo hides: what pass@8 = 0.85 means when the agent isolates hosts on its own
- Honest negative 1: turning up temperature buys pass@k by spending pass^k
- Honest negative 2: pass^k at high k is noisy, the plug-in is biased, and pass@1 cannot predict either
- Why this, not that: when pass@k is the right number, when pass^k is, and when NOT to trust either
- The way a pro does it differently: report both curves, the per-task table and the n