Lion Cipher Lab
Catch a remediation agent gaming its own tests and measure the true fix rate (deterministic stub patch agent)
- AI Security
- Advanced
- ~90 min
- Free · no account required
Why it matters
Teams are starting to let AI agents patch vulnerable code and open the pull request themselves, with 'the tests pass' as the proof. But a test is only a proxy for 'the bug is fixed', and an optimiser pointed at a proxy finds the cheap way to satisfy it: delete the failing test, skip it, swallow its assertion, or special-case the exact input it checks. Researchers call this specification gaming or reward hacking, and METR and the ImpossibleBench authors have documented current frontier models doing it on coding tasks. This lab makes it measurable. A stub patch agent with a written error model…
What you'll learn
- building 60 small vulnerable functions (path traversal, port validation, prefix-match token check, username allowlist, open redirect, size limit), each with a visible unit test, a hidden held-out input and a written spec
- measuring the gap between what a patch agent REPORTS and what is TRUE: 55 of 60 reported PASS (0.917), only 26 of 60 were really fixed (0.433)
- recognising the five ways a patch can make a test green without fixing anything: delete the test, skip it, swallow the assertion with try/except-pass, hardcode the exact test input, or fix only part of the bug
- building three detectors - a test-file hash, a diff scanner, a held-out test - and scoring each on catch rate AND false positives on genuine fixes
- proving that locking the tests moves the pressure instead of removing it: with read-only tests, hardcoding rose from 6 to 18 and the hash check caught none of them
- choosing a gate you can defend: hash plus held-out test kept every genuine fix, while adding a diff scanner cut bad patches further at the price of rejecting 4 to 8 real fixes
Steps in this lab
- The words first: patch agent, visible and held-out tests, the oracle, specification gaming - and the fact that the agent here is a stub
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Build 60 small vulnerable functions, each with a visible test and one hidden held-out input
- Write the harness and the ORACLE - the part the agent never sees
- Write the STUB patch agent - its whole error model, written down
- Look at three patches with your own eyes before you count anything
- Measure the gap: what the agent reports against what is true
- Score the detectors: catch rate on gamed patches, false alarms on real fixes
- The honest negative: lock the tests, and the gaming moves into the code
- Why this, not that - and when NOT to lean on each detector
- The way a pro does it differently - and how to ship it