Lion Cipher Lab
Catch silent agent regressions: pin an AI agent's tool-call trajectory across model swaps
- AI Security
- Advanced
- ~75 min
- Free · no account required
Why it matters
The moment you give an LLM tools — read a file, run a query, delete a record, send a message — you have an agent, and its risk is no longer 'is the wording good?' but 'what did it actually DO to get there?'. Swap the model (a new Ollama release, a bump from an 8B to a 7B, a provider migration) and the final answer can stay perfectly plausible while the agent quietly changes its behaviour: it calls a destructive tool it used to steer around, skips the verification step it used to run first, loops three extra times, or stops pausing for approval. A final-answer eval — even a good golden-set one…
What you'll learn
- recording an agent's trajectory (ordered tool calls + arg shapes + step count)
- freezing a versioned golden trajectory in source control
- swapping the underlying Ollama model and replaying the same frozen tasks
- writing pytest assertions on behaviour: required tools, forbidden tools, step budget, order
- distinguishing load-bearing safety invariants from cosmetic trajectory details
- failing CI (non-zero exit) on a behavioural regression, not just a text-quality drop
- judging when trajectory testing matters (side-effecting/dangerous tools, model bumps, provider migrations)
Steps in this lab
- The blind spot: a good answer can hide bad behaviour
- Set up a free, local, two-model bench
- Build a tiny security agent with read-only tools — and one dangerous one
- Freeze the tasks and record a GOLDEN trajectory from the trusted model
- Decide which trajectory properties are invariants — and which are cosmetic
- Write the pytest harness that asserts on BEHAVIOUR
- Swap the model, replay, and watch a behavioural regression fail the gate
- Gate CI: run the trajectory test on every model bump