Lion Cipher Lab

Make an incident-response agent write its plan down first, then diff the plan against what actually ran — and measure the drift you can and cannot see

AI Security
Advanced
~95 min
Free · no account required

Why it matters

There is a moment coming for every security team that lets software act on its own: an agent isolates a host at 3am, and in the morning someone has to answer 'what did it do, and was that what it said it would do?' If the agent is a ReAct loop — think, act, observe, think again — there is no answer, because there was never a plan to compare anything to. The intent was invented one step at a time and the only record is a transcript of the thinking that produced it, which is not evidence of anything; a transcript can rationalise whatever happened. PLAN-THEN-EXECUTE fixes that by construction.…

What you'll learn

designing a TYPED PLAN SCHEMA for an autonomous responder — every step carrying its tool, its arguments, its preconditions, its expected effects and its declared blast radius — so that the agent's intent becomes a machine-readable artifact instead of a paragraph in a log
writing a PLAN VALIDATOR that runs before a single tool is called and can refuse: allow-list checks, tool-contract checks, blast-radius-versus-authorisation checks, precondition satisfiability, and a written evidence-before-destruction ordering policy
building an EXECUTOR that records a real trace — every call, its arguments as they were at call time, and the world-state change the call actually produced — and understanding why the tool's own success message is the one thing you must not believe
implementing a DIFF ALGORITHM over plan-versus-trace that classifies drift by kind: extra step, skipped step, reordered step, same tool with different arguments, an argument that changed between plan time and call time (TOCTOU), an observed effect that does not match the declared one, and a blast radius that grew
writing an EQUIVALENCE POLICY as levelled, testable code — which differences between a plan and a run are permitted — and measuring the false-positive rate it buys you, because benign nondeterminism (a retry, a late-bound ticket id, a legitimately reordered independent step) will fire a naive detector on every single run
scoring a detector honestly against a labelled corpus: drift rate, per-class detection rate, recall, precision and false-alarm count, with a naive exact-match baseline scored on exactly the same runs
attacking your own control: crafting malicious drift that hides inside an equivalence class you yourself declared benign, and pricing the tightening that would catch it in false alarms per 240 runs
the judgment to name what this control cannot see — a plan that was wrong, executed perfectly — and to say out loud that a plan-versus-run diff proves FAITHFULNESS, not CORRECTNESS, and needs a second control beside it

Steps in this lab

The frame: a plan you can read, a run you can prove, and the gap between them
Open a terminal, make a folder, and build the incident corpus
The response tools are SIMULATIONS over a world dictionary — nothing here touches your computer
The planner is a DETERMINISTIC STUB, and here is the plan artifact it emits
The validator: refuse the plan before anything runs — and watch it reject your own runbook
The executor: run the plans, record a real trace, and inject drift you can score against
Write the diff — and start with the naive one, so you feel why it is unshippable
The semantic diff: an alignment, seven drift classes, and a written equivalence policy
Stop reading rates and read the runs: every catch, every miss, every false alarm by name
Price the equivalence policy: tight means noisy, loose means blind, and there is no third option
Attack your own detector: two malicious runs that live inside your equivalence classes
The drift class you cannot see: a wrong plan, executed perfectly
When NOT to use plan-then-execute — and why this, not that
The way a pro does it
OPTIONAL appendix: put a real local model behind the planner

Open the full interactive lab →