Lion Cipher Lab

Make an autonomous SOC agent's tool calls idempotent with a write-ahead run journal — then crash it 70 times and prove it never opened the same ticket twice

• AI Security
• Advanced
• ~110 min
• Free · no account required

Why it matters

The first time an agent costs a company real money it is almost never because the model said something stupid. It is because something retried. A tool call timed out at the gateway and the SDK retried it; a container was rescheduled mid-run and the queue redelivered the job; a supervisor saw a non-zero exit code and started the run again from the top. The model behaved perfectly all three times. And now there are three incident tickets for one alert, the on-call has been paged three times at 3am, and a service account has been disabled twice — the second time after the first disable had…

What you'll learn

• telling a SIDE EFFECT apart from a read, and recognising the exact class of agent tool call — open a ticket, block an IP, disable an account, page a human — where a retry is not a free do-over but a second real-world action
• deriving an IDEMPOTENCY KEY from the SEMANTICS of a call (run id, plan position, tool, canonical arguments) and being able to say, with measurements, why a fresh uuid4 per attempt and a key with time.time() in it are the same bug wearing two costumes
• recognising the opposite and more dangerous key failure — a key so coarse that two legitimately different calls collide, so the second real firewall block is silently swallowed and never happens — and measuring how many actions it deletes
• building a WRITE-AHEAD JOURNAL from primitives: single-call atomic appends through an O_APPEND file descriptor, os.fsync for durability, a per-record SHA-256 checksum that makes a torn write detectable, and a write-temp/fsync/os.replace checkpoint for the state snapshot
• implementing the three-phase INTENT / EXECUTE / RESULT pattern and reasoning correctly about the four distinct crash windows it creates — including the one window where the truth is genuinely unknowable from your own records
• writing a RECONCILIATION pass for that unknowable window, and knowing the difference between the cheap version (re-check only unresolved intents) and the expensive full sweep you need when the journal itself was damaged
• injecting crashes deliberately — a real child-process kill, not a mocked exception — at before-intent, after-intent, after-call and mid-write-of-the-result, and reading the recovery behaviour of each
• measuring duplicate side effects the honest way: counting rows in the simulated world with the journal off, with the journal on, and with reconciliation on, over the same fixed crash schedule
• stating the residual risk in numbers: why AT-LEAST-ONCE is the best you can buy against a remote tool with no idempotency support, how wide that window actually is on your workload, and what the journal buys you instead of a distributed-systems miracle
• pricing the guarantee: the exact number of durable writes and bytes the journal adds per tool call, and what fsync actually costs on your disk

Steps in this lab

1. The frame: a retry is not a free do-over once the call had a SIDE EFFECT
2. Open a terminal and make a scratch folder (start here even if you never have)
3. Build the world: five SOC tools that are deliberately NOT idempotent
4. The incident corpus and the agent — which is a DETERMINISTIC STUB, said plainly
5. See the damage first: one crash, one restart, two tickets
6. The idempotency key: name the ACTION, not the ATTEMPT
7. The write-ahead journal: atomic appends, a checksum per record, and os.replace
8. The three-phase runner: INTENT, then EXECUTE, then RESULT
9. Crash at every point in the sequence and read the recovery
10. Measure it: 70 injected crashes, 40 incidents, three ways
11. The badly-derived key, measured in both directions
12. The honest limit: AT-LEAST-ONCE is what you can buy, EXACTLY-ONCE is not
13. When the journal itself is the thing that breaks
14. What the guarantee costs on the happy path
15. When you do NOT need this — and why a journal, not the two alternatives
16. The way a pro does it

Open the full interactive lab →