Lion Cipher Lab

Compact a long-running IR agent's context again and again, and measure which critical facts survive

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Agents are being given investigations that run for hours: hundreds of tool calls, far more transcript than any context window holds. Every agent harness answers that the same way - it compacts, folding old turns into a summary - and then does it again, summarising its own summary. Nobody reads the twelfth summary to check what fell out. This lab plants an answer key in 200 long synthetic investigations and grades exactly what survives. Recursive summarising kept 0.938 of the critical facts after one compaction and 0.530 after eight; at handover it had lost three quarters of the constraints,…

What you'll learn

• building a long synthetic incident-response investigation with a planted answer key (hosts, initial-access vector, IOCs, containment actions with status, scope, and a do-NOT-reboot constraint) so compaction loss can be graded exactly
• writing down a summariser's loss model - drop by salience, identifier paraphrase, look-alike merge, a fact cap - and seeing how a per-pass loss compounds when a summary is summarised again
• comparing four compaction strategies on the same 200 investigations: truncate oldest, recursive summarise-the-summary, summary plus a pinned JSON ledger, and ledger plus a searchable raw archive
• grading retention exactly as well as loosely, and catching identifiers that are 'retained' but corrupted (truncated hashes, widened IPs, normalised host names)
• measuring the safety outcome that matters - how often an agent that lost the constraint proposes rebooting a host whose memory capture is pending - next to the token bill of each strategy
• adding the pro's rules (fold-time identifier extraction and a pre-action archive check) one at a time, and seeing one of them make the constraint problem worse before the other fixes it

Steps in this lab

1. The words first: context window, compaction, rolling and recursive summaries, a pinned ledger, an archive, exact versus loose retention - and the fact that the summariser here is a stub
2. Where this sits: what five neighbouring labs already teach, and what only this one measures
3. Open a terminal and make a scratch folder (Windows, macOS, Linux)
4. Build 200 synthetic investigations with a planted answer key of critical facts
5. Write the STUB summariser: its whole loss model, written down, and what eight passes do to one fact
6. Write the harness: four compaction strategies, a pinned JSON ledger, an archive, and a stub agent that may propose the reboot
7. Measure all 200 investigations: retention after 1, 2, 4 and 8 compactions, per fact type, at handover
8. Read the compounding: a summary of a summary drifts further every pass
9. The constraint and the bill: who proposed the reboot, and what each strategy cost
10. The honest negatives: the ledger protects only what was written, 'retained' is not 'correct', and the archive helps only if asked
11. Prove what the numbers rest on: switch every loss off and re-run
12. The way a pro does it: write identifiers down by rule, and check before anything destructive - measured one rule at a time
13. Why this, not that: a ledger versus a bigger window, a better summary prompt, or retrieval alone
14. When NOT to build all of this
15. What to ship, what to log, and an optional real-model swap

Open the full interactive lab →