Lion Cipher Lab

Engineer the context you hand a subagent: five assembly policies over a pile of host-compromise evidence, and the measured point where adding more evidence makes the verdict worse

• AI Security
• Advanced
• ~115 min
• Free · no account required

Why it matters

A subagent gets a fresh, empty context window. That is the whole point of it, and it is also the moment the interesting question starts, because an empty window is a blank cheque and somebody has to decide what to spend it on. Watch what people actually do: they pass everything. The host's alerts, its process tree, a fortnight of netflow, the asset record, every prior ticket — dump it in, the window is big, let the model sort it out. On the 240-host compromise-assessment corpus you are about to generate, passing everything scores an accuracy of 0.692. Calling every single host benign without…

What you'll learn

• building a CONTEXT-ASSEMBLY STAGE as a real component with its own interface — evidence in, a budgeted list of slots out — instead of the string concatenation that sits in front of most subagents by accident
• measuring TASK SUCCESS AGAINST CONTEXT SIZE and finding the INVERTED-U with data: on this corpus the naive chronological dump peaks at a 500-token budget (accuracy 0.771) and is down to 0.692 by 1,400, and the marginal-return table shows exactly where the next hundred tokens turned negative
• implementing and grading five assembly policies against each other on identical decisions — everything, recency-only, relevance-ranked, summarise-the-tail and structured-facts-only — with common random numbers, so a difference between two policies is the policy and not luck
• keeping an honest negative: summarise-the-tail is the relevance policy plus one step that feels free, and it scored WORSE than the dumb chronological baseline (0.750 against 0.771) while producing 79 false positives at a 180-token budget against relevance's 21
• demonstrating POSITIONAL FAILURE deliberately — the same evidence, the same token count, one decisive alert moved through nine positions — and reading the U-shaped recall curve that appears at 900 tokens and is completely absent at 300
• pricing the defence rather than assuming it: edge-anchored ordering was worth nothing at all below a 350-token context and +0.067 accuracy at 500
• counting TOKENS PER DECISION with an honest proxy and computing ACCURACY PER 1,000 TOKENS, which is the column that changes what you would actually deploy: structured facts scored 0.950 on 20.3 tokens a host against the full dump's 0.692 on 1,158
• learning WHAT TO DROP FIRST from an ablation rather than from taste — dropping every evidence item scoring under 4 on the relevance lexicon raised accuracy from 0.692 to 0.900 while cutting the context 18x
• finding the flaw in your own ranker: all 118 exculpatory evidence items in the corpus score exactly ZERO on a keyword lexicon built from scary words, so relevance ranking silently deletes the evidence that would have cleared the host — and measuring the fix (accuracy 0.871 to 0.917, false positives 21 to 10)
• stating the ceiling of a structured-facts policy out loud: its accuracy is a statement about your extraction rules, not about the evidence, proven by deleting one rule at a time and watching the number fall

Steps in this lab

1. The words first: context window, token, context engineering, budget, recency, relevance, summarisation, lost-in-the-middle, precision, recall, inverted-U
2. Open a terminal and make a scratch folder (all three operating systems)
3. Generate 240 compromise-assessment cases whose answers you already know
4. Build the assembly stage — and look at what each policy actually hands over
5. The reader is a DETERMINISTIC STUB — read this before you paste it
6. The scorer: accuracy, precision and recall on the compromised class
7. The baseline: pass everything, and watch it barely beat doing nothing
8. The sweep: five policies, ten budgets, 12,000 decisions — and the inverted-U
9. Accuracy per token, and the rule you leave with
10. The honest negative: summarising what you dropped is worse than dropping it
11. Lost in the middle, on purpose — then price the cheapest defence there is
12. Structured facts: the cheapest policy in the lab, and the ceiling it comes with
13. What to drop first, and why — an ablation instead of a taste
14. When NOT to do this, and why this rather than that
15. The way a pro does it
16. OPTIONAL appendix: swap the stub for a real local model

Open the full interactive lab →