Lion Cipher Lab

Fan out subagents over a security corpus and merge their findings — measure the duplicates, and measure the campaign that no shard can see

• AI Security
• Advanced
• ~100 min
• Free · no account required

Why it matters

Your SIEM produced twelve thousand events last night and you would like an agent to hunt through them. It does not fit. Not 'it is slow' — it does not fit: the corpus is 146,796 tokens against a 60,000-token window, and when you point one agent at it, the agent reads the first 5,000 events, reports confidently on those, and says nothing whatsoever about the other 7,233. It does not error. It does not warn. In this lab it finds 12 of the 36 planted incidents and hands you a clean-looking answer about a night it mostly missed. So you do what everybody does: you cut the corpus into eight shards,…

What you'll learn

• SHARDING a security corpus on purpose instead of by accident: writing down, for every detection rule you run, how many HOSTS and how many SECONDS of evidence it needs to see at once, and choosing the cut axis that is not on that list
• measuring the three things a fan-out actually costs, not just the one it is sold on: wall-clock (167 simulated seconds serial down to 25 across eight workers, 6.6x), the extra tokens (31% more events read once you add a lookback), and the recall you paid with (36/36 down to 29/36)
• building a real MERGE stage — group by a dedupe key, union the evidence — and measuring BOTH of its failure directions on the same 41 findings: duplicates left in the queue, and genuinely distinct incidents collapsed into one alert nobody ever splits back apart
• proving that a merge cannot recover what a shard never emitted: measuring structural invisibility (7 incidents that NO subagent held in full under by-time, 6 under by-host, all 36 under random sharding) as a property of the cut rather than of the model
• reading a shard-strategy table against a rule-requirements table and predicting the recall before you run it — by-time keeps 42 hosts and loses 6 hours, by-host keeps 24 hours and loses 41 hosts, random keeps both spans and destroys the DENSITY that every threshold depends on
• finding the maximum in a K sweep, which is nowhere near either end: 12/36 at K=1 (window overflow), 35/36 at K=4, and 29/36 at K=48 while paying 2.3x more and running slower end to end
• implementing the map-reduce fix properly — subagents emit sub-threshold PARTIAL evidence, the merger re-runs the rules over the union — and measuring the emit floor at which it stops working (F=3 recovers all 36 in 59,024 tokens; F=5 silently loses the campaign while still reporting recall 0.972)
• the judgment to say when a fan-out is the wrong answer: below ~800 events it is slower, below one context window it is also less accurate, and for any rule whose evidence spans wider than a shard on both axes it is not a slow answer but no answer

Steps in this lab

1. The frame: eight readers, one corpus, and the one thing none of them can see
2. Every word you need, defined once, in order
3. Open a terminal, check Python, make a scratch folder
4. Build the corpus and plant the ground truth — including the campaign
5. The four rules, the scorer, and the ORACLE — the ceiling every later run is measured against
6. The analyst agent is a DETERMINISTIC STUB — read this before you paste it
7. Baseline: one agent, one window, and the 59% of the night it never mentions
8. The three ways to cut a corpus — and the table that predicts the whole lab
9. Fan out — 6.6x faster, the whole corpus read, and the first honest negative
10. The merge stage, and the failure direction nobody measures
11. Does the shard strategy change what you can find? Measure it — including the campaign
12. The real fix: make the shards emit partial evidence, and measure what it costs
13. Sweep K, find the maximum, and get the decision rule out of it
14. When NOT to fan out — and why this, not that
15. The way a pro does it

Open the full interactive lab →