Lion Cipher Lab

Add an adversarial verifier subagent to a noisy alert pipeline — a second agent whose only job is to disprove the first — then measure the false positives it removes AND the real intrusions it wrongly kills

• AI Security
• Advanced
• ~105 min
• Free · no account required

Why it matters

Your detection rule fires 263 times on a 900-alert day. Sixty-five of those are real; 198 are the backup agent running an encoded PowerShell command at three in the morning, exactly as it has every night for two years. A precision of 0.247 is not a broken rule — it is an ORDINARY rule, and it is why alert fatigue is the SOC's oldest occupational disease: an analyst who has been wrong about a hundred and ninety-eight cases in a row does not read the hundred and ninety-ninth as carefully as the first. The fashionable fix is a second agent. Not another detector — a VERIFIER, an adversarial…

What you'll learn

• building a TWO-STAGE agent pipeline over a labelled corpus of security alerts — a FINDER that produces findings from evidence, and an adversarial VERIFIER whose single instruction is to disprove each one by returning to the evidence and demanding a specific, checkable fact
• reading a full CONFUSION MATRIX before and after verification on the same 900 alerts, and refusing to report precision without reporting the true positives that paid for it: 65/198/20 before, 54/65/31 after
• naming the honest cost out loud — 133 false positives removed AND 11 real intrusions dismissed at K=2, 187 and 20 at K=3, 198 and 52 at K=5 — and understanding why the row with the best F1 is not automatically the row a SOC should ship
• sweeping a verifier's STRICTNESS and plotting the precision/recall curve it traces, so the knob is chosen from a table instead of from a feeling
• diagnosing CORRELATED FAILURE: a verifier that reads the same evidence as the finder confirms 93.2% of findings, rubber-stamps 150 of the 155 confusable benigns you most needed challenged, and LOWERS F1 from 0.374 to 0.370 while costing 29% more work
• fixing it the only way that works — different EVIDENCE, not a different prompt — and measuring the evidence overlap between your two stages as a number (0.100 here) instead of assuming it is zero
• pricing verification honestly in agent calls, wall clock and analyst hours, including the row nobody prints: true positives per 1,000 agent calls FELL from 72.2 to 46.4, because a verifier can never find anything, only delete
• recognising when a verifier is the WRONG tool — 47.5% of the false positives came from one automation account, and a free finder-side suppression rule removed 94 of them for zero agent calls, at a cost of 13 real intrusions the blanket version also deleted
• the judgment to distinguish an adversarial verifier from the three cheaper things it is confused with: a second opinion, a confidence threshold, and a model checking its own work

Steps in this lab

1. The words first: subagent, verifier, false positive, confusion matrix, precision, recall, correlated failure
2. Open a terminal and make a scratch folder (all three operating systems)
3. Generate the labelled corpus: 900 alerts whose truth you already know
4. Both agents are DETERMINISTIC STUBS — read this before you paste it
5. The scorer: a confusion matrix, and the three ratios that come out of it
6. Stage one alone: 263 findings, 65 of them real, and an analyst with a bad afternoon
7. Stage two: a verifier whose only job is to disprove
8. The bodies: name the eleven real intrusions your verifier dismissed
9. Sweep the strictness knob and plot the precision/recall curve
10. The trap: a verifier that shares the finder's blind spot rubber-stamps everything
11. The fix is different EVIDENCE, not a different prompt — and you can measure how different
12. Price it: cost, latency, and the precision you actually bought per unit of work
13. When a verifier is the WRONG tool: fix the finder instead
14. The way a pro does it differently
15. What you built, what it cost, and the one thing to change first

Open the full interactive lab →