Lion Cipher Lab
Measure per-rule alert precision and retire your noisiest detections
- SOC Analyst / Blue Team
- Intermediate
- ~60 min
- Free · no account required
Why it matters
Every SOC has a rule that fires forty times a day and has not caught anything real since it was written. Nobody retires it, because retiring a detection feels like lowering your guard -- so it stays, and every one of those forty alerts costs an analyst a few minutes of attention they will never get back. Multiply that across a few dozen rules and you get alert fatigue: the queue becomes mostly noise, analysts start closing tickets on reflex, and the one real intrusion gets closed as a false positive along with the rest. The fix is not heroics, it is bookkeeping. Your ticketing system already…
What you'll learn
- turning a SOC queue's closed-ticket export into a per-rule scorecard: alert volume, precision (true positives / alerts), analyst-hours burned, and true positives caught
- telling apart the three dispositions an analyst closes a ticket with -- true positive, benign true positive (the rule fired correctly on real but authorized activity), and false positive -- and why only the first counts toward precision
- putting a 95% Wilson score interval on every precision number, and refusing to judge a rule whose interval is too wide to mean anything (a DCSync rule with 2 alerts sits at 9.5%-90.5%)
- building a Pareto view of analyst time and reading it: in the seeded export 6 of 30 rules burn 46% of all hours while catching only 12% of the true positives
- making a defensible keep / tune / retire call per rule -- tune when one entity (a service account, a VPN egress, a backup server) causes most of the noise, retire only when precision is confidently below the bar -- and measuring the BEFORE vs AFTER: tier-1 hours 1577 -> 612 with 235 of 258 true positives kept
- pricing every retirement in MITRE ATT&CK coverage (29 -> 25 techniques) and refusing to retire the only rule watching a rare, high-impact technique (Kerberoasting, T1558.003) -- route it to a slower tier instead
- auditing the labels before trusting them: a 60-ticket relabel sample finds 6 real attacks closed as false_positive on one rule, lifting its precision from 1.2% to 10.6% and flipping it from RETIRE to KEEP
- remembering what this whole scorecard cannot see: precision is measured only on alerts that fired, so it says nothing about recall -- the attacks no rule caught
Steps in this lab
- What you are measuring -- every term, defined once
- Set up Python and generate the 90-day ticket export
- Write the scorecard -- one script, five views
- The per-rule scorecard: precision, with its error bars
- The Pareto view: a few rules make most of the noise
- Decide per rule, and measure the before vs after
- The labels lie: a relabel audit flips a retirement
- The honest negatives: what the scorecard gets wrong
- When to do this, why it works, and how to do it better
- What you built -- and the reflex to keep