Lion Cipher Lab
Stop a threat-hunting agent at the right time with a sequential probability ratio test
- AI Security
- Advanced
- ~90 min
- Free · no account required
Why it matters
A threat hunt is a run of questions: did this host spawn an odd child process, does it beacon, did someone clear its security log. An AI hunting agent asks those questions one at a time, and every question costs something - a query against the SIEM, an API call, tokens, an analyst's attention when it escalates. So the agent has to decide, after each answer, whether it has seen enough. Stop too early and you get the classic System 1 mistake that clinicians call premature closure: the first scary indicator becomes the verdict, or three quiet answers become 'clean'. Always ask a fixed number of…
What you'll learn
- building an answer-keyed set of 400 synthetic hosts whose evidence streams come from eight indicators with written-down likelihoods under compromised and clean, so every stopping decision can be graded
- turning each piece of evidence into a log-likelihood ratio and keeping a running sum, the core of Wald's sequential probability ratio test (SPRT)
- setting SPRT's two stopping thresholds from a target false-positive rate (alpha) and false-negative rate (beta), and checking the measured rates against those targets
- measuring three stopping policies on the same hosts - premature closure, fixed-N and SPRT - by false-positive rate, false-negative rate and average queries per host, instead of by accuracy alone
- showing with numbers that SPRT's promise breaks when evidence is duplicated (the same EDR event seen through the SIEM) and repairing it by de-duplicating on the event id, while pricing the repair
- showing that wrong likelihoods shift the error rates, and that re-estimating them from a labelled sample helps but does not fully restore the target
- pricing a max-queries cap honestly: how many hosts become 'undecided - escalate to a human' at each cap, in analyst hours
Steps in this lab
- The words first: hunt, indicator, likelihood, log-likelihood ratio, false positive and negative, alpha and beta, premature closure, SPRT - and the fact that the agent is a stub
- Open a terminal, check Python, make a scratch folder (Windows, macOS, Linux)
- Build the keyed hosts: 400 hosts, 100 compromised, 40 evidence results each
- Write the stub hunting agent: premature closure, fixed-N and SPRT
- The agent is a STUB, and here is its whole behaviour in writing
- Measure: false-positive rate, false-negative rate and queries per host for every policy
- Read it honestly: what the table proves and what it does not
- Break it twice: echoed evidence and wrong likelihoods, and what each repair buys
- Cap the queries: what 'undecided - escalate to a human' costs at each cap
- When, why, and how a pro does it differently
- Ship it: the policy, what to log, and how to see what the stub is doing