Lion Cipher Lab
Catch base-rate neglect in a triage model's verdict confidence, and fix it with an explicit prior
- AI Security
- Intermediate
- ~70 min
- Free · no account required
Why it matters
A detector that catches 99% of attacks and false-alarms on only 1% of clean events sounds excellent. Put it on a network where 1 event in 1,000 is malicious and, when it fires, the alert is malicious about 9% of the time. That is not a flaw in the detector; it is arithmetic, and Stefan Axelsson showed in 2000 that it is the central difficulty of intrusion detection: when attacks are rare, false alarms outnumber true ones unless the false-positive rate is tiny. People get this wrong reliably - Tversky and Kahneman documented in 1974 that we judge by how well the evidence fits and neglect how…
What you'll learn
- building three synthetic sites of 50,000 events each with known prevalence (0.1%, 1% and 10% malicious) and three detectors with known sensitivity and false-positive rate, so every probability a model states can be graded
- measuring probability calibration instead of accuracy: the Brier score, expected calibration error (ECE), a reliability table, and a count of alerts stated at 90% or more against how many of them were really malicious
- working Bayes' rule by hand once (99% sensitive, 1% false positives, 0.1% prevalence gives a 9% posterior) and then as prior odds times likelihood ratios in a few lines of Python
- combining two independent detectors by multiplying their likelihood ratios, and seeing that the prior fixes calibration while the second detector adds sharpness
- measuring three ways the fix breaks: a prior carried to the wrong site, a prior estimated only from the alerts analysts chose to investigate (112x too high here), and two correlated detectors multiplied as if independent
- estimating prevalence from the detector's firing rate over ALL events, and knowing why that estimate is noisy when attacks are rare
Steps in this lab
- The words first: prevalence, base-rate neglect, Bayes' rule, likelihood ratio, calibration - and the fact that the model is a stub
- Open a terminal, check Python, make a scratch folder (Windows, macOS, Linux)
- One-step recap: the race lab measured accuracy by difficulty; this lab measures whether the stated probability is true
- Build three sites that differ only in how rare attacks are
- Write the System-1 stub: it reports the detector's hit rate and never looks at the base rate
- Write the calibration ruler: Brier score, ECE and a reliability table
- Measure BEFORE: the stub's probabilities at three sites
- System 2, by hand once: 99% sensitive, 1% false positives, 0.1% prevalence is a 9% posterior
- Measure AFTER: Bayes with each site's own prior, one detector and then two
- Three honest negatives: the wrong prior, verification bias, and two detectors that are not independent
- Why this, not that - and when NOT to bolt Bayes onto a triage model
- The way a pro does it: priors per site and per time, likelihoods from a random audit, a log of every posterior
- Ship it: re-run, break a constant on purpose, and clean up