Lion Cipher Lab

Pick an alert classifier's threshold from a cost matrix, instead of the default 0.5

• AI Security
• Intermediate
• ~75 min
• Free · no account required

Why it matters

Almost every classifier you will put in front of a SOC ships with a hidden decision: predict() flags an alert when the model's probability is above 0.5. Nobody chose that number for your alerts. It is the threshold that maximises accuracy when both kinds of mistake cost the same, and in a SOC they never do. Closing a benign alert costs an analyst a few minutes. Missing an intrusion costs hours or days. This lab makes you write those costs down, in analyst-minutes, and pick the threshold that minimises them. On the lab's synthetic alert stream, the default 0.5 cost 20,458 analyst-minutes a…

What you'll learn

• writing an alert classifier's cost matrix in analyst-minutes (here: false positive 15, true positive 15, missed intrusion 600) and turning it into expected cost per day
• deriving the cost-optimal probability threshold with Elkan's formula, p* = 15 / (15 + 585) = 0.025, and checking it against a threshold tuned on labelled validation data (real run: 9,342 vs 9,318 analyst-minutes a day, against 20,458 at the default 0.5)
• spotting when the formula is wrong: class_weight='balanced' pushed the model's average probability to 0.25 on a 2.1% problem, and Elkan's threshold then flagged 1,810 alerts a day and cost 27,144 minutes, worse than flagging nothing
• repairing it two ways and measuring both: CalibratedClassifierCV (Elkan cost back to 9,440) or tuning the threshold on labels, including scikit-learn's TunedThresholdClassifierCV with a custom cost scorer
• adding an analyst capacity limit: the cost-optimal threshold flagged 317 alerts a day for a team that can work 120, and working them in arrival order cost 19,140 minutes a day against 11,280 highest-score-first
• re-deriving the threshold when the base rate changes after deployment (at 8% malicious the old threshold cost 24,594 a day; Elkan's base-rate correction brought it to 17,462)
• measuring how far the threshold moves when the cost guess is wrong (C_FN 150 to 6,000 moved it from 0.077 to 0.0026) and how noisy a threshold tuned on a small validation set is (bootstrap 95th-percentile cost 14,144 vs 9,441)

Steps in this lab

1. The words first: probability, threshold, cost matrix, calibration, base rate
2. Open a terminal, make a folder, install numpy and scikit-learn
3. Write the cost matrix down, then derive the threshold by hand
4. Measure BEFORE: the default 0.5, then the cost-based threshold
5. The trap: on an uncalibrated model the formula is wrong
6. Fix 1: calibrate the probabilities, then trust the formula again
7. Fix 2: skip the formula and tune the threshold on labels, with TunedThresholdClassifierCV
8. Add the team: a capacity limit the cost-optimal threshold ignores
9. Honest negative 1: the base rate moves after you deploy
10. Honest negatives 2 and 3: the costs are guesses, and a small validation set is noise
11. When to use which, when NOT to bother, and how a pro does it

Open the full interactive lab →