Lion Cipher Lab

Label alerts on a budget with active learning, and measure it against random sampling

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Every security team that wants a machine-learning detector hits the same wall: the alerts are free, the labels are not. A label here means an analyst looked at the alert and decided malicious or benign, and that costs real minutes of a scarce person. If only 3% of alerts are malicious, labeling at random spends 97% of the budget confirming boring things, and 300 random labels buy about nine attacks to learn from. ACTIVE LEARNING is the idea that the model should choose which alerts the analyst labels next, usually the ones it is least sure about. The research literature reports large label…

What you'll learn

• framing alert labeling as a budget problem: a pool of 20,000 unlabeled alerts, a simulated analyst who charges one label per question, and 300 labels to spend
• implementing pool-based active learning with scikit-learn: random sampling, uncertainty sampling (closest to P = 0.5) and two hybrids, in batches of 20
• scoring every strategy on a random held-out test set with PR-AUC and recall at a 1% false-positive rate, averaged over 5 seeds with the spread reported (real run at 300 labels: random 0.705 +- 0.065, uncertainty 0.799 +- 0.010)
• measuring the cold start: uncertainty sampling was behind random at 60 and 100 labels before pulling ahead at 160
• proving sampling bias with numbers: the uncertainty-labeled set was 37.3% malicious when the true rate was 2.9%, so any metric computed on it describes the wrong population
• catching a rare attack sub-type that uncertainty sampling never finds (0.00 recall on all 5 seeds) and recovering it with a farthest-first diversity half-batch (0.81-0.87 recall, PR-AUC 0.885)
• simulating a 10% noisy analyst and reading what it costs each strategy (random fell 0.705 to 0.592, uncertainty 0.799 to 0.766)
• planning a label budget with binomial arithmetic (998 random labels for a 95% chance of seeing one 0.3% sub-type) and testing a stabilizing-predictions stop rule before trusting it

Steps in this lab

1. The words first: label, oracle, pool, budget, active learning - and the fact that everything here is synthetic
2. Open a terminal, make a folder, install numpy and scikit-learn
3. Build the pool: 20,000 synthetic alerts, a separate test set, and a simulated analyst
4. Write the experiment: four ways to spend 300 labels, scored on data none of them saw
5. Measure: random vs uncertainty vs hybrid, five seeds each
6. Honest negative 1: the cold start - uncertainty is only as good as the model asking
7. Honest negative 2: sampling bias - the labeled set lies about the world
8. Honest negative 3: the rare sub-type nobody asked about - then fix it with diversity
9. Honest negative 4: a tired analyst who gets one label in ten wrong
10. Plan the budget and test a stop rule before trusting it
11. Do it better: the way a pro spends a label budget, and when not to bother

Open the full interactive lab →