Lion Cipher Lab

Split a phishing classifier's data by time, and measure how much a random split flatters it

• AI Security
• Intermediate
• ~60 min
• Free · no account required

Why it matters

Almost every tutorial evaluates a classifier the same way: shuffle the rows, hold out 25 or 33 percent, report the score. For phishing that number is close to fiction. Phishing arrives in campaigns, and one campaign sends hundreds of near-identical messages over a few weeks. Shuffle them and the model is tested on copies of messages it trained on. The attackers also change over the year: new phishing kits appear, and the features that worked in January stop working by autumn. In deployment the model only ever sees messages from AFTER its training data, so that is how it has to be tested. In…

What you'll learn

• generating a seeded, synthetic year of phishing and benign feature data where campaigns arrive as bursts of near-duplicates and new kit families appear over time
• scoring one scikit-learn model under a random split and a time split and reading the gap (real run: PR-AUC 0.992 vs 0.607, recall 0.892 vs 0.167)
• decomposing the gap with a campaign-grouped split: campaign leakage alone was worth 0.992 -> 0.836, future information the rest, 0.836 -> 0.607
• measuring leakage directly: what share of test phishing comes from a campaign or kit family the training set already contained (100% vs 4.6% of campaigns)
• showing that deduplication does not fix a random split (exact dedup removed 0 rows; a rounded near-dup filter removed 665 and PR-AUC stayed 0.990)
• adding an embargo gap before the test window and explaining why the honest score went DOWN (0.607 -> 0.557)
• running rolling-origin evaluation with sklearn TimeSeriesSplit(gap=...) and reporting a spread, not one number (8 folds, PR-AUC 0.503 to 0.781)
• proving that monthly retraining still misses brand-new campaign families (recall 0.000 on them in every month they appeared)

Steps in this lab

1. The words first, and why a shuffled split lies about phishing
2. Set up: a terminal, Python, numpy and scikit-learn
3. Generate a synthetic year where phishing evolves
4. Measure BEFORE and AFTER: random split vs time split
5. Why: take the 0.385 gap apart into campaign leakage and future information
6. Honest negative 1: deduplicating does not rescue the random split
7. The fix, part 1: a time split with an embargo gap
8. The fix, part 2: rolling-origin folds with TimeSeriesSplit, and honest negative 2: one window misleads
9. Honest negative 3: retraining every month still misses brand-new kit families
10. Do it better: how to evaluate a security classifier you will actually deploy

Open the full interactive lab →