Lion Cipher Lab

Catch training-serving skew in a phishing model's feature pipeline, before it ships

• AI Security
• Intermediate
• ~75 min
• Free · no account required

Why it matters

A phishing model is trained in one place and used in another. The data scientist computes features in a notebook or batch job over historical mail. The mail gateway computes them again, live, in code someone else wrote from a design doc. If those two pieces of code disagree, even slightly, the model in production is scoring numbers it never saw in training. This is TRAINING-SERVING SKEW. Google's Rules of Machine Learning names it as a common production failure, and the ML Test Score paper calls the test for it 'perhaps the most important and least implemented'. Nothing crashes. Accuracy on…

What you'll learn

• building a controlled skew experiment: one trained model, one set of held-out emails, two feature pipelines, so any score change is caused by the pipelines alone (real run: recall 0.938 -> 0.787, PR-AUC 0.990 -> 0.930)
• spotting seven realistic skew bugs in feature code: case, percent-decoding order, a 'www.' prefix, 0 instead of NaN for a missing header, local time instead of UTC, a different tokenizer, and a vocabulary built at serving time
• pricing each skew by swapping one serving feature at a time, and learning that 'rows that differ' is not 'damage done' (host_dots differed on 1,068 rows and cost 0.001 PR-AUC; body_kw differed on 321 and cost 0.019)
• writing a golden-set feature parity test that reports the number of skewed features and the number of skewed rows per feature (v1 failed 7 of 9 features on 40 of 40 emails)
• proving a parity test only covers the cases in its golden set: a double-encoded-link bug passed on 40 emails and then hit 26 of 2,400 live emails
• moving feature logic into one shared function used by training and serving, and showing that shared code still skews when serving receives pre-parsed headers (2,400 of 2,400 rows wrong, recall 0.938 -> 0.822)
• monitoring logged serving features with PSI and the two-sample KS test, and measuring the blind spot: a column joined by position instead of by id kept PSI at 0.000 while accuracy fell 0.970 -> 0.952
• running the row-level check (log serving features with the email id, recompute offline, diff) that caught the shuffled join on 135 of 240 sampled rows

Steps in this lab

1. The words first: features, pipelines, and training-serving skew
2. Set up: a terminal, Python, numpy and scikit-learn
3. Generate a synthetic mailbox
4. Two pipelines, one feature list, seven differences
5. Measure BEFORE: same model, same emails, two pipelines
6. Fix 1: a feature parity test on a golden set
7. Honest negative 1: a parity test only covers its golden set
8. Fix 2: one shared feature function for training and serving
9. Honest negative 2: shared code still skews when the input arrives differently
10. Fix 3: log serving features and compare their distributions to training
11. Honest negative 3: a skew that keeps the distribution, and the row-level check that catches it
12. Do it better: the layered setup, and when each layer is worth it

Open the full interactive lab →