Lion Cipher Lab

Probe a phishing classifier's hidden layers, catch the shortcut it learned, and prove with an intervention which concepts it actually uses

• AI Security
• Advanced
• ~2 hours
• Free · no account required

Why it matters

A phishing classifier that scores 95% on its test set is the kind of result that ships. This lab builds one that does exactly that and is close to useless, and then teaches the two-part method that catches it. The training mail was collected so that a harmless mail-gateway header tag happened to ride along with 95% of the phishing and almost none of the legitimate mail - the kind of accident that happens when one feed of known-bad mail comes through a different relay than everything else. A small neural network, trained the normal way, scores 0.9520 on a test set drawn the same way. On clean…

What you'll learn

• planting a spurious shortcut in a synthetic phishing dataset (a harmless mail-gateway header tag that tracks the label 95% of the time in training and is a coin flip in a clean test set) and measuring the damage: 0.9520 in-distribution accuracy against 0.5830 on clean mail, a 0.3690 gap the usual test split cannot see
• pulling the hidden-layer activations out of a scikit-learn MLPClassifier by hand from coefs_ and intercepts_ with numpy, and proving the hand-rolled forward pass matches predict_proba to a max difference of 0.0
• fitting a linear probe (logistic regression) per layer for three concepts - a real lookalike-domain signal, the shortcut tag, and an irrelevant attachment flag - on held-out clean data with chance printed beside every score
• reading the probe table: the shortcut tag decodes at 0.998 and 0.981 from the two hidden layers of the shortcut-trained model, while the real lookalike concept fades from 0.925 at the input to 0.713 at the last hidden layer
• keeping the central honest negative: the attachment flag probes at 0.916 from hidden layer 1 (chance 0.578) yet projecting that direction out flips only 1.5% of verdicts and permuting the input flips 2.5% - the same as the pure-noise hash columns - so decodable is not the same as used
• probe, then intervene: projecting the shortcut direction out of hidden layer 1 flips 28.0% of verdicts and lifts clean accuracy from 0.5830 to 0.7175, which is the causal evidence a probe score alone can never give
• fixing the shortcut by decorrelating the tag in training and measuring both sides: clean accuracy 0.5830 to 0.8150, last-layer shortcut probe 0.981 to 0.638, and in-distribution accuracy FALLING from 0.9520 to 0.8160, which is the old test set punishing the honest model

Steps in this lab

1. The words first: shortcut learning, MLP, hidden layer, linear probe, and why decodable is not the same as used
2. Open a terminal, make a scratch folder, and install scikit-learn into a throwaway virtual environment
3. Build the mail: eight features, a real signal, and a planted shortcut that only exists in training
4. Train the network and measure the gap the normal test split hides
5. Why the usual evaluation could not catch it, and what a clean test set has to be
6. Pull out the hidden layers by hand and fit a linear probe per layer
7. Read the probe table, and notice the row that should worry you
8. Probe, then intervene: remove each concept and count how many verdicts flip
9. Fix the data, retrain, and measure every side of the fix
10. What the fix did not fix, measured
11. Why this, not that - and how a professional does it better
12. Ship it: the three numbers to put in a model review, and clean up

Open the full interactive lab →