Lion Cipher Lab

Distil an expensive model's phishing verdicts into a tiny local classifier, and measure what the student inherits

• AI Security
• Advanced
• ~100 min
• Free · no account required

Why it matters

Every security team that has tried a frontier model on phishing triage hits the same wall: it labels emails well, but it is too slow and too expensive to run on every message. The standard move is DISTILLATION - let the big model label a few thousand emails once, train a tiny model on those labels, and run the tiny one on everything. It works: here a student that costs nothing to run, answers in microseconds instead of 1.2 seconds, and handles 92% of email on its own inside a cascade matched the expensive model's accuracy for $4.65 per 10,000 emails instead of $60. But a student learns its…

What you'll learn

• distilling a big model into a small one by hand: a teacher labels emails, a hashed-feature logistic-regression student learns from those labels with stochastic gradient descent, in pure standard-library Python
• comparing four ways to teach the student - 300 human gold labels, teacher hard labels, teacher soft labels at temperature 1 and 3, and the teacher's spoken confidence - on accuracy, agreement with the teacher, labelling cost and running cost
• reading a label-count curve: hard-label students went 0.762, 0.803, 0.826, 0.829, 0.829 from 100 to 6,000 teacher labels, so past about 1,000 labels more teacher calls bought nothing
• measuring when soft labels pay: +2.5 points at 100 labels with real log-probabilities, nothing at the plateau, and nothing at all from a badly calibrated spoken confidence (ECE 0.121)
• building a cascade in which the student answers what it is sure about and defers the rest: coverage 0.922, accuracy 0.848 (the teacher alone was 0.844), $4.65 per 10,000 emails instead of $60
• catching the inherited blind spot: the teacher misses QR-code lures (0.069), the student misses them all (0.005) and more confidently, while agreeing with the teacher 89% of the time
• repairing it with a small, targeted human budget (200 labels, $300: QR lures 0.000 to 0.850) and detecting a new campaign with a cheap drift monitor before re-distilling for $6

Steps in this lab

1. The words first: teacher, student, distillation, hard and soft labels, calibration, cascade, drift - and the fact that the teacher here is a stub
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Where this sits: you are building the cheap model, not choosing one
4. Build 12,300 answer-keyed emails: nine lure and legitimate families, a hidden-image lure, and a week-2 campaign
5. Write the STUB teacher: its whole error model, including the blind spot, written down
6. Write the student: hashed words, logistic regression, and SGD by hand
7. Distil four ways and draw the label-count curve
8. Soft labels helped less than hoped - and the spoken confidence helped not at all
9. Build the cascade: the student answers what it is sure about, the teacher gets the rest
10. The honest negative: the student inherits the blind spot, more confidently, and agreement hides it
11. Repair it with a small, targeted human budget - and see why adding labels alone was not enough
12. Prove where the failure came from: give the teacher eyes and re-run
13. Week 2: a new campaign - the teacher bends, the student breaks, and a $0.60 monitor notices
14. Read the terms before you distil: some providers forbid training on their outputs
15. Why this, not that: distil versus keep calling, a bigger student, or human labels only
16. When NOT to distil
17. The way a pro does it differently, and what to ship

Open the full interactive lab →