Lion Cipher Lab
Build gradient-boosted trees from scratch for phishing URL detection, and match scikit-learn
- AI Security
- Advanced
- ~2 hours
- Free · no account required
Why it matters
Gradient-boosted trees are the default answer for tabular security data: URL features, login records, process telemetry, email metadata. Teams reach for them because they usually win. Most people use them as a black box with three magic settings. That is fine until something goes wrong: the model overfits, the probabilities are off, the importance chart points at nonsense, or a phisher changes two cheap things and walks past it. This lab takes the box apart. You write the algorithm yourself in numpy, straight from the 2001 paper that defined it. Then you prove your version gives the same…
What you'll learn
- writing gradient boosting for binary log-loss from Friedman (2001) in about 150 lines of numpy: log-odds start, residuals y - p, a depth-limited regression tree, Newton leaf values sum(r) / sum(p(1-p)), and a learning rate
- proving a from-scratch model against a library row by row: with min_samples_leaf=5 the largest gap from sklearn's GradientBoostingClassifier over 4,000 test rows was 3.89e-16
- explaining a real mismatch instead of hiding it: with min_samples_leaf=1, 12 of 4,000 rows differed (largest gap 0.118) because tiny leaves had tied splits and sklearn breaks ties with a random feature order
- showing why trees beat a straight line on interaction-driven data: boosting reached PR-AUC 0.4999 where logistic regression got 0.3344 and one depth-3 tree got 0.2472
- reading train and validation loss curves: at learning rate 0.3 validation loss bottomed at 25 trees (0.2047) and rose to 0.234 by 300; at 1.0 the Newton steps blew up in both implementations
- using early stopping on a held-out set and knowing its limits (patience 20 stopped at 85 trees and kept 65; the 300-tree curve showed a later minimum at 100)
- checking calibration with a reliability table, and timing exact boosting against histogram boosting (24.5 s scratch, 14.7 s sklearn exact, 1.2-1.5 s HistGradientBoosting on one thread)
- catching the impurity-importance trap with a planted random feature (impurity ranked it 9th of 13, permutation ranked it last) and measuring how far cheap URL edits drop recall (52.1% to 24.4%)
Steps in this lab
- What you are building, and the words you need
- Open a terminal, make a folder, install numpy and scikit-learn
- Generate the synthetic URL features (with a hidden rule and a planted trap)
- Measure BEFORE: a straight line and a single tree
- Write the booster: residuals, a regression tree, and a Newton step in every leaf
- Prove it: match scikit-learn row by row, and explain the rows that differ
- Learning rate, number of trees, and early stopping, measured
- Honest negative 1: the probabilities are close on average, not in every range
- Honest negative 2: your version is slow, and exact boosting is slow next to histograms
- Honest negative 3: impurity importance gives a random column real weight
- Honest negative 4: good synthetic scores say nothing about real phishing
- When to use boosting, when not to, and how a pro does it better