Lion Cipher Lab

Build an autograd engine from scratch, and prove its gradients with a numerical check

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Every model in security tooling, from a phishing classifier to an LLM, is trained by gradients that an automatic differentiation engine computed. You never see that engine. You call backward() and trust it. In Part 1 you derived backprop by hand for one small network. That works once and does not scale, because every new layer means new calculus and new chances to slip. This lab builds the machine that does the calculus for you, in about 100 lines, so the frameworks stop being magic. Then it teaches the habit that separates engineers from tutorial-followers: you do not trust a gradient you…

What you'll learn

• building a scalar reverse-mode automatic differentiation engine in about 100 lines of standard-library Python: a Value with data, grad, children and op, rules for +, *, pow, exp, log, tanh and relu, and a topological-sort backward()
• writing a central-difference numerical gradient check with a relative-error test, and proving the engine on 200 random graphs (800 gradients, 0 real failures, worst relative error 8.15e-07) and on all 121 parameters of an MLP (worst 5.17e-07)
• reading the epsilon U-curve from your own run: error falls 100x per 10x smaller eps while truncation dominates (1.87e-02 at 1e-1 down to 1.93e-08 at 1e-4), then climbs again as floating-point cancellation takes over (1.00 at 1e-13)
• catching two classic bugs with the check: overwriting grad instead of adding (x*x at 3 gave 3.0, not 6.0; 121 of 121 MLP gradients failed) and forgetting zero_grad (every gradient exactly 2.000x too big)
• knowing the check's blind spots: a plain chain hides the overwrite bug (9.66e-11), relative error alone raises false alarms on near-zero gradients (11 of 800), and relu kinks make correct code fail (16 of 121 with one all-zero event)
• training the MLP on 600 seeded synthetic login events from 0.775 (majority baseline) to 0.920 test accuracy, and showing a model that learned nothing (shuffled labels, 0.775, recall 0.000) passes the same gradient check
• setting an honest tolerance by precision: float64 reaches 2.94e-12, float32 bottoms out at 2.17e-06, and torch.autograd.gradcheck's defaults (eps 1e-6, atol 1e-5, rtol 1e-3) assume double precision
• matching PyTorch autograd on the same graph to 5.55e-16 in float64, and measuring why nobody trains with scalar autograd: one forward+backward on 400 events took 4057 ms against torch's 56 ms (72x)

Steps in this lab

1. What you are building, and Part 1 in one paragraph
2. Set up: a terminal, Python, and a lab folder
3. Write the engine: a Value that remembers how it was made
4. Prove it: a numerical gradient check on 200 random graphs
5. Check all 121 MLP parameters, then find the right epsilon (the U-shaped curve)
6. Plant two classic bugs and watch the check catch them
7. Train it on synthetic login events: before and after
8. Honest negative 1: a passing gradient check does not mean a good model
9. Honest negative 2: the relu kink, where correct code fails the check
10. Honest negative 3: float32 vs float64, and how slow scalar autograd really is
11. Do it better: how a professional uses gradient checks

Open the full interactive lab →