Lion Cipher Lab

Route a SOC alert queue across a small model, a large model and a human — then prove the router beats both extremes on cost per correct decision

• AI Security
• Intermediate
• ~75 min
• Free · no account required

Why it matters

Every team that puts a model near a real queue eventually hits the same wall. The big model is good and it is expensive. The small model is nearly free and it is wrong often enough to scare you. And the human, who is better than both, does not scale and costs more per decision than either. The reflex is to pick one and defend it in a meeting. The engineering answer is to stop picking and start ROUTING: send each item to the cheapest tier that is likely to get it right, and escalate the rest. That idea is not new — FrugalGPT formalised the LLM cascade, RouteLLM learned the routing decision…

What you'll learn

• designing a three-tier CASCADE ROUTER (cheap local model -> large model -> human escalation lane) over a real workload, and writing the routing policy as code you can ablate rather than a paragraph of intent
• building the workload honestly: a synthetic-but-realistic labelled SOC alert corpus generated from a seeded RNG (random.Random(20260909)) so every number in the lab is reproducible on your machine
• standing up decision tiers as DETERMINISTIC STUBS with explicit accuracy and cost curves, so you can measure the ROUTER without installing, paying for, or trusting any model
• pricing the errors asymmetrically: a missed escalation (a real intrusion closed as noise) is the expensive mistake, a needless escalation only burns analyst minutes — and encoding that asymmetry as two DIFFERENT confidence thresholds instead of one
• measuring cost-per-correct-decision against both naive baselines (everything to the big model, everything to the small model) and reading the result even when it embarrasses your design
• the ablation habit: symmetric vs. asymmetric thresholds, with vs. without a hard feature rule, three tiers vs. two — running each and keeping only the parts that pay for themselves
• the judgment to say when NOT to route: the queue where the winner is a plain single-tier policy, the safety threshold that collapses the router into an all-human pipeline, and the price sheet that flips the whole conclusion

Steps in this lab

1. The frame: this lab decides WHO answers, not WHETHER to answer
2. Build the queue: 400 labelled alerts from a seeded RNG
3. The three tiers are DETERMINISTIC STUBS — and here is why that is the honest choice
4. Price the errors: the missed escalation is the expensive one
5. Measure the two extremes first — you cannot beat a baseline you never ran
6. Is confidence actually a routing signal? Check before you build on it
7. Build the router: two thresholds, not one, because the errors are not equal
8. Read the result — including the part where your router got LESS accurate
9. Ablate: find out which of your ideas actually paid for itself
10. The safety dial: tune for safety and your router stops being a router
11. The whole conclusion rests on one number you invented — so move it
12. When NOT to route — why this, not that
13. The way a pro does it
14. OPTIONAL appendix: swap the stubs for real local models

Open the full interactive lab →