Lion Cipher Lab

Race System-1 (snap answer) against System-2 (deliberate reasoning) on one alert-triage batch, and find the band where thinking longer stops paying

• AI Security
• Intermediate
• ~90 min
• Free · no account required

Why it matters

The moment a team wires a language model into alert triage, someone asks the reasonable-sounding question: should we let it think step by step before it answers? The reasoning literature says longer deliberation lifts accuracy, so the instinct is to turn it on everywhere - and the bill, in tokens and latency, arrives everywhere too. This lab takes that instinct apart with an answer key. It builds ONE deterministic stub reasoner with two modes - a System 1 snap that glances at a few salient enrichment channels and names a severity in one fixed 15-token pass, and a System 2 deliberate that…

What you'll learn

• building ONE stub reasoner with two modes from pure standard-library Python - a System 1 snap that reads a few salient channels in a single fixed-cost pass, and a System 2 deliberate that scores every channel, weighs them and concludes at 7x the token cost - so the whole fast-vs-slow tradeoff is something you can read and change rather than something a vendor sets
• measuring accuracy AND cost per difficulty band instead of quoting one overall number: 0.9933 vs 0.9400 on easy, 0.8133 vs 0.8600 on moderate, 0.6333 vs 0.7267 on hard, 0.4667 vs 0.4733 on ambiguous, each against a fixed 15-token snap and 105-token deliberate, so the payoff is priced band by band
• keeping the honest negative that deliberation HURTS on easy alerts: the caution slip flipped 9 of 150 already-correct easy verdicts to wrong and recovered none, so on a band the snap already nails, System 2 spends 7x the tokens to lose accuracy
• reading a CEILING off a measurement - on the ambiguous band deliberation moved accuracy by +0.0067 because the evidence is genuinely uninformative, which is the difference between a hard problem (thinking helps) and an ambiguous one (thinking cannot help because the information is not there)
• pricing blanket deliberation honestly: +14 correct answers out of 600 for 54,000 extra tokens is about 3,857 tokens per additional correct answer, a number that only looks bad because most of it is spent on alerts that were never in doubt
• turning the result into an operating rule - gate the slow path on the fast path's own confidence, deliberate only the 405 low-margin alerts, and land at 0.7667 accuracy (above the 0.7500 you get from deliberating everything) for 72% of the tokens
• seeing accuracy PEAK THEN FALL as the deliberation gate widens (0.7567, 0.7617, 0.7667, 0.7617, 0.7533, 0.7500), which is the band where more thinking stops paying made visible as a single column
• making every result reproducible byte-for-byte with a fixed seed and hashlib-derived (never process-randomised hash()) decisions, so the stub's slip fires on exactly the same alerts on any machine

Steps in this lab

1. The words first: System 1 and System 2, snap and deliberate, difficulty band, answer key, accuracy and cost - and the fact that the model here is a stub
2. Open a terminal and make a scratch folder (all three operating systems)
3. The corpus: 600 alerts, four difficulty bands, correct severity planted by construction
4. Write the reasoner: one stub, two modes - snap and deliberate - and their costs
5. The model is a STUB, and here is its entire behaviour in writing
6. Race them: accuracy and cost, per difficulty band
7. Read the result honestly: no gain on easy, a payoff in the middle, a ceiling on ambiguous
8. When to deliberate: gate the slow path on the fast path's own confidence
9. Why this, not that - and when NOT to deliberate at all
10. Ship it: the policy to run in a triage pipeline, and what to write down

Open the full interactive lab →