Lion Cipher Lab

Right-size your model: run one phishing-triage task across three sizes, chart the capability-vs-cost Pareto frontier, and pick the smallest model that clears your bar

• AI Security
• Intermediate
• ~110 min
• Free · no account required

Why it matters

The reflex, the moment a language model has to do a real job, is to reach for the biggest one that fits and defend it in a meeting. It is the most expensive habit in applied AI, and it is almost never checked, because checking it takes a scored task and a frontier and most teams have neither. This lab builds both on a workload every security team recognises - phishing triage, deciding whether an email is a phish or legitimate - and runs ONE such task across three model sizes to answer the only question that matters: what is the smallest model that is actually good enough here? You generate 48…

What you'll learn

• running ONE scored task across THREE model sizes and measuring the four numbers that decide the choice: accuracy against a planted key, p50 and p95 latency, and a cost proxy whose unit you state out loud (here, wall-clock seconds, because on hardware you already own time is what you actually spend)
• building the capability-vs-cost PARETO FRONTIER by hand and marking the DOMINATED points - a size is dominated when another is at least as accurate AND at least as cheap, and a dominated size is never the right answer no matter how good it looks on one axis
• setting an accuracy BAR before you look at the costs, then picking the SMALLEST non-dominated size that clears it - a rule that survives contact with a boss who wants the biggest model and a vendor who wants to sell you tokens
• reading COST PER CORRECT DECISION instead of raw accuracy or raw latency: the 3b answered right for 0.1496 s of wall-clock per correct decision against the 7b's 0.3540 s, so the big model cost 2.31x as much to be LESS accurate
• keeping the honest negative that this lab was built to surface - the biggest model (7b, accuracy 0.9167) is Pareto-DOMINATED by the mid model (3b, accuracy 0.9375) on this task, and can name WHY: the 7b missed the same single phish the 3b did, and lost the top spot purely to over-flagging legitimate marketing as phishing three times
• separating 'bigger is generally more capable' from 'bigger is better for THIS task' - capacity helps on the ambiguous tail, but a clean binary triage with a decisive answer key does not need it, and buying it can import a systematic failure mode you did not have before
• writing the whole study as DETERMINISTIC STUBS so the frontier is reproducible to the digit and free, then swapping in three real Ollama models behind the identical interface when you want to measure your own hardware instead of assuming it

Steps in this lab

1. The words first: capability, cost, the Pareto frontier, a dominated point, the accuracy bar - and the fact that the graded path is a stub
2. Open a terminal and check Python (all three operating systems)
3. PRIMARY path (optional): install Ollama and pull three free models of different sizes - and how to skip one if disk is tight
4. The corpus: 48 phishing-triage emails whose correct verdict you already know
5. The three sizes are DETERMINISTIC STUBS - and here is the entire error model in writing
6. Shared scoring: state your cost unit and mean it
7. Measure all three sizes on the one task
8. Chart the Pareto frontier, mark the dominated size, set the bar, pick the smallest survivor
9. Read the result - the biggest model was the wrong choice, and here is exactly why
10. When bigger IS the right call - because sometimes it is
11. The way a pro does it
12. OPTIONAL appendix: swap the stubs for the three real Ollama models

Open the full interactive lab →