Lion Cipher Lab
Do one malware-report extraction two ways - one giant prompt against a five-step prompt chain - and measure the compounding error that decides which one you should ship
- AI Security
- Intermediate
- ~80 min
- Free · no account required
Why it matters
Everyone who has put a language model near a real analyst workflow has had the same meeting. Somebody says the one big prompt is unreliable, somebody else says we should break it into steps, everyone nods, and the thing gets built. It is a good instinct and it is usually right - in this lab, decomposing one malware-report extraction into five steps took end-to-end accuracy from 0.330 to 0.763, more than doubling the rate at which a usable artefact comes out the other end. What almost nobody in that meeting does is the arithmetic, and the arithmetic is brutal. If five steps must each be right…
What you'll learn
- DECOMPOSING one real analyst task (read an incident report -> emit family, capabilities, IOCs, ATT&CK techniques and a written verdict) into a fixed five-step PROMPT CHAIN, and being able to say out loud why you cut it where you did
- deriving the compounding arithmetic before you measure it: why five steps at 0.95 each is 0.774 end to end, why nobody on the team ever sees a bad per-step number, and why the bad number is always the product
- testing the independence assumption against a chain you actually ran, and finding it wrong in BOTH directions at once - optimistic by 0.051 against the product of intrinsic per-step accuracy, pessimistic by 0.152 against the product of observed per-field accuracy
- measuring the HANDOFF as a first-class cost: isolating what the boundaries themselves take (0.807 with the boundaries switched off against 0.763 with them on) and watching that tax triple as you cut the same work into more pieces
- reading a chain-length sweep instead of a single data point - 1, 2, 3, 4, 5, 7 and 9 steps on the same corpus - and finding the interior optimum, the different optimum for tokens per correct report, and the point where more decomposition starts losing
- diagnosing a POISONED HANDOFF: one wrong extraction at step 2 that every later step builds on faithfully, producing a fluent, well-formed, entirely wrong write-up that calls a ransomware case contained
- building a validation gate on every handoff and MEASURING what it cannot do - 78.9% of transit damage caught and repaired, 0 of 22 plausible-wrong extractions caught - so you stop mistaking a schema check for a truth check
- the judgment to say when NOT to chain: short tasks, tasks where one pass already has all the context, and tasks whose sub-answers are not independently checkable - plus the sub-task in this very lab where the monolith beat the chain
Steps in this lab
- The frame: one task, two shapes, and the arithmetic that decides between them
- Open a terminal (yes, from zero) and check you have Python
- Build the corpus: 300 synthetic incident reports with a known right answer
- The extractor is a DETERMINISTIC STUB - read this before you paste it
- Baseline first: the monolith. One prompt, five fields, and a correlated failure
- Build the chain: five steps, four handoffs, and one deliberate poison route
- The arithmetic, and both ways it is wrong
- Sweep the chain length - one data point is an anecdote, the curve is the finding
- The mitigation that only partly works: a validation gate on every handoff
- When NOT to chain - and why this, not that
- The way a pro does it
- OPTIONAL appendix: swap the stub for a real local model