Lion Cipher Lab

Extract IOCs from a threat report with a local LLM, score it against your own regex, and catch the IOCs it invents

• Threat Intel Analyst
• Intermediate
• ~75 min
• Free · no account required

Why it matters

Pasting a threat report into a chatbot and asking 'what are the IOCs?' is now a normal Tuesday for an analyst, and the answer looks great: neat, refanged, de-duplicated. Some of it is invented. In this lab the model's reply contained three indicators that exist nowhere in the report - a sibling domain, a 'completed' URL and a hash with one character wrong - and a hash that is one character wrong is worse than no hash, because it looks checked. A plain regex invents nothing but cannot read: it flagged a software version number as an IP and a vendor's update server as an IOC. The skill this lab…

What you'll learn

• writing a ~40-line regex IOC extractor in Python that refangs hxxp, [.] and (dot) before it matches, and reading exactly why it scored 52.9% precision and 90.0% recall on a seeded report
• scoring any extractor against an answer key with precision, recall and a HALLUCINATED count, where a hallucinated IOC is one whose value appears nowhere in the report by exact string search
• spotting the three ways a model invents IOCs in practice: a plausible sibling domain, a 'completed' URL path, and a hash copied with one character wrong
• constraining a local model to a JSON schema and making it quote the source span for every IOC, then dropping any IOC whose span is not in the report or does not contain the value
• measuring the gate's honest cost: a strict verbatim check also threw away a real hash that was split across a line, and a whitespace-blind check won it back
• knowing what no gate fixes: a context error (an IP the report says is NOT related) has a perfectly real span, so it survives, and only a human reading the sentence catches it
• building a review queue from two extractors (agree / regex-only / model-only) instead of pushing either one's output straight to a blocklist

Steps in this lab

1. The words first: IOCs, defanging, regex, a local LLM, precision, recall, and a hallucinated IOC
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Write the synthetic report and its answer key - and read the seven traps planted in it
4. Write your own regex extractor in about 40 lines
5. Write the scorer and score the regex: 90% recall, 53% precision, zero inventions
6. Score the STUB model's plain answer: better precision, three invented IOCs
7. OPTIONAL, live: install Ollama, run a real small model on the same report, and score it
8. The better way: force JSON, make every IOC quote its source span, and drop the ones that cannot
9. The honest negatives: what the gate cannot fix, and why the result is a review queue, not a blocklist
10. The judgment: regex vs model, when NOT to use a model at all, and how a pro does it

Open the full interactive lab →