Lion Cipher Lab

Mine log templates with Drain, then measure an LLM parser against it (deterministic stub LLM)

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Almost every log anomaly detector, and every 'new event type seen' alert, starts by turning raw log lines into TEMPLATES - 'Failed password for <*> from <*> port <*> ssh2' - so that 50,000 lines become 40 event types you can count. Get that step wrong and everything downstream inherits the error. Today there are two ways to do it: Drain, a fast tree-based algorithm from 2017, or asking a language model. This lab builds both on 2,000 answer-keyed security log lines and measures them honestly. Drain ran in well under a second with no model at all and grouped 88% of lines correctly out of the…

What you'll learn

• building a 2,000-line answer-keyed corpus of sshd, sudo, pam, ufw, web and cron logs from 22 known templates, with a rare intrusion burst planted near the end
• implementing Drain from the ICWS 2017 paper in under 100 lines of standard-library Python: a fixed-depth tree by length then leading tokens, a similarity threshold, and regex masking
• grading a log parser with the two standard Loghub metrics, grouping accuracy and parsing accuracy, and explaining why one stray line can make a whole group of 300 count as wrong
• sweeping the similarity threshold and tree depth and reading why no threshold fixed this corpus: the web-log over-merge and the short-pam-line split sit on the same 0.833 boundary
• comparing a per-line LLM parser on accuracy, tokens, seconds and repeatability: PA 0.950 but GA 0.018, about 487,000 tokens and 591 seconds per 1,000 lines, and 209 of 2,000 templates changing between two runs
• designing a hybrid that calls the LLM once per Drain group instead of once per line: PA 0.970 for about 1% of the per-line token bill
• turning templates into a security signal (new-template alerts) and measuring the false-alert cost of an inconsistent parser
• showing that one hand-written preprocessing regex moved grouping accuracy from 0.883 to 1.000, more than any threshold, depth or model choice did

Steps in this lab

1. The words first: templates, variables, Drain, the two accuracy metrics - and the fact that the LLM here is a stub
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build 2,000 answer-keyed security log lines from 22 known templates, with an intrusion planted near the end
4. Write Drain from the paper: masks, a fixed-depth tree, a similarity threshold, and one pass
5. Grade it: grouping accuracy and parsing accuracy at three levels of preprocessing
6. Read the failures: variables in front, variables of any length, and masks that eat constants
7. Sweep the two knobs: similarity threshold and depth - and find that no threshold fixes this corpus
8. Write the STUB LLM parser, with its whole error model written down
9. Race them: Drain, the per-line LLM and two hybrids, on accuracy AND cost per 1,000 lines
10. Read the race honestly: the best templates, the worst groups, and a bill three orders of magnitude bigger
11. The security payoff: alert on templates that are new, and count the false alerts
12. Why this, not that - and when NOT to use either
13. The way a pro does it differently - and one regex that beats every knob

Open the full interactive lab →