Lion Cipher Lab
Draft Sigma rules with an LLM — then prove they actually work
- AI Security
- Intermediate
- ~60 min
- Free · no account required
Why it matters
A language model is genuinely good at the boring half of detection engineering: turn "I want to catch PowerShell running an encoded command" into a plausible Sigma rule in seconds, with the fields laid out and the YAML indented. That is a real time-saver. The trap is believing the other half. A model — especially a small one you run for free on your own laptop — will confidently invent field names that do not exist in your logs, pick the wrong logsource, and use modifiers that are not valid, and it will do all of that in output that looks completely correct. A wrong detection is worse than…
What you'll learn
- using a local LLM to draft Sigma detection rules from plain English
- validating a rule with sigma check (pySigma validators)
- converting a rule to a real backend query with sigma convert
- testing whether a rule actually matches the event it should catch
- spotting model hallucinations: wrong logsource, invented field names, bad modifiers
- measuring a model's draft quality instead of trusting it
- judging when to trust an LLM's output and when to verify it against the real schema
Steps in this lab
- Why draft-then-verify beats draft-and-trust
- Install Ollama and a small local model
- Install the verification tooling — sigma-cli and pySigma
- Draft a rule from plain English with the model
- Verify (a): validate the rule with `sigma check`
- Verify (b) and (c): convert it, then test it against the sample event
- Catch the model lie and fix it against the real schema
- Draft several, and MEASURE the hit rate
- When to trust the model, when not — and how to do it better