Lion Cipher Lab
Draft Semgrep rules with a local LLM, then prove them against a labelled corpus
- AppSec / DevSecOps
- Intermediate
- ~75 min
- Free · no account required
Why it matters
Writing a custom static-analysis rule is slow, fiddly work, and a code model will hand you a confident first draft in seconds. That draft is the trap. In this lab the draft looks right, says it ignores safe code, and runs without a single warning, yet a third of its findings are false alarms and it misses a real injection. Teams that ship rules like that teach developers to ignore the scanner. The skill an application security engineer is hired for is not prompting; it is proving. You will build the evidence (a labelled corpus, a precision and recall score, a Semgrep test suite and a held-out…
What you'll learn
- prompting a free local code model (Ollama) for a first-draft Semgrep rule
- catching invented rule syntax with semgrep --validate, and knowing that a plain scan will not catch it for you
- scoring a rule on a labelled corpus: true positives, false positives, misses, precision and recall
- turning a corpus into a Semgrep test suite with ruleid and ok annotations and semgrep --test
- fixing an over-broad rule with pattern-not, and a missed variant with patterns plus metavariable-regex
- telling safe subprocess use (argument lists, shlex.quote, constants) from OS command injection (CWE-78)
- using a held-out set to catch a rule you have tuned to its own test file
- deciding when a model draft is worth keeping and what a human still has to prove
Steps in this lab
- What you are proving, and the words you need
- Set up: a lab folder and Semgrep in a virtual environment
- Save the labelled corpus: 10 vulnerable snippets, 10 safe ones
- Save the scorer: precision and recall from the labels
- Get a draft: a live local model, or the labelled stub
- Gate 1: does the draft even use real Semgrep syntax?
- Measure BEFORE: score the raw draft on the corpus
- Turn the corpus into a Semgrep test suite
- Fix it by hand, one failure mode at a time, and re-measure each round
- Measure AFTER, then run a held-out set you did not tune on
- If you used a live model: measure the variance, not one lucky run
- The judgment: when to trust the draft, and how a pro does this