Lion Cipher Lab
Measure the false-refusal rate of an LLM safety guardrail on legitimate security work
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
A safety guardrail is code (or a model) that sits in front of an AI system and decides whether a request is allowed through or refused. The obvious failure everyone worries about is the guardrail letting a genuinely harmful request through. There is a second failure, quieter and far more common, that this lab is about: over-refusal, also called exaggerated safety - the guardrail refusing a completely legitimate request because it pattern-matches on scary-sounding words. A security team asks "how do I read my own auth logs for failed SSH logins," "explain what this CVE does so I can tell if we…
What you'll learn
- explaining over-refusal (exaggerated safety) and why it is a real failure mode, not a nuisance
- assembling a benign-but-scary-sounding test set of genuine defensive-security requests
- assembling a harmful test set as labelled intent stand-ins, never operational instructions
- writing a deterministic keyword/score guardrail classifier in pure-stdlib Python
- measuring the False Refusal Rate (FRR) and the harmful-allow rate from a labelled set
- sweeping a strictness threshold and reading the tradeoff curve between the two errors
- adding intent/context features to a classifier and showing the curve shift with real numbers
- choosing an operating point for the users you actually have (SOC vs public bot)
Steps in this lab
- What over-refusal is, and why a defender should care
- The two numbers you are going to measure
- Build the prompt sets and a naive keyword guardrail
- Sweep the threshold and read the tradeoff curve
- Add intent and context features, then re-measure
- The honest negatives - read these before you trust any number