Lion Cipher Lab
Measure safety-alignment erosion from fine-tuning: build a refusal-rate eval harness that catches when a fine-tune un-aligns your model
- AI Security
- Advanced
- ~60 min
- Free · no account required
Why it matters
Fine-tuning is treated as a capability upgrade: teach the model your domain, ship it. But a landmark result (Qi et al., 2023) showed that fine-tuning an ALIGNED model can significantly erode its safety — it starts complying with requests it used to refuse — and this happens even when the fine-tuning data is entirely benign, and even more so with just a few adversarial rows. Follow-on work showed the same safety training can be cheaply undone with LoRA. The practical consequence: every team that fine-tunes a model is running a safety experiment whether they know it or not, and almost none of…
What you'll learn
- building a balanced safety eval set (a 'should-refuse' class and a 'should-comply' class)
- writing a deterministic keyword/regex refusal classifier and measuring its own precision
- computing a refusal rate and an over-refusal rate — and why you need both
- measuring safety-alignment erosion as a baseline-vs-after DELTA with real numbers
- running the harness offline with deterministic stub models, or against a live local model
- gating CI on a refusal-rate drop so a safety regression blocks a model release
- knowing the mitigations (held-out safety eval, mix refusal data back in, re-tune after) and their limits
- judging that a fine-tune is a SAFETY event, not just a capability change
Steps in this lab
- The finding: safety alignment is shallow, and fine-tuning erodes it
- The honest metric is a PAIR: refusal rate AND over-refusal rate
- Build the balanced safety eval set
- Get responses two ways: deterministic stubs offline, or a live local model
- Write the refusal classifier — and measure its OWN precision
- Run it and read the erosion delta (real numbers)
- Gate CI on the erosion so a regression blocks the release
- Mitigations and their limits — plus the named failure modes
- The judgment: a fine-tune is a safety event, not just a capability change