Lion Cipher Lab
Audit and defend a model's unintended memorisation with the Secret Sharer
- AI Security
- Advanced
- ~1 hour
- Free · no account required
Why it matters
A language model does not only generalise — it can quietly memorise and later recite exact strings from its training data: an API key, a social-security number, a credit-card number, a private message. This is not the same failure as overfitting, and you cannot see it by looking at accuracy or a loss curve. Carlini and colleagues showed it is real and measurable on production-scale models, and later showed you can pull verbatim training text — including personal data — straight out of GPT-2 with the right prompts. So 'did our model memorise the secrets in its training data?' is now a shipping…
What you'll learn
- the difference between overfitting and unintended memorisation (verbatim regurgitation of secrets)
- planting a canary of known format a controllable number of times into a training corpus
- training a small character-level LSTM language model on a CPU
- computing the EXPOSURE metric: log2(|R|) - log2(rank) over a well-defined candidate space
- ranking the true secret's perplexity against every equally-likely candidate
- extracting a memorised secret verbatim by greedy decoding from its prefix
- showing exposure rise with insertion count and with training length
- deduplication as the highest-leverage cheap defence (duplication drives memorisation)
- early stopping / less over-training as a cheap partial defence
- when to reach for DP-SGD as the principled guaranteed fix, and its utility cost
Steps in this lab
- The risk: memorisation is not overfitting, and you cannot see it in the loss
- The metric: what EXPOSURE means and why it beats eyeballing prompts
- Set up: a clean folder, PyTorch and numpy
- Build the shared code: a corpus with a planted canary, a tiny model, and the metrics
- Measure it: exposure rises with insertions, and the secret extracts verbatim
- Defend and re-measure: deduplication and early stopping drive exposure back down
- The judgement: the fix hierarchy, DP-SGD as the guarantee, and the failure modes
- What you just did, and where to push it