Lion Cipher Lab
Reconstruct training data with a model inversion attack (AI privacy)
- AI Security
- Advanced
- ~60 min
- Free · no account required
Why it matters
A trained model is a lossy compression of its training data: to score inputs well it has to memorize structure about what it saw, and that memory leaks through the numbers it hands back. Membership inference asks a yes/no question — 'was THIS record in the training set?'. Model inversion is scarier: it hands you a representative input for a class you never had. Fredrikson et al. (CCS 2015) showed that for a face-recognition model you can start from a blank image and, using nothing but the model's confidence outputs, hill-climb the pixels until a recognizable face of a real person in the…
What you'll learn
- what model inversion is, and how it differs from membership inference and attribute inference
- seeing a model as a lossy compression of its training data whose CONFIDENCE outputs leak what it memorized
- running gradient ascent ON THE INPUT (not the weights) to reconstruct a class prototype
- the real-gradient path (torch autograd on x.requires_grad_) vs a gradient-free / numerical-gradient fallback
- measuring reconstruction quality with cosine similarity to the class mean, and reading it honestly
- observing that attack quality tracks class separability and overfitting (distinctive class = recognizable, overlapping class = mush)
- implementing Fredrikson's own countermeasure: coarsen/round the confidences or return only the top-1 label, and measuring the degradation
- the judgment of when confidences are safe to expose, and the honest limits of the attack
Steps in this lab
- The threat: a model is a leaky compression of its data
- Train the target model (digits, exposes confidences)
- The attack: gradient ascent on the input
- Measure it: cosine similarity, and separability
- The realistic threat surface: a black-box confidence API
- The countermeasure: coarsen the output, watch it collapse
- Defenses in practice (beyond coarsening)
- The judgment: when it's a real risk, and the honest limits