Lion Cipher Lab
Infer a training set's global properties with a property inference attack — and defend it (AI privacy)
- AI Security
- Advanced
- ~60 min
- Free · no account required
Why it matters
A trained model is a compressed memory of its training data, and it leaks that memory three different ways. Membership inference asks 'was THIS record in the set?'. Model inversion asks 'show me what a class looks like'. PROPERTY (or distribution) inference asks the question the model owner usually thinks is safe: 'what was true of the dataset as a WHOLE?' — 'was it ~70% group G?', 'was feature X over-represented?', 'what was the mix of sub-populations?'. That is an AGGREGATE fact, not any one person's record, which is exactly why it is dangerous and easy to miss: a hospital can strip every…
What you'll learn
- what property (distribution) inference is, and how it differs from membership inference and model inversion
- the core idea that a model leaks AGGREGATE facts about its training set, not just facts about individual records
- the shadow-model meta-classifier attack: vary a property across many shadow datasets, train a model on each, learn (model-weights -> property)
- turning a trained model into a feature vector the meta-classifier can read (flattened coefficients / a canonical representation)
- measuring attack accuracy against the 50% random baseline over many held-out target models
- seeing WHY a naive deep-net-weight attack sits at chance (neuron permutation symmetry) and why Ganju's permutation-invariant representation exists
- measuring that strong regularization does NOT stop property inference, because the property lives in the useful aggregate signal
- measuring a defense that DOES drop the attack toward 50% (parameter noise) and naming the utility cost it charges
- the judgment: why differential privacy for MEMBERSHIP doesn't automatically stop PROPERTY inference, and when this is a real business/fairness/health leak
Steps in this lab
- The threat: a model leaks facts about the whole dataset, not just records
- Set up: a dataset whose global property you control
- The attack: shadow models + a meta-classifier
- Why a naive deep-net attack fails: permutation symmetry
- Defense that DOESN'T work: strong regularization
- Defense that DOES work — at a price: parameter noise
- Defenses in practice (and why membership/DP isn't a free pass)
- The judgment: when it's a real risk, and how it differs from its cousins