Lion Cipher Lab
Attack a model's privacy with a membership inference attack
- AI Security
- Intermediate
- ~90 min
- Free · no account required
Why it matters
The standard reassurance about machine learning is that a trained model is not a copy of its training data — it is 'just weights', 'just patterns', the data is 'gone'. In this lab you will personally disprove that. You will train a model on a set of records, then, using nothing but the answers it gives to ordinary queries, work out which records it was trained on — correctly about three times out of four, against a 50% coin flip. That capability has a name in the research literature, membership inference, and it matters because membership is often the secret. A model trained on the records of…
What you'll learn
- membership inference attacks
- what overfitting leaks about training data
- reading a model's confidence as a side channel
- shadow models
- measuring an attack with accuracy, true/false positive rates and ROC AUC
- scikit-learn basics (make_classification, train_test_split, RandomForestClassifier)
- regularisation as a privacy control
- differential privacy (concept) and what it costs
- why 'the model does not store the data' is not a privacy argument
Steps in this lab
- What you are about to do, in plain language
- Five words you need before you touch anything
- The ethics line — read this before you run anything
- Open a terminal and install Python
- Make a project folder and a virtual environment
- Install scikit-learn — one package, no GPU, no downloads later
- Build the victim: a model that memorises
- Run the attack — three ways to ask 'were you trained on this?'
- If it did not work: the four things that go wrong
- You cheated — now do it properly with a shadow model
- Interrogate individual records — the part that should bother you
- Why it works: overfitting IS the leak
- The defence: turn down the memorisation, and count the cost
- The trap in your own results: averages hide the worst case
- Beyond regularisation: differential privacy and what it actually buys
- Where this bites in the real world
- What you just did