Lion Cipher Lab
Audit your security eval's gold labels for noise before trusting a score: Cohen's kappa by hand, a planted rule dispute, and a model ranking that flips (deterministic STUB models)
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
Every security eval score - phishing-detection accuracy, alert-triage accuracy, the number in a vendor's slide - is a comparison against an answer key that humans wrote, and humans disagree. If the answer key is wrong on one item in six, a perfect model cannot score above about 0.83, and a model that happens to share the labeller's blind spot can score higher than a perfect one. This lab makes that concrete on 300 phishing and alert items with two simulated annotators whose mistakes are planted, so you can check each audit step against the truth. Their raw agreement looked respectable at…
What you'll learn
- building a 300-item phishing and alert eval with two simulated human annotators whose mistakes are planted, so the true answer key is known and every audit step can be checked
- computing raw agreement, chance agreement and Cohen's kappa by hand from a 2x2 table (0.810, 0.495, 0.624) and cross-checking it against scikit-learn
- reading disputes as patterns instead of noise: 25 of 30 sanctioned-simulation items disputed with annotator A on the malicious side every time, which is a guideline gap and not carelessness
- measuring the accuracy ceiling a noisy answer key imposes: a perfect model scored only 0.830 against the shipped labels, and a model that copied the annotator's habit scored 0.870 - above perfect
- showing a model ranking flip: X beat Y by 0.083 on the shipped labels, Y beat X by 0.070 on the adjudicated labels and by 0.077 on the planted truth
- adjudicating properly - clarify the rule first and apply it to every item it covers, then send the remaining disputes to a third annotator - which cut wrong labels from 51 to 9
- knowing why dropping every disputed item is the wrong shortcut: it inflated X from 0.827 to 0.922 and still crowned the wrong model
Steps in this lab
- The words first: gold labels, annotators, agreement, Cohen's kappa, adjudication, the accuracy ceiling - and the fact that the models here are stubs
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Build the 300-item eval and plant two imperfect annotators
- Measure agreement by hand: the 2x2 table, raw agreement, chance agreement, Cohen's kappa
- Read the disputes as patterns, not as noise
- Score two STUB models against the labels as shipped - and against the truth
- The honest negative: just dropping the disputed items makes it worse
- Adjudicate properly: fix the rule first, then a third label for what is left, and re-measure
- The judgment: when to audit the answer key, when not to, and how a pro does it better
- Ship it: the checklist, one experiment to run yourself, and an optional local model