Lion Cipher Lab
Size an audit sample for a control test, then measure what it actually proves
- GRC / Compliance
- Intermediate
- ~60 min
- Free · no account required
Why it matters
Every SOC 2 Type II report, every internal audit, and every ISO 27001 surveillance visit eventually reaches the same moment: someone exports a list of thousands of tickets and says 'pull a sample.' What happens next decides whether the report means anything. Pull the first 25 because they are easy, find nothing, write 'no exceptions noted,' and you may have signed off a control that broke in September. Pull 25 at random, find nothing, and you have proved much less than it sounds like -- only that the failure rate is probably below about 11%, not that it is zero. Ask for 60 and you are better…
What you'll learn
- speaking the vocabulary of control testing the way an internal auditor or SOC 2 readiness lead does: population, sampling unit, attribute, deviation, tolerable deviation rate (TDR), confidence, sampling risk, and upper deviation limit
- writing the test procedure (what counts as a deviation) down as code BEFORE looking at a single sample, so every sample, simulation and full-population test applies the same definition
- computing attribute-sampling sample sizes exactly with the Python standard library: the zero-expected-deviation shortcut ceil(ln(1-conf)/ln(1-TDR)) and the general binomial search, reproducing the classic 25 / 40 / 60 sizes (25 = 90% confidence at a 9% TDR, 59 = 95% at 5%) and the 93 you need if one deviation is allowed
- computing an exact one-sided Clopper-Pearson upper deviation limit by bisection, and reading it honestly: a clean sample of 25 proves only that the rate is below 11.29% at 95% confidence, not that it is zero
- drawing a seeded, documented, reproducible random sample and showing why a convenience sample (the first 25 tickets of the year) cannot reach failures clustered in one team late in the year
- measuring sampling plans by simulation against a known truth: over 10,000 audits each, convenience-25 is wrong 100% of the time, random-25 17.7%, random-40 6.3%, random-60 1.5%, and zero-tolerance plans wrongly fail a working control 33.8% to 62.2% of the time
- knowing when NOT to sample: testing 100% of a complete export with a script answers the question exactly (160 of 2,400 = 6.67%) and shows where the failures live, which no sample of 25 can
- seeing that auditor judgment dominates the math: the same file is 6.67% (ineffective) under a strict deviation definition and 3.96% (effective) under a lenient one
Steps in this lab
- What a control test is -- every term, defined once
- Generate a year of access tickets with a hidden problem
- Write down what counts as a deviation -- before you look
- Compute the sample sizes yourself -- where 25, 40 and 60 come from
- Measure what a clean sample proves -- the upper deviation limit
- Draw a convenience sample and three seeded random samples, and test them
- Run 10,000 audits per plan and count how often each one is wrong
- Skip sampling: test all 2,400 tickets with a script
- The honest negatives -- what sampling cannot do
- When to sample, how big, and how a pro writes it up
- What you built -- and the reflex to keep