Lion Cipher Lab
Vote across sampled reasoning paths with self-consistency and find where more samples stop helping
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
Self-consistency is one of the cheapest reliability tricks for language models: ask the same question several times with sampling switched on, let each run reason its own way to an answer, and take the majority. It works, and the reason it works is easy to forget - the runs have to make DIFFERENT mistakes. This lab measures both halves on 800 answer-keyed alerts and a stub reasoner whose behaviour is written down. Voting lifted easy alerts from 0.835 to 0.995 and moderate ones from 0.695 to 0.925. But on one family of alerts - remote-management tools with a valid signature, a real abuse…
What you'll learn
- building an 800-alert answer key in which two kinds of error are kept apart: error each reasoning path makes on its own (voting can cancel it) and error every path shares (voting cannot)
- running self-consistency by hand: sample N reasoning paths per alert, take the plurality verdict, and sweep N = 1, 3, 5, 9, 15, 25 with the token bill printed next to the accuracy
- reading a self-consistency curve per group: easy saturated at N = 5 (0.995), moderate kept climbing to 0.925 at N = 25, hard flattened near 0.67
- catching the failure that matters: on a family with a shared bias, more votes made accuracy WORSE (0.390 to 0.345, malicious alerts 0.109 to 0.040) and the wrong answers got confident
- using vote agreement as a free uncertainty signal to escalate alerts to a human, fitting the threshold on one half and scoring it on the other
- seeing why agreement cannot detect a shared bias, and fixing it with a rule instead: coverage went from 0.345 to 0.618 at the same accuracy target
- cutting tokens with adaptive sampling (stop early when the paths already agree) and noticing that it saves the most exactly where the votes are confidently wrong
Steps in this lab
- The words first: reasoning paths, sampling, self-consistency, agreement, and the difference between errors that cancel and errors that do not - and the fact that the reasoner here is a stub
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Build 800 answer-keyed alerts: three difficulty bands and one family with a planted blind spot
- Write the STUB reasoner: one sampled path, a vote, agreement, and the shared bias written down in full
- Sweep N: accuracy and token cost at 1, 3, 5, 9, 15 and 25 sampled paths
- Read the curve: big gains where errors are independent, saturation, and where the knee is
- The honest negative: on the shared-bias family, more votes made it worse - and more confident
- Use agreement as a free uncertainty signal: escalate low-agreement alerts to a human
- Adaptive sampling: stop early when the paths already agree, and see where the savings come from
- Prove where the failure came from: set the shared bias to zero and re-run
- Why this, not that: voting versus one longer chain, a debate, or a different model
- When NOT to use self-consistency
- The way a pro does it differently, and what to ship