Lion Cipher Lab

Race an N-agent debate against one calibrated call on the same ambiguous alerts, count the agents that get talked out of the right answer, and go looking for the difficulty band where the extra rounds pay for themselves

• AI Security
• Advanced
• ~110 min
• Free · no account required

Why it matters

Multi-agent debate is the most seductive idea in agent engineering: spin up several models, let them read each other's answers, argue for a few rounds, and vote. It has a real literature behind it, it feels like how a good incident-review call works, and it is trivially easy to ship into a SOC as 'the ambiguous alerts go to the panel'. What almost nobody does is put it on the same ambiguous alerts as one honest single call and print the two numbers next to the two bills. This lab does exactly that, on a corpus of 600 alerts per difficulty band with a planted answer key, and the result is…

What you'll learn

• building ONE CALIBRATED CALL as a baseline that cannot be beaten by a straw man: a single agent that emits a label AND a probability, with the probability fitted against the answer key on a held-out calibration split - expected calibration error 0.0915 down to 0.0260 at temperature 1.96, Brier 0.1426 down to 0.1354, and not one label moved
• reading a RELIABILITY DIAGRAM in a terminal with no plotting library, and knowing what a Brier score and an expected calibration error each measure and what each one hides that the other shows
• building an N-agent DEBATE from scratch - independent round 1, then rounds where every agent sees the others' labels, confidences and rationales - with the update split into an INFORMATION term and a CONFORMITY term so the two can be measured apart
• counting HERDING with a number instead of a worry: 63 agent-decisions out of 2,000 where an agent held the correct label in round 1 and a wrong majority talked it out of it, against 239 it genuinely rescued, and 50 alerts where somebody on the team knew and the vote threw it away against a 0.9825 best-agent ceiling
• proving the conformity term is a NET NEGATIVE by ablating it - set it to zero and the vote rises from 0.8575 to 0.8700 while herding falls from 63 to 22, so the evidence-sharing is the entire gain and the arguing is the entire cost
• showing DIVERSITY is the engine: five copies of one agent disagree on 0 of 400 alerts and add +0.0000 accuracy for 19.7x the bill, while accuracy tracks evidence COVERAGE (4 channels 0.7800, 20 channels 0.8550) rather than agent count
• beating the RIGHT baseline - ONE WIDE CALL holding all the evidence that fits, not the deliberately starved scoped call that makes every ensemble look clever
• pricing each architecture in accuracy per dollar and in dollars per EXTRA correct decision across five bands, and being able to say out loud that three-round debate costs $13.17, $13.17 and $4.39 per extra right answer on the bands where it wins at all and buys nothing at all on the two where it does not
• the judgment to tell an evidence-sharing protocol from an argument, and to reach for fan-out plus one merge call ($0.22 per extra right answer) rather than a debate ($1.19) when the evidence genuinely will not fit in one window

Steps in this lab

1. The words first: calibrated call, confidence, Brier score, expected calibration error, reliability diagram, round, majority vote, herding
2. Open a terminal and make a scratch folder (all three operating systems)
3. The corpus: 600 ambiguous alerts per band, with the answer key planted
4. The agent is a STUB, and here is its entire error model in writing
5. The scoreboard: Brier score, expected calibration error and a reliability diagram in pure stdlib
6. Architecture 1: one calibrated call - the baseline that cannot be beaten by a straw man
7. Architecture 2: build the debate, and watch one alert get herded
8. The headline: every architecture, every band, one table - and the baseline nobody prints
9. Count the herding, then prove it is a net cost by ablating it
10. Diversity is the whole engine: turn it off and the panel is a photocopier
11. The one honest case for a panel: when the evidence will not fit in any single call
12. The bottom line in money: dollars per EXTRA correct decision, on five bands
13. When NOT to reach for a debate
14. The way a pro does it differently

Open the full interactive lab →