Lion Cipher Lab
Measure whether a triage model caves when an analyst pushes back, then fix it with a blind re-judge (sycophancy in alert triage, deterministic STUB model)
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
Every team that puts a language model into alert triage keeps a human in the loop, and the human-in-the-loop story quietly assumes something: when the analyst disagrees with the model, the conversation that follows makes the verdict better. It only does that if the model changes its mind for the right reason. Models trained on human feedback have a measured habit of changing their mind for the wrong one. Ask 'are you sure?' and they apologise and switch, whether or not the first answer was right. That habit is called sycophancy, and it has been measured across several production assistants.…
What you'll learn
- building a keyed sycophancy eval for alert triage: 600 synthetic alerts, a first verdict from a stub model, and one simulated analyst pushback per alert that always argues the opposite label, so every flip can be graded as either a harmful cave or a useful correction
- measuring the two numbers that one overall accuracy hides: the WRONG-FLIP rate (how often the model abandons a correct verdict under pressure) and the USEFUL-CORRECTION acceptance rate (how often it takes a correction it should take)
- proving with a table that authority and tone carry no information about who is right (junior 17.8%, peer 17.9%, lead 22.2% analyst-right, independent by construction) while the fast social reply caves on 25.2% of correct verdicts for a junior and 64.6% for a lead
- showing that thinking harder is not the fix: a deliberate System 2 path that still sees the authority and tone caved on 71 of 484 correct verdicts, against 212 for the fast reply and 4 for the blind re-judge
- building a blind re-judge that strips authority and tone from the pushback and re-scores from the evidence plus any NEW evidence the analyst attached, taking net accuracy from 0.6133 (fast social reply) to 0.9133
- keeping the honest negative: the blind re-judge accepted 0 of the 42 valid corrections that came with no new evidence, so useful-correction acceptance fell from 96/116 to 68/116
- pricing the pro remedy for that negative: keep the re-judge blind and escalate contested, low-margin, no-evidence alerts to a second human, and read the queue-size versus corrections-recovered trade-off off a printed table
Steps in this lab
- The words first: sycophancy, pushback, the two kinds of flip, and the fact that the model here is a stub
- Open a terminal and make a scratch folder (all three operating systems)
- Write the stub triage model: one first verdict, then three ways to answer a pushback
- The model is a STUB, and here is its entire behaviour in writing
- Build the keyed corpus and one pushback per alert, and check that tone carries no information
- Measure before and after: wrong-flip rate, useful-correction acceptance and net accuracy
- Read it honestly: blinding is the fix, more thinking is not, and the blind re-judge refuses every no-evidence correction
- The pro move for the negative: keep the re-judge blind, and escalate contested low-confidence alerts to a second human
- Why this, not that - and when a blind re-judge is the wrong tool
- Ship it: the eval to run against your real model, and what to log