Lion Cipher Lab
Calibrate an LLM judge against human labels (and expose its biases)
- AI Security
- Advanced
- ~75 min
- Free · no account required
Why it matters
The moment your app calls an LLM, you need to grade LLM output at a scale no human can keep up with — so you reach for an LLM-as-a-judge: a strong model that reads a question and one or two answers and returns a verdict. It is the backbone of modern eval harnesses, A/B model comparisons, and RLHF-style preference data. And it is trusted almost entirely on faith. The standard failure is to wire up a judge, read a single 'model B wins 61% of the time' number, and ship — without ever checking whether the judge agrees with a human, or whether that 61% is an artifact of the judge's own biases.…
What you'll learn
- building an LLM-as-a-judge as a pairwise preference scorer (question + answer A + answer B -> verdict)
- why agreement with humans, not vibes, is the only thing that makes a judge usable
- measuring judge/human agreement with Cohen's kappa (sklearn.metrics.cohen_kappa_score) and why kappa, not raw accuracy, is the right metric
- reproducing the three classic judge failures from Zheng et al. (MT-Bench): POSITION bias, VERBOSITY bias, SELF-ENHANCEMENT bias — each as a measured number
- position-flip rate: scoring both A/B orders and counting how often the verdict flips
- debiasing: order-averaging with an agreement requirement, a length-controlled rubric, and a judge from a different model family than the one under test
- re-measuring after debiasing and reading the improvement honestly (kappa up, flip rate down)
- the judgment of when an LLM judge is trustworthy versus when a human must stay in the loop
Steps in this lab
- The threat: an unvalidated judge is a made-up number
- Build the labeled gold set (humans decide, once)
- Build the judge (real via Ollama, or a deterministic stub)
- Measure agreement with Cohen's kappa (not accuracy)
- Expose POSITION bias: score both orders, count flips
- Expose VERBOSITY bias: does longer just win?
- Expose SELF-ENHANCEMENT bias: does it prefer its own kind?
- Debias: order-average, length-control, cross-family — re-measure
- The judgment: when to trust a judge, when to keep a human