Lion Cipher Lab
Evaluate an LLM with a rubric and a test set
- AI Security
- Intermediate
- ~60 min
- Free · no account required
Why it matters
Ask most people whether an AI model is any good and you get a shrug and a vibe: 'it seems smart', 'it felt off today'. A vibe cannot be compared, cannot be tracked over time, and cannot tell you whether the change you just made helped or hurt -- so teams that ship AI on vibes are flying blind, and they find out their model got worse from angry users instead of from a number. Evaluation is the fix, and it is not exotic: you write down a fixed set of test prompts, you write down what a good answer to each one must contain or do, you run the model, and you score every answer against those…
What you'll learn
- what evaluation ('eval') means for a language model and why 'it seems good' is not a measurement
- writing a test set: a fixed list of prompts each paired with the criteria a good answer must meet
- the idea of ground truth -- the known-correct answer you score an output against
- scoring three dimensions that matter in production: correctness, refusal-when-appropriate, and format
- writing simple deterministic scoring rules in Python (substring checks, refusal detection, JSON/format checks)
- calling a local model (Ollama) from Python once per test case
- computing a pass rate and printing a clear PASS/FAIL score table
- measuring safety objectively: testing that a model DECLINES requests it should decline, with no real harm produced
- regression testing: re-running the same test set after a change to prove a model got better or worse, not just different
Steps in this lab
- The problem: 'it seems good' is not a measurement
- The words you need, defined once
- Install the stack: Ollama, a small model, and one Python library
- Write your test set: prompts paired with expected criteria
- Call the local model once per test case
- Score correctness against ground truth
- Score refusal-when-appropriate: measuring safety objectively
- Score format: does the output have the shape you asked for
- Run the whole harness and print the pass-rate table
- Catch a regression: change the model and re-run the same test set
- What you built, its limits, and where it goes next