Lion Cipher Lab

Tune a triage prompt on a dev split, then measure the optimism a sealed, hash-committed test set reveals (deterministic stub model)

• AI Security
• Advanced
• ~100 min
• Free · no account required

Why it matters

Here is a loop every team that ships an LLM feature runs. The triage prompt scores 0.70 on the eval. Someone tries a new instruction wording: 0.71. Another few-shot set: 0.69. A stricter verdict threshold: 0.72. Eighty tries later, one variant scores 0.765 and goes into the release notes as '+6.5 points'. Nothing was faked. But the 0.765 is the MAXIMUM of eighty noisy measurements on the same 200 alerts, and the maximum of noisy numbers is high partly because it was lucky. In this lab that variant's real improvement is +1.5 points, and on a sealed test it scores below the baseline. Cawley and…

What you'll learn

• building an answer-keyed alert-triage eval split into a 200-item dev set you tune on freely and a 400-item test set you seal, with no alert in both
• committing before you look: writing the SHA-256 of the test file and the one variant you will test into commit.json, and refusing to score if the file changed
• running a single preregistered paired comparison on the sealed test, with the decision rule written down before the number exists
• measuring the winner's curse: how far the best-of-K dev score sits above the same variant's test score as K grows from 1 to 80, over 200 fresh splits
• showing that when every variant is truly identical the tuning loop still produces a 'winner' that beats the baseline on dev - and a paired test that calls it significant 41.5% of the time
• implementing Thresholdout, the reusable holdout of Dwork et al., and measuring where it cuts the optimism (a vote built from many holdout looks) and where it does not (picking one best variant)
• quantifying what a small sealed test cannot do (confirm a real 1.5-point win 10.5% of the time) and how fast a test set turns into a dev set once you peek at it

Steps in this lab

1. The words first: dev split, sealed test, the winner's curse, adaptive overfitting, a hash commitment - and the fact that the model is a stub
2. Open a terminal, check Python, make a scratch folder (Windows, macOS, Linux)
3. Where this lab sits: the bootstrap lab, the prompt-format lab and the contamination lab each touch this; here is the boundary
4. Write the eval and the STUB model: 80 prompt variants, a small true effect each, and fixed per-alert luck
5. Split once: a dev file you may tune on, and a test file you seal and fingerprint
6. Tune on dev the way everyone does, and watch the best-of-80 score run away from the truth
7. Commit before you look: hash the sealed test, freeze the one choice, write down the rule
8. Unseal once: the single preregistered comparison, and a log that makes a second look visible
9. One split is an anecdote: measure the winner's curse over 200 fresh splits, then with every variant truly equal
10. Reuse a holdout many times: raw, and through Thresholdout's noisy, thresholded answers
11. Two more honest negatives: a 400-alert test is too small for a real small win, and every peek turns the test back into a dev set
12. Why this, not that - and when NOT to bother with a sealed test
13. The way a pro does it: budget the looks, write the rule first, report the number you did not choose by
14. Ship it: re-run, change one constant to see what is the stub and what is the method, clean up

Open the full interactive lab →