Lion Cipher Lab

Fix RAG retrieval misses with hybrid search, Reciprocal Rank Fusion and a reranker, measured with recall@k, MRR and nDCG on a SOC runbook corpus

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

A RAG system (retrieval-augmented generation: a language model that first looks things up, then answers) fails silently when the lookup fails. If the runbook chunk for event 4625 never reaches the model, the model writes a fluent answer from memory and nobody can tell from the output. So the retrieval stage deserves its own measurement, and the field already has the tools: a labelled query set and three numbers (recall@k, MRR, nDCG) that information retrieval has used for decades. This lab builds that measurement on a 240-chunk SOC runbook corpus and 60 labelled analyst queries, then improves…

What you'll learn

• building a labelled retrieval test set (queries, the chunks that answer them, and a relevance grade) split into a dev half you tune on and a held-out half you only score at the end
• computing recall@k, MRR and graded nDCG@10 by hand in the Python standard library, and checking the metric code on a ranking you can work out on paper
• measuring BM25 against a dense retriever per query class: BM25 0.996 vs 0.365 nDCG@10 on exact-identifier queries, 0.223 vs 0.904 on paraphrased ones
• seeing why averaging raw BM25 and cosine scores mis-fuses (BM25 runs 0 to 19.5, cosine 0 to 1, so BM25 decides) and how Reciprocal Rank Fusion works on ranks instead
• sweeping the RRF constant k and finding that the default k = 60 buried dense-only answers (one went from #1 to #42), while k = 2 chosen on dev lifted nDCG@10 from 0.730 to 0.833
• adding a second-stage reranker over the top N and reading recall@N as its hard ceiling, its latency bill, and the knee (N = 5 was best; N = 100 cost 18 times more and scored lower)
• catching the honest negatives: hybrid is worse than BM25 alone on identifier queries, and a +0.253 dev gain shrank to +0.056 on held-out queries, half of it traced to dev-tuned synonyms

Steps in this lab

1. The words first: retrieval, chunks, qrels, recall@k, MRR, nDCG, BM25, dense retrieval, fusion, RRF, rerankers - and the fact that two of the models here are stubs
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build the corpus and the qrels: 40 runbook chunks, 200 filler chunks, 60 graded queries in three classes
4. Write the retrievers and the metrics: BM25 in one step, the dense STUB with its error model, fusion, and recall@k, MRR and nDCG checked on paper
5. Stage 1, measured: BM25 alone against the dense stub alone, per query class
6. Stage 2: fuse the two lists - why averaging raw scores mis-fuses, what min-max does, and what RRF does instead
7. Stage 3: rerank the top N - the latency bill, the knee, and recall@N as a ceiling no reranker can break
8. The held-out scorecard: what survives queries you never tuned on
9. Prove where the dev gain came from: switch off the dev-tuned synonyms and re-run
10. Why this, not that: retrieval metrics versus answer metrics, RRF versus score fusion, a reranker versus a better embedding model
11. When NOT to use hybrid search and reranking
12. The way a pro does it differently, and what to ship

Open the full interactive lab →