Lion Cipher Lab

Calibrate an embedding-similarity threshold for alert de-duplication, and find what no threshold can fix

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Every SOC that adds embeddings to its pipeline ends up writing one line that matters more than the model: if similarity >= SOMETHING, merge these alerts into one incident. SOMETHING is usually 0.8 or 0.85 because a blog post or a library default said so. This lab measures what that costs on 1,157 synthetic alerts with an answer key. The borrowed 0.8 merged 1,791 pairs it should not have and turned 72 incidents into 10 tickets - every single incident hidden inside another. The reason is geometric, not a bug: the embeddings point roughly the same way (every pair scored at least 0.5), so raw…

What you'll learn

• building a labelled PAIR set for de-duplication: every pair of alerts marked same-incident or different-incident, split into a calibration half and a held-out test half by incident, so no threshold ever grades its own homework
• measuring a borrowed default honestly: raw cosine >= 0.8 merged 1,791 different-incident pairs on test (precision 0.213) and collapsed 72 true incidents into 10 tickets
• reading a cosine histogram for ANISOTROPY - every one of 13,125 pairs scored 0.5 or more, so the whole decision lives in a narrow high band
• mean-centering embeddings and measuring what it does and does not change: the gap between the medians doubled (0.233 to 0.478) while AUC barely moved (0.992 to 0.993)
• choosing a threshold by a cost-weighted objective (a wrong merge costs 10, a missed merge 1) on calibration data and scoring it on held-out data
• diagnosing why the optimiser pinned the threshold at 0.960: twin incidents that differ in ONE critical field merge at any threshold that still finds real duplicates, and only a hard field guard fixes them
• fitting one threshold per alert type (test cost 292 to 174) and measuring a month-2 template change that turned a month-1 threshold into 283 false merges

Steps in this lab

1. The words first: de-duplication, embeddings, cosine similarity, thresholds, pairs, precision and recall, anisotropy - and the fact that the embedder here is a stub
2. Where this lab sits next to the embedding labs you may already have done
3. Open a terminal and make a scratch folder (Windows, macOS, Linux)
4. Build 1,157 answer-keyed alerts over two months: three alert types, incidents, and twins
5. Write the STUB embedder and the scoring helpers, and see the problem in three pairs
6. Measure the borrowed default: merge when raw cosine >= 0.8
7. Read the histogram: anisotropy, and why it bites a trigram stub and a transformer alike
8. Fix (a): mean-center the vectors, and measure what it changes and what it does not
9. Fix (b): pick the threshold by COST on the calibration split, score it on held-out test - and watch the optimiser hit a wall
10. The honest negative that tuning cannot fix: add a hard field guard, then fix (c) with one threshold per alert type
11. The honest negative that arrives next month: a new template makes the fitted threshold wrong
12. From pairs to tickets: what each policy does to the incidents an analyst actually sees
13. Why this, not that: a calibrated threshold versus clustering, exact matching, or a bigger model
14. When NOT to de-duplicate by embedding similarity
15. The way a pro does it differently, what to log, and an optional real embedder

Open the full interactive lab →