Lion Cipher Lab

Put prompting, retrieval and a trained model on one ATT&CK-mapping task with one answer key, score all three on accuracy AND cost AND staleness, and walk away with the rule for which one to reach for and when

• AI Security
• Advanced
• ~130 min
• Free · no account required

Why it matters

Every team that puts a language model near a security workload hits the same fork within a month: do we write a better prompt, do we build retrieval, or do we fine-tune? The literature answers it three separate times, once per strategy, always on a different task with a different scorer, and always on accuracy alone. Accuracy alone is the axis with no denominator, and it picks the wrong strategy almost every time. This lab puts all three on ONE security task - reading an incident-report narrative and labelling it with the ATT&CK technique it describes - with one planted answer key, one…

What you'll learn

• putting all THREE adaptation strategies on ONE security task - mapping incident-report narratives to ATT&CK techniques - with one planted answer key, one scorer and one held-out split, so the comparison is a number rather than a preference
• seeing that the three strategies are the SAME arithmetic with the weights coming from three different places - pretraining, the prompt, the retrieved passage, or gradient descent - and being able to use that as the actual decision question: which source owns the fact you need, and who pays when it changes
• sweeping few-shot exemplars until they stop paying and then hurt: 0.6925 zero-shot, a peak of 0.8875 at six per technique, and 0.8500 at twelve each for 1.87x the bill - plus the label-balance result where an identical twelve-exemplar budget drawn from three techniques instead of twelve scores 0.2200 against 0.7225
• writing a BM25 retriever from scratch in the standard library - inverted index, Robertson-Sparck-Jones IDF, k1 saturation and b length normalisation - and measuring recall@k (0.4250, 0.6075, 0.7975, 0.8725, 0.8850) as the hard ceiling on everything RAG can ever do
• catching RAG failing SILENTLY: on the retrieval-miss band it scores 0.4198 against 0.6543 for the same model asked with no passages at all, and on the 19 reports whose rank-1 passage is the lookalike technique it is wrong every single time at 0.985 stated confidence
• training a real multiclass logistic regression by hand-written gradient descent - softmax, cross-entropy, 1,704 parameters, a real train/validation/test split - reading its overfitting curve (train 1.0000 against validation 0.9900 by epoch 100) and finding the ~125 labelled reports where it overtakes the best prompt
• pricing the decision as a function of QUERY VOLUME rather than as an opinion: up-front $0 / $126 / $168 / $1,400, per-call $1.4422 / $10.9462 / $2.0452 / $0.000001 per thousand, and a break-even at 116,388 queries against few-shot prompting that collapses to DAY ONE if your labels already exist in closed tickets
• measuring STALENESS, the axis nobody scores: injecting a campaign that renames 40 per cent of the vocabulary and brings two techniques that were never in the label set, then measuring the fall (RAG -0.3912, prompting -0.3146, the trained model -0.1871) and the repair curve for each
• knowing what a repair COSTS in the only currency that matters - 14 documents and a 1.2 ms reindex for RAG ($76, back to 0.9208), 84 fresh labels and a permanently longer prompt for few-shot ($147, 0.8438), 84+ labels and a retrain for the trained model - and that until it is retrained the trained model cannot emit a new label at all, at any confidence, for any price
• using a hybrid for the right reason: a trained router bought +0.0000 on in-distribution traffic and 0.9437 on shifted traffic, beating both the frozen model (0.7979) and repaired RAG alone (0.9208), because the frozen model states 0.344 confidence on what it has never seen and 0.657 on what it has

Steps in this lab

1. The words first, and the one idea that makes all three strategies the same shape
2. Open a terminal and make a scratch folder (all three operating systems)
3. The task: 1,880 incident reports mapped to ATT&CK techniques, with the answer key planted and the world moving halfway through
4. The generative model is a STUB, and here is its entire error model in writing
5. Strategy one: prompting, and the point where more exemplars start making it worse
6. Strategy two, part one: write a BM25 retriever from scratch and find the ceiling on everything RAG can ever do
7. Strategy two, part two: RAG end to end, and the failure that does not show up in the output
8. Strategy three: train a real model by hand, and read the honest disclosure that comes with it
9. The third axis: staleness, what it costs each strategy, and what it costs to repair each one
10. The combinations, because in production the answer is almost never one of the three
11. The bakeoff: all three axes in one place, the crossover, and the framework you take away
12. The honest negatives, collected in one place, because they are the reusable part
13. Make it yours: the constants worth changing, what transfers, and what to do next

Open the full interactive lab →