Lion Cipher Lab

Extract ATT&CK techniques from reporting -- and measure whether your mapping is reliable

• Threat Intel Analyst
• Advanced
• ~135 min
• Free · no account required

Why it matters

Every artefact a threat-intelligence function ships downstream -- the coverage heatmap on the wall, the adversary emulation plan, the detection engineering backlog, the slide that tells a board where next year's money goes -- rests on a step nobody measures: a human read a paragraph of English prose and decided which ATT&CK technique ids it demonstrates. That step is treated as though it were a reading off a gauge, and it is not. It is a judgment made under ambiguity by a person with a deadline, and two competent people making it on the same report routinely produce different answers. This…

What you'll learn

• building a labelled CTI corpus with SENTENCE-LEVEL ground truth, and writing into it the seven things that actually make technique mapping hard: the same behaviour in three surface forms, a sentence that supports two techniques equally, a technique implied only by a tool name, a sentence that NAMES a technique while describing something else, vendor marketing, explicit negation, and hedged claims
• writing an ATT&CK technique extractor from scratch in the standard library -- weighted lexicon, regex layer for command-line shapes, hard-negation versus CONTRAST-pivot scoping, hedge downgrading, tool-to-technique mapping, document-section awareness and a tuned score threshold -- and measuring what each layer is worth separately (section awareness alone cut false positives 123 to 9)
• reading a recall failure properly: opening the missed sentences instead of adding words to the lexicon, and telling apart the four distinct causes measured here -- an output shape that cannot represent a two-technique sentence, your own negation scoping eating a true positive, narrative business English with no cue at all, and a genuine lexicon gap (only the last is worth fixing that way)
• implementing Cohen's kappa, Fleiss' kappa and Krippendorff's alpha by hand, including the coincidence-matrix formulation and the (n-1) denominator people get wrong, and being able to argue with somebody who quotes one at you
• why percent agreement is worthless on a sparse label space -- 95.53% of a (160 x 148) mapping grid is empty, so two raters who do nothing agree 95.53% of the time -- demonstrated by putting 97.07% agreement next to kappa 0.586 on identical data, and by a pair whose percent agreement is HIGHER and whose kappa is the lowest in the table
• measuring agreement at three granularities and understanding both results: kappa swings +33.5% simply by collapsing sub-technique to technique, and then FALLS at tactic level because prevalence rises from 3.7% to 44.4% -- the kappa paradox, which means coarsening is not a free win
• quantifying downstream rank instability rather than arguing about it: top-N overlap, Spearman rank correlation and discordant-pair counts, plus why Spearman over a full sparse matrix is a trap that reports stability while the funded part of the backlog is being rewritten
• measuring each remediation instead of asserting it -- a written rubric of executable decision rules (kappa 0.586 to 0.872), a required evidence SPAN with reviewer check (19.6% of one analyst's mappings did not survive), evidence-based adjudication, a confidence field that reveals tool-name evidence is the LEAST reliable class, and ATT&CK data-source mapping that found one mapping in five is unfalsifiable in the given environment
• naming and quantifying five failure modes: double-counting one behaviour as three techniques (8.9% of cells), mapping the report's WORDS instead of the adversary's behaviour (mappings citing a recommendations sentence are 91.1% wrong), circular reporting and single-vendor dominance (47% of the corpus, top-10 backlogs sharing 4 of 10 between corpus halves), ATT&CK version drift (11.6% of an archive silently orphaned across v6 to v7, plus a deprecated technique that SPLITS into two), and the base-rate error of reading 0% coverage on a technique nobody has been hit by
• the judgment: when a mapping programme is worth the effort and when it is theatre, the five questions to demand of anybody else's mapping, the ceiling on agreement that no rubric removes because it lives in the source text, and the honest position that ATT&CK is a communication framework rather than a measurement instrument -- so the error is not using it, it is doing arithmetic on top of it as though the mapping step were a reading off a gauge

Steps in this lab

1. [1] The claim under every heatmap you have ever seen
2. [2] When a mapping programme is worth the effort -- and when it is theatre
3. [3] Set up -- assume nothing. Open a terminal and check Python.
4. [4] The corpus: seven kinds of sentence, and why each one is in there
5. [5] Create mkcorpus.py
6. [6] Run it -- and read the one number that frames the entire lab
7. [7] Look inside a single report before you automate anything
8. [8] What an extractor actually has to do -- the five layers
9. [9] Create extract.py
10. [10] Run it: what each layer of the extractor is actually worth
11. [11] The threshold sweep, and the cliff at 1.2
12. [12] Per-tactic precision and recall -- the aggregate number is hiding a 2x spread
13. [13] Where recall collapses -- and open the missed sentences to find out why
14. [14] The ceiling nobody mentions: one sentence, two techniques
15. [15] Why percent agreement is a lie, and what chance correction actually does
16. [16] The three analyst strategies you are about to simulate
17. [17] Create reliab.py and run it -- what each rater produced
18. [18] How close is each rater to the truth -- and the granularity trick that changes the answer
19. [19] The argument, in one table: percent agreement next to kappa
20. [20] GRANULARITY -- and an honest surprise (the kappa paradox)
21. [21] Where the two analysts systematically diverge
22. [22] The five fixes, and what people get wrong about each
23. [23] Create fixes.py and measure the rubric
24. [24] Evidence spans -- a reviewer opens the sentence and asks one question
25. [25] Adjudication -- and why it must be against the evidence, not against each other
26. [26] Confidence and source grade -- including the thing they do NOT buy you
27. [27] ATT&CK data sources: is the mapping even falsifiable?
28. [28] The mapping is never the deliverable
29. [29] Create damage.py and build the coverage heatmap
30. [30] The headline number: how much the detection backlog moves
31. [31] Failure mode 1 -- DOUBLE COUNTING one behaviour as three techniques
32. [32] Failure mode 2 -- mapping the report's WORDS instead of the adversary's BEHAVIOUR
33. [33] Failure mode 3 -- CIRCULAR REPORTING, and an honest negative result
34. [34] Failure mode 4 -- ATT&CK VERSION DRIFT, simulated end to end
35. [35] Failure mode 5 -- the BASE RATE error, or why 0% coverage means nothing
36. [36] What to demand of somebody else's mapping
37. [37] The honest close -- ATT&CK is a communication framework, not a measurement instrument
38. [38] Clean up, and what to try next

Open the full interactive lab →