Lion Cipher Lab
Write a strategic threat forecast you can be held to: base rates, calibrated probability language, and a Brier score that grades your own analysis
- Threat Intel Analyst
- Advanced
- ~90 min
- Free · no account required
Why it matters
A CISO planning next year's budget asks one question: should we fund this? Most cyber threat intelligence teams answer it with a sentence like 'ransomware remains a significant threat and edge devices are increasingly targeted.' That sentence cannot be acted on, cannot be wrong, and therefore cannot be graded — which is precisely why it gets written. It is intelligence-shaped text with no evidence and no accountability behind it. This lab replaces it with something a leader can actually spend money against: a numbered probability, a fixed time window, a resolution test written down in…
What you'll learn
- separating tactical, operational and strategic intelligence — and knowing which one a decision actually needs
- computing a real base rate from the CISA KEV catalog with a short standard-library Python script
- stating what a base rate does and does NOT license you to claim, including the bias its collection process bakes in
- using the ODNI ICD 203 probability ladder so a word and a number always travel together
- keeping PROBABILITY (of the event) and CONFIDENCE (in your sourcing and reasoning) on two separate axes
- bridging a base rate to a probability for a bounded question with a Poisson tail — and naming the assumption that bridge makes
- registering a forecast with a resolution date and frozen, mechanical resolution criteria before the outcome is known
- writing and running a Brier-score harness with the Murphy decomposition and a calibration table
- beating a reference forecaster — and accepting that a forecast which cannot is adding no value
- reading your own calibration table for a personal bias signature and correcting for it
Steps in this lab
- Three levels of intelligence — and the one almost nobody actually produces
- The three legs a defensible forecast stands on
- Get the data: verify the live CISA KEV feed, then lay down the offline snapshot
- Compute the base rates — and see counts and proportions behave differently
- What a base rate licenses you to say — and what its collection process quietly does to it
- Bridge the base rate to a probability with a Poisson tail — and name the assumption
- Why 'likely' is not an analytic judgement: Kent's problem and the ICD 203 ladder
- Rewrite three vague sentences into calibrated ones (do this by hand)
- Probability and confidence are two different axes — never collapse them
- Register the forecast before you know: a falsifiability linter
- Build a year of resolved forecasts — the register you will be graded on
- Write the Brier harness and run it: the score and its decomposition
- Read the calibration table for your own bias signature — then correct it
- Beat the reference forecaster, or you added nothing
- Write the product: a worked strategic assessment, then one of your own
- The six named failure modes
- When this is worth the effort, when it is not, and how a pro does it better
- The judgement: what being scored does to analysis