Lion Cipher Lab

Catch fabricated and misattributed CVE IDs in LLM threat summaries (synthetic CVE and KEV snapshot, deterministic stub summariser)

• AI Security
• Intermediate
• ~75 min
• Free · no account required

Why it matters

Analysts now ask language models to summarise advisories, and the summary comes back studded with CVE IDs that look exactly right. Some are not. A model can invent an ID that has the perfect shape and points at nothing, attach a real ID to the wrong product, or say a bug is exploited in the wild when CISA's catalog says no. OWASP lists this as LLM09, Misinformation: confident, credible-looking output that is false. The damage is concrete - a patch ticket for the wrong product, a missed real bug, an executive briefing that cites a CVE nobody can find. The obvious defence, 'check the ID looks…

What you'll learn

• naming the three ways an LLM gets a CVE claim wrong - a FABRICATED ID that exists nowhere, a MISATTRIBUTED ID that is real but belongs to another product, and a wrong 'exploited in the wild' (KEV) claim - and why each needs a different check
• building an offline snapshot shaped like real feeds: CVE records with CVE JSON 5 field names (cveMetadata.cveId, affected vendor/product) and KEV entries with the CISA feed's names (cveID, vendorProject, product, dateAdded)
• measuring three gates against an answer key: the format regex caught 4 of 47 errors and 0 of 15 fabricated IDs; the existence lookup caught all 15 fabricated and 0 of 10 misattributed; the claim-level check caught 38 of 47
• measuring name aliasing instead of ignoring it: raw string compare made 25 false product mismatches, normalising names cut it to 3, a two-entry rename table to 0
• showing snapshot staleness in your own output: 10 correct claims about CVEs published after the snapshot were flagged exactly like the 15 fabricated ones, and 2 correct 'exploited' claims were flagged because CISA added them to KEV later
• replacing a yes/no gate with PASS / BLOCK / RECHECK so a stale snapshot sends real-but-new claims to a live lookup instead of calling them lies: BLOCK held 29 claims and none were correct
• knowing what a claim check cannot see: 9 of 10 wrong-severity claims passed every gate, because the ID, the product and the KEV status were all right

Steps in this lab

1. The words first: CVE IDs, CVE records, KEV, and the three ways a summary gets them wrong - and the fact that the model and the data here are made up
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build a synthetic CVE world, a snapshot frozen on 2026-06-30, and 150 advisories
4. Write the STUB summariser and read its whole error model
5. Write the three gates and run them: format, existence, then the whole claim
6. v3 checks the claim, not the ID - and name aliasing nearly sinks it until you measure and fix it
7. The honest negative: a stale snapshot calls new, real CVEs fabricated - so give the gate a third answer
8. The second honest negative: a right ID, the right product and a wrong severity sail straight through
9. OPTIONAL, online: re-check a REAL CVE against the live KEV feed and the NVD API (read-only, no key)
10. Why this, not that - when NOT to use this gate, and how it differs from its neighbours
11. The way a pro does it differently - and one experiment that proves where the false flags came from

Open the full interactive lab →