Lion Cipher Lab
Build your own threat-intel collection pipeline in Python
- Threat Intel Analyst
- Intermediate
- ~90 min
- Free · no account required
Why it matters
Following threat feeds by hand works right up until it doesn't. Six sources, checked properly, is forty-five minutes a day of opening tabs, re-reading the same three headlines you already saw yesterday, and copying indicators into a note — and the first busy week you skip it, then you never restart. A hundred and twenty lines of Python fixes that permanently: it fetches your sources on a schedule, remembers what it has already shown you, pulls the indicators out of the text, ranks each item against the intelligence requirements you actually wrote down, and leaves a short Markdown brief on…
What you'll learn
- building a config-driven collection pipeline in Python
- Python virtual environments (venv) and pip
- polite HTTP with requests: User-Agent, timeouts, error handling
- conditional requests (ETag / If-None-Match / If-Modified-Since)
- parsing RSS and Atom feeds with feedparser
- normalizing many sources into one schema
- deduplicating with SQLite and a stable hash key
- regex IOC extraction (IPv4, domains, URLs, MD5/SHA-1/SHA-256, CVE)
- refanging defanged indicators
- filtering indicator false positives
Steps in this lab
- What you are building, and why it is the next rung
- First, the judgment: when automating is leverage and when it is a hobby
- Set up: a folder, Python, and a virtual environment
- Write the collection plan down as config, not as code
- Start the script: imports and paths that survive being run from anywhere
- Memory: a SQLite table and a stable deduplication key
- Fetch politely — and do not re-download what has not changed
- Normalize: force every source into one shape
- Extract indicators — but refang first, or you will find nothing
- Be suspicious of your own regex: the three filters that make the output usable
- Score against your PIRs — crude ranking, not intelligence
- Emit a brief a human will actually read, then run the whole thing
- Run it a second time — this is the moment it becomes a daily tool
- Now read your brief critically — your ranking is probably wrong
- Say it out loud: NEVER auto-ingest these indicators into a blocklist
- Schedule it: cron on Linux/macOS, Task Scheduler on Windows
- Build or buy: when this script is right, and when it is MISP or OpenCTI
- What you own now