Lion Cipher Lab

Build a resumable workflow engine from scratch, then kill it mid-enrichment and prove it resumes from the failed node instead of re-running your whole IR workflow

• AI Security
• Advanced
• ~110 min
• Free · no account required

Why it matters

Every SOAR platform, every orchestrator, every 'agent framework' with a run history is doing the same three things underneath the marketing: it turns your steps into a directed graph, it decides an order, and it writes each step's output somewhere durable so that when the process dies - and it dies - the next process can pick up where the last one left off. That is the whole product. If you have only ever driven one of those from a UI, the interesting part has been hidden from you, and the day it behaves strangely you will have no model of what it is doing. So build it. The workload is a real…

What you'll learn

• writing a DAG runner from scratch: Kahn's algorithm implemented by hand, returning PARALLEL WAVES rather than a flat order, with a cycle detector that names the actual cycle instead of hanging forever
• cross-checking your own topological sort against graphlib.TopologicalSorter from the Python standard library, so the ordering is verified rather than believed
• node-level CHECKPOINTING to durable storage, and the arithmetic that says whether it was worth it: node executions, simulated tool time and simulated tool spend re-done on resume, with and without checkpoints, at every possible crash point
• CONTENT-ADDRESSED node keys - hashing the node id, its code version, its parameters, its upstream nodes' keys and a digest of every external file it read - so a checkpoint becomes unfindable the moment one of its inputs changes
• surviving three different injected crashes and measuring each: a hard process kill (os._exit, no cleanup, no flush), a torn checkpoint write, and a byte-level corruption that still parses as valid JSON
• the atomic write discipline (write to a temp file, fsync, os.replace) and the measured proof that the in-place alternative destroys 8 checkpoints out of 8 when you kill it mid-write
• a per-node retry policy with attempt limits, and the distinction between a transient failure the engine should retry and a hard failure that should stop the run with every completed node still durable on disk
• finding the CROSSOVER with data: the node duration and crash rate below which checkpointing is pure overhead and the naive re-run wins outright, computed from your own disk's measured fsync cost
• the judgment to say when NOT to build this - the workflows where idempotent re-runs beat durability, where an event-sourced log beats checkpoints, and where the honest answer is a queue and a cron job

Steps in this lab

1. The frame: what a workflow engine actually is, and the four labs this is not
2. Open a terminal and make a folder - the zero-assumption version
3. Build the case file: one alert, six indicators, and two versions of the intel feed
4. The engine, all of it: 336 lines that are the whole product
5. The IR enrichment workflow: two stubs and eleven nodes
6. Prove the ordering: waves, the parallel budget, and a cycle the engine catches
7. Run it for real - and watch it die on a provider that is down
8. Now kill it properly: os._exit in the middle of an enrichment
9. Measure it: kill the run at all eleven nodes and total the damage
10. The honest negative: find the crossover where checkpointing costs more than it saves
11. Two ways a checkpoint file lies: the torn write and the flipped byte
12. The dangerous one: a stale checkpoint that resumes into a confidently wrong incident
13. Why this, not that: checkpoints vs. an event log vs. just being idempotent
14. When you would NOT build this
15. The way a pro does it
16. OPTIONAL appendix: threads for the waves, SQLite for the store

Open the full interactive lab →