Lion Cipher Lab

Supervise an agent team with reassignment and a circuit breaker — spot the stuck worker, move its work, and measure what an over-eager supervisor costs

• AI Security
• Intermediate
• ~90 min
• Free · no account required

Why it matters

The way an agent team fails in production is not the way you plan for. Nothing crashes. No exception is thrown. Your dashboard shows six workers at 100% utilisation and a queue that is not moving. One worker went silent forty minutes ago and has emitted nothing since — no beat, no error, no tokens — and it is still holding a slot. Another is calling the same threat-intel lookup with the same indicator for the four hundredth time, at full speed, spending money, learning nothing. A third has flipped between 'malicious' and 'benign' eleven times. A fourth handed back a confident, well-formatted…

What you'll learn

• building the four LIVENESS SIGNALS an agent team actually needs — a heartbeat timeout, a per-task deadline, a no-progress detector and a repeated-tool-call detector — plus an oscillation detector, and timing each one ALONE against each stuck mode instead of assuming the one you built is the one that fires
• reading a detector-versus-failure-mode matrix and understanding why every signal is blind to three of the four: heartbeat catches a hang in 61s and NEVER catches a loop, repeated-tool catches a loop in 15s and never catches anything else, and the per-task deadline catches everything ten times too slowly
• recognising the failure liveness monitoring cannot see at all — a fast, confident, well-formed verdict citing zero evidence — and adding the output contract that catches it, while being clear that a contract charges you for the whole task before it fires
• implementing REASSIGNMENT with a per-task retry budget and a dead-letter queue: carrying the partial case file to a different worker, capping attempts so a poison task cannot wedge the whole team in turn, and measuring the handoff's real cost rather than assuming it is free
• designing for AT-LEAST-ONCE from day one — measuring the 15 duplicate tickets your own supervisor creates the moment it can retry work, and knowing that the fix is idempotency keys rather than care
• writing a TEAM-LEVEL CIRCUIT BREAKER with all three states, a documented trip condition (rolling window, failure-rate threshold, minimum sample count) and an exponential cooldown — then measuring the recovery overshoot, when the breaker keeps refusing a dependency that recovered 1,745 seconds ago
• proving why HALF-OPEN is not optional by shipping the version without it and counting the difference: 86 tasks fed into a dead API against 24, and 9 alerts lost against 3
• tuning a supervisor from a measured distribution rather than intuition — sweeping the stuck-detection threshold to find the exact band where the supervisor starts executing healthy workers, and pricing that band in duplicated tokens, duplicate side effects and alerts lost
• the judgment to know when a supervisor is the wrong answer, including a measured two-task run where the supervisor 'wins' and is still not worth building

Steps in this lab

1. The failure this lab is about: a worker that is busy, expensive and permanently stuck
2. Open a terminal, check Python, make a scratch folder
3. Build the shift and the workers -- and read the paragraph about stubs before you paste anything
4. The unsupervised baseline: six workers, sixty alerts, nobody watching
5. Five liveness signals, timed one at a time -- and the failure none of them can see
6. Reassignment: take the work off, carry the case file, hand it to somebody else
7. The team circuit breaker: closed, open, half-open, and the cost of coming back too late
8. The honest negative: the supervisor that tried too hard
9. When NOT to build a supervisor -- and why a threshold, not a smarter judge
10. The way a pro does it

Open the full interactive lab →