Lion Cipher Lab
Propagate a deadline through an IR agent's tool calls, and measure the work it stops wasting
- AI Security
- Advanced
- ~75 min
- Free · no account required
Why it matters
An analyst asks an AI agent a simple question during an incident: is this host compromised? The agent calls an EDR query, a threat-intel lookup, a SIEM search and maybe a sandbox. Most of the time that takes 15 seconds. Sometimes one tool stalls for a minute or two, because real systems have long tails. If nobody tracks the total time, the answer arrives after the analyst has already moved on, and the agent keeps burning tool time nobody will read. Setting a timeout on each call feels like the fix, but the timeouts add up along the plan, so the whole request is still unbounded. The pattern…
What you'll learn
- explaining why per-call timeouts do not bound a whole request: tight timeouts on four tools still missed a 30-second budget in 3.4% of 2,000 runs, because stage timeouts add up along the plan
- propagating one ABSOLUTE deadline through every tool call (timeout = min(own timeout, time remaining)) and cancelling in-flight calls when it passes: misses fell from 12.7% (naive) to 0.0% and p99 from 107.3 s to 29.5 s
- measuring wasted work as tool-seconds spent after the caller gave up: 8,482 naive, 260 with per-call timeouts, 0 with a propagated deadline
- running asyncio on a virtual clock so 2,000 heavy-tailed runs are deterministic and take about a minute, while still using real tasks, wait_for and cancellation
- building a partial answer that marks unchecked fields 'unknown', and measuring the over-trust it prevents: treating missing as clean added 19 wrong CLEAN verdicts on compromised hosts
- hedging a slow-tail read (send a second SIEM search after a latency percentile, take the first, cancel the other) and reading its real cost curve: +1.2% requests at p95 for 3 wins, +13.9% at p50 for 97
- proving cancellation is cooperative: a blocking call in a thread kept running to 3.2 s after a 1 s deadline, and the fix (a cancel token the work checks) stopped it at about 1.2 s
- choosing where deadlines belong next to retries, circuit breakers, fleet budgets and tracing, and knowing which calls must never be hedged
Steps in this lab
- The question, the budget, and the words you need
- Open a terminal, check Python, make a folder
- Write the simulation: four fake tools, a stub planner, a virtual clock
- Measure BEFORE: no overall deadline, and the work nobody reads
- Tight per-call timeouts: better, and still not bounded
- Propagate ONE absolute deadline, cancel at the deadline, and measure AFTER
- Honest negative 1: a partial answer must say what it did not check
- Hedge the slow-tail SIEM search, and read the cost curve
- Honest negative 3: cancellation only stops code that cooperates
- When NOT to do this, and the way a pro does it