Lion Cipher Lab
Decide when an incident-response agent should ask a clarifying question instead of acting: score ambiguity x blast radius, sweep the threshold, and measure when asking stops paying
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
The moment an AI agent can isolate a host, block an IP or reset a password, its most dangerous mistake is not a wrong answer - it is a right action on the wrong target. Tickets are written by tired humans: 'isolate web-lon-0' matches three servers, 'reset rlee' matches a real account that is not the one they meant, and the asset inventory is a week behind reality. The two lazy fixes both fail. Letting the agent act on its best guess took a healthy production server offline 29 times in this lab's 400 tickets. Making it ask whenever there is any doubt cut wrong actions from 88 to 27 but…
What you'll learn
- building a deterministic stub incident-response agent in pure standard-library Python that resolves a ticket's free-text target into scored candidates (difflib) and whose four tools - isolate_host, reset_user, block_ip, tag_ticket - only flip fields on a simulated in-memory inventory
- scoring AMBIGUITY from the candidate list (how many look-alikes, and how small the lead of the top one is) and multiplying it by a per-action BLAST RADIUS weight so the same doubt triggers a question for a server isolation and not for a ticket tag
- measuring before and after on 400 answer-keyed tickets: always-act made 88 wrong-target actions (52.2 blast-weighted harm); ask-when-costly at 0.35 made 45 (20.0 harm) with 65 questions, against always-ask's 27 wrong (16.7 harm) for 135 questions
- sweeping the ask threshold from 0.00 to 0.70 and reading the trade-off curve - harm stays near the always-ask floor down to about 0.40, then climbs to 31.2 at 0.50 and 42.2 at 0.70 as the questions dry up
- pricing questions and wrong actions in one currency and showing that with a 20%-wrong, 30-minute analyst, asking pays only for host isolation, flips 14 correct tickets to wrong, and pushes the cheapest threshold from 0.35 to 0.50
- proving that a clarifying question cannot fix what the agent does not know it does not know: 27 wrong actions (23 stale inventory, 4 exact-name collisions such as rlee vs rlee2) survive every policy, including always-ask
- deciding when NOT to ask: reversible, low-blast actions, and any question whose answerer is less accurate than the agent's own top pick
Steps in this lab
- The words first: ticket, target, inventory, candidate, ambiguity, blast radius - and the fact that the agent is a stub
- Open a terminal and make a scratch folder (all three operating systems)
- Build the simulated world: 191 inventory records and 400 tickets with the right answer planted in each
- The agent stub, all of it: resolve, score, decide, act - and the analyst it asks
- Before and after: always-act, always-ask, and ask-when-costly on the same 400 tickets
- Sweep the threshold and read the trade-off curve
- Honest negative 1: a slow, fallible analyst makes asking a loss for everything but isolation
- Honest negative 2: the wrong actions a question can never reach
- When NOT to ask - and why this, not that
- The way a pro does it
- Ship it: the rule, what to log, and clean up