Lion Cipher Lab
Rewrite tool error messages so an IR agent can recover from them, and measure the recovery rate
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
An incident-response agent spends much of its life getting tool calls slightly wrong. It passes 'ws017' where the tool wants 'ws017.corp.example'. It sends a date as 10/02/2026 14:00, or a timestamp in milliseconds where the tool expected something else. It calls the field 'hostname' when the tool calls it 'fqdn'. What happens next depends almost entirely on one thing you control: the text of the error message. If the tool says 'Error 400: invalid input', the agent has nothing to go on, so it retries the same call, guesses, or gives up, and a containment action that should have taken two…
What you'll learn
- building 100 answer-keyed incident-response tasks over four pretend SOC tools, where most first calls are wrong in the ways real agents get them wrong: a short hostname, a date in the wrong format, epoch milliseconds, a wrong field name, a forgotten argument, a one-letter host typo
- writing the same validation failure three ways - a terse 'Error 400', an actionable RFC 9457 Problem Details body (which field, why, what was expected, a valid example, whether a retry can help), and the same plus 'did you mean' - and measuring completion, retries, failed calls and give-ups for each
- writing a deterministic stub agent under one stated rule, that it can only fix what the error text tells it, so the error message is the only variable
- measuring the wrong-target rate of 'did you mean' on near-name hosts (dc01, dc10, dc11) and fixing it for destructive tools with exact-match-only plus an independent verification lookup
- treating error text as an attack surface: measuring internal-detail leaks (CWE-209) and attacker input echoed back into an agent's context (indirect prompt injection), and pricing a sanitizer that keeps the messages actionable
Steps in this lab
- The words first: tool call, error style, Problem Details, FQDN, RFC 3339, did-you-mean, prompt injection - and the fact that the agent is a stub
- Open a terminal, check Python, make a scratch folder (Windows, macOS, Linux)
- Build 100 keyed tasks: what the agent should send, and what it actually sends first
- Write the gateway: four tools, one validator, and three error styles for the same failure
- Write the stub agent under one rule: it can only fix what the error text tells it
- Measure the recovery rate: completions, retries, failed calls and give-ups per error style
- Honest negative 1: 'did you mean' sends the agent to the wrong host, so a destructive tool must require an exact match
- Honest negative 2: error text is an attack surface - leaked internals and echoed attacker input - and what a sanitizer costs
- Why this, not that - and when NOT to make errors more helpful
- The way a pro does it differently
- Ship it: the checklist, what to log, and how to see what the stub is doing