Lion Cipher Lab
Stop LLM output from reaching a shell, path or HTML sink in a response playbook (deterministic stub model)
- AI Security
- Advanced
- ~75 min
- Free · no account required
Why it matters
Response playbooks are starting to let a language model read an alert and fill in the next action: which host to isolate, where to save the evidence, what to show the analyst. Each of those values then lands in a SINK - a shell command, a file path, a web page - and the alert the model read was partly written by the attacker. OWASP lists this as LLM05, Improper Output Handling: the bug is not in the model, it is in code that trusts what the model returns. This lab makes it measurable. A deterministic stub model copies attacker text into its answer the way real models do, and three naive sinks…
What you'll learn
- treating every field a language model returns as untrusted input, and finding the SINK each field lands in: a shell, a file path, an HTML page
- measuring attack success per sink on a seeded 300-alert corpus: before any fix, 31/45 shell, 36/45 path and 39/45 HTML payloads fired
- fixing each sink in its own terms - an argument list with no shell, resolve-and-confine for paths, html.escape for HTML - and proving each falls to 0/45
- pricing a fix with its false rejection rate: a hostname regex rejected 34 of 235 legitimate hosts and a filename allowlist 19 of 264 legitimate names, while confinement and escaping rejected none
- showing that escaping for the WRONG context does nothing (html.escape on a shell argument: still 26/45) and that a types-and-lengths schema changes nothing (300 of 300 passed)
- separating FORM from CONTENT: 34 well-formed swaps to a domain controller all passed the regex, and only an inventory check bound to the alert's attested host stopped them
- deciding when to let a model choose a value at all, versus building the value server-side or putting a human approval gate in front of the action
Steps in this lab
- The words first: LLM05, sinks, untrusted output, the three injections - and the fact that the model here is a stub
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Build 300 alerts: honest ones, three kinds of injected text, and one kind that is perfectly well-formed
- Write the STUB model and read its whole error model: it never invents an attack, it only copies
- Write the three sinks, each the naive way and the safe way
- Measure before and after: attack success per sink, and false rejections on the legitimate outputs
- Read the results: why each fix works, and what each one costs
- The honest negatives: wrong-context escaping, a schema that changes nothing, and a valid hostname the attacker chose
- Why this, not that - and how this lab differs from its neighbours
- When NOT to let the model choose the value at all - and how a pro does it differently
- One experiment, then clean up