Lion Cipher Lab

Cut a triage agent loop's cost with prefix caching by ordering its context so the cache actually hits (STUB serving engine, measured)

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

An agent loop is expensive in a way a single chat is not: a triage agent working one alert makes six, ten, twenty calls, and every call re-sends the same system prompt, the same tool definitions, the same playbook and everything said so far. Serving engines can skip recomputing a prompt prefix they have already seen - vLLM, SGLang, llama.cpp and the hosted APIs all do some form of it - but only an EXACT prefix, so whether you get the discount depends on something as mundane as where you put the timestamp. This lab builds a stub serving engine that caches the way vLLM documents it, runs a…

What you'll learn

• explaining prefix caching from the inside: KV blocks keyed by a hash chain of parent key plus block tokens, full blocks only, reuse stops at the first block that differs
• building a pure-Python STUB serving engine with a block-hashed prefix cache, an LRU eviction policy and an illustrative cost and latency model, and grading it against a brute-force longest-common-prefix answer key (144/144 calls agree for every layout)
• measuring five context layouts for a SOC triage agent loop: naive 0.0% cached and 298.296 cost units against cache-friendly 93.3% cached and 62.294 units (79.1% lower), with mean latency 1380.6 -> 688.6 ms
• showing that ONE dynamic token at the top of the prompt (a timestamp) drops the cached share to 0.0%, the same as the fully naive layout
• finding cache-killers without any cache by diffing two consecutive prompts and reading off the first differing token
• weighing context compaction against caching: re-summarizing every call cut input tokens 29% yet raised cost from 78.91 to 111.17 units, while compacting in batches of 8 brought it down to 65.35
• spotting the limits: output tokens and decode time are untouched, a cache smaller than one prompt thrashes to 0.0%, and a shared cache is a timing side channel that a per-tenant salt closes

Steps in this lab

1. The words first: tokens, prefill, the KV cache, prefix caching, and the fact that the engine here is a STUB
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build the STUB serving engine: block-hashed prefix cache, LRU eviction, illustrative bill
4. Build the STUB triage agent and its five context layouts
5. Measure all five layouts and grade the simulator against an answer key
6. Read the table: one dynamic token on top costs everything, and why the middle rows land where they do
7. The pro check: find the cache-killer without a cache by diffing two consecutive prompts
8. Honest negative 1: caching never touches output tokens, so latency only halved
9. Honest negative 2: compacting the history every call saved tokens and cost more
10. Honest negative 3: under concurrency a small cache evicts what it was about to reuse
11. The security angle: a shared prefix cache is a timing side channel, and a salt closes it
12. OPTIONAL appendix: watch a free local model skip prompt work with Ollama
13. Ship it: the layout policy, what to log, and one experiment to run yourself

Open the full interactive lab →