Lion Cipher Lab
Measure lost-in-the-middle recall for an evidence-reading agent, then fix it by reordering
- AI Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
An incident-triage agent is handed a long evidence bundle - hundreds of log lines - and asked to find the one that matters: a single malicious IOC, a C2 beacon to a specific IP, a suspicious parent-child process. Liu and colleagues showed in 2023 that a long-context reader does not weigh every line equally: it recalls a fact at the START or END of its context far better than one in the MIDDLE, a U-shaped curve. So the fact can be sitting right there in the context and the agent still misses it - not a retrieval failure, an attention failure. This lab builds a deterministic stub whose recall…
What you'll learn
- explaining the 'lost in the middle' failure (Liu et al., 2023): a long-context reader recalls a fact at the START and END of its context far better than one in the MIDDLE, a U-shaped recall curve
- building a deterministic stub reader whose recall is a written function of a target's POSITION and the context length, so the documented U-curve can be reproduced and tested offline
- sweeping a single malicious IOC across every position in an evidence bundle and MEASURING recall by position by counting hits, not by reading the constant
- applying and measuring two mitigations - reorder the salient evidence to the ends, and chunk-and-rerank so each chunk is too short to have a lost middle - and reading the before/after numbers honestly
- seeing the circularity in reordering (it only helps if you can already tell which line is salient) and the reranker dependency in chunking (a missed chunk is recall zero) as MEASURED gaps, not warnings
- knowing the context length past which even moving a fact to the best position leaves recall degraded, so a lost middle is a budget to design around, not a bug to patch away
Steps in this lab
- The words first: long context, recall, a needle in a haystack, the U-shaped lost-middle curve, salience, reorder, chunk and rerank - and the fact that the reader here is a stub
- Where this sits: three neighbouring AI-agent labs, and what only this one measures
- Open a terminal and make a scratch folder (Windows, macOS, Linux)
- Write the stub reader: its whole positional-recall model, written down, and what context length does to the middle
- Measure the recall-by-position curve by sweeping the target across every position
- Apply and measure the two mitigations - and measure their honest negatives in the same run
- The honest negatives, stated plainly - why each fix has a catch you must design around
- When to reach for which fix, and how a pro treats a lost middle
- What you built, and the one sentence to keep