Lion Cipher Lab

Add a Reflexion self-critique loop to a triage agent: let it reflect on its own failed run and retry, then measure what it recovers and what it wrecks

• AI Security
• Advanced
• ~110 min
• Free · no account required

Why it matters

A Reflexion loop is the most seductive upgrade you can bolt onto an agent: let it look at its own failed run, write down what it thinks went wrong, and try again. Shinn and colleagues showed it lifts real agents on real benchmarks, and every team that reads the paper wants it in their triage pipeline by Friday. The trouble is that a self-critique loop is a two-edged instrument and almost nobody measures the second edge. When an agent reflects and retries, it does two things at once: it rescues some tasks it had wrong, and it talks itself out of some it had right. If you only count the rescues…

What you'll learn

• building a Reflexion loop (Shinn et al.) around ONE tool-using triage agent from nothing but the Python standard library: a single-shot baseline, a verbal self-reflection written into an episodic-memory buffer on a distrusted run, and a bounded retry with that reflection in context - no pip, no model download, no API key, and a deterministic stub agent so every number reproduces byte for byte
• measuring a self-critique loop in BOTH directions on the same batch: pass@1 0.7208 rising to pass@2 0.9083, the 58 tasks reflection RECOVERED (wrong on attempt 1, right at the end) set against the 13 it WRECKED (right on attempt 1, talked into wrong) - keeping the honest negative instead of quoting only the gain
• pricing the loop: 4.19 extra tool calls per recovered task, and reading a per-k table where the THIRD attempt recovers nothing the second did not and flips seven more right answers to wrong, so the loop is capped at k=2 by measurement rather than by taste
• proving that a reflection loop is only ever as good as the thing that TRIGGERS it: a selective self-check that fires on a distrusted verdict (precision 0.7160) nets +45, while unconditioned ALWAYS-reflect (precision 0.2792) nets -6 and drops the agent BELOW its own single-shot baseline, 0.7208 down to 0.6958
• seeing the ceiling of a self-critique loop stated as a number: 9 blind-spot escalate alerts carried a report that never conflicted with the closing verdict, so the trigger never fired and no amount of reflecting could reach them - a loop can only fix what its verifier flags
• the judgment to say when NOT to let an agent argue with itself: when the self-check is imprecise, when the retry strategy over-corrects a heuristic that was already right, and when the marginal attempt stops recovering and only adds flips

Steps in this lab

1. The words first: agent, tool, triage, single-shot, pass@1, Reflexion, episodic memory, self-check, recovered, flipped - and the fact that the agent here is a stub
2. Open a terminal and make a scratch folder (all three operating systems)
3. The corpus: 240 alerts whose correct verdict you already know
4. The agent, its one tool, and its self-check - watch it recover one alert and wreck another
5. Stage 1 - the baseline: single-shot triage, and where it fails
6. Stage 2 - add the Reflexion loop, and measure both edges
7. Stage 3 - the honest negative and the number that decides whether to run the loop at all
8. The rule: when a reflection loop is worth its cost, and when NOT to let an agent argue with itself

Open the full interactive lab →