Lion Cipher Lab

Build a planner / worker / critic agent team from scratch, race it against one agent on the same scored alert-triage batches, and find the difficulties where the two extra roles do not pay for themselves

• AI Security
• Advanced
• ~120 min
• Free · no account required

Why it matters

The agent-team diagram is the most persuasive drawing in modern software and one of the least measured. A PLANNER decomposes the job. WORKERS do the pieces. A CRITIC checks the work and sends it back. It looks obviously better than one agent grinding through everything alone, it maps onto how a team of humans actually works, and it is very easy to ship without ever asking the only question that matters: on YOUR job, against the SAME data, does it win, and by how much, and for how much? This lab makes you answer that. The job is a real one — a batch of security alerts to triage into…

What you'll learn

• defining an AGENT TEAM precisely and building one in pure Python: a PLANNER that decomposes a job into assignments and never does the job, WORKERS that execute one assignment each in their own window, a CRITIC that reviews results against the plan and can send work back, and the ORCHESTRATION that moves work between them
• scoring both architectures on one rubric against planted ground truth, so 'the team is better' becomes an F1 number with a token bill, a call count and a wall-clock next to it instead of an architecture diagram
• running the comparison across four task difficulties and finding the point where the overhead stops being repaid — the team beat one agent by 0.618 F1 on the brutal batch and by 0.066 F1 on the easy one, for the same 7.5x token bill
• beating the RIGHT baseline: not the naive one-shot agent but plain CHUNKING — the same single agent called once per fifteen alerts with no roles at all — which is the cheapest thing that also fixes the window problem and the thing a team actually has to beat
• naming and demonstrating the four team-specific failure modes with data: a plan the workers cannot execute, a critic that never converges, ROLE COLLAPSE where the critic simply agrees, and a plan that goes stale the moment a worker discovers something
• writing a PLAN VALIDATOR that rejects an unexecutable plan before a single worker is called, at a cost of zero model calls, and understanding the two rules it checks
• setting an ITERATION CAP, measuring what it costs and what it saves, and then fixing the underlying design so the loop terminates by AGREEMENT instead of by crash barrier
• running an ABLATION — team minus critic, team minus planner, team minus both — to attribute the win to a role instead of to the architecture, and finding that which role earns its keep flips with difficulty
• the judgment to say when a team is the wrong answer, and to tell a real role apart from the same agent wearing a second name badge

Steps in this lab

1. The words first: agent, agent team, role, planner, worker, critic, orchestration, iteration cap, ablation
2. Open a terminal and make a scratch folder (all three operating systems)
3. The cyber job: twenty alert batches whose right answers you already know
4. The meter: tokens, calls and a wall-clock that is honest about being simulated
5. The roles are DETERMINISTIC STUBS — read this before you paste it
6. The rubric: precision, recall and F1 against the planted truth
7. Architecture A: one agent, one window — and watch it fall off a cliff
8. Build the planner: decomposition is a design decision, not a chunk size
9. The workers, the critic, and the orchestration loop that joins them
10. The headline number — and the baseline the headline is usually hiding
11. Failure mode 1: the planner writes a plan the workers cannot execute
12. Failure mode 2: the critic that never stops, the cap that stops it, and what the cap costs
13. Failure mode 3: role collapse — pricing a critic that simply agrees
14. Failure mode 4: the plan goes stale the moment a worker learns something — and one agent wins
15. The ablation: which role actually earned its keep (it is not the same one at every difficulty)
16. When a team is the wrong answer
17. The way a pro does it differently

Open the full interactive lab →