Lion Cipher Lab

Grade an agent on the path it took instead of the answer it gave: build a trajectory and tool-choice evaluation harness from scratch, run eight agents through it, and count the runs where an output-only eval happily passed a path that isolated a host it had never identified

• AI Security
• Advanced
• ~120 min
• Free · no account required

Why it matters

This is the last rung of the agent-engineering arc, and it exists because of a gap the previous eighteen labs opened and none of them could close. Those labs built agents: a cost-and-capability router, a resumable workflow engine, a plan-then-execute agent that diffs intent against reality, an idempotent run journal, a budgeted fleet with circuit breakers, a subagent harness with context isolation, typed IO contracts, an adversarial verifier, a planner-worker-critic team, a shared blackboard, an agent-to-agent protocol, a debate, a supervisor. Every one of them was measured. Every one of them…

What you'll learn

• building a TRAJECTORY representation and a scored harness over it from scratch - seven dimensions, one scorecard per run, 2,400 runs graded across 300 synthetic SOC cases and 8 agent policies, in pure Python standard library with no numpy, no sklearn and no network
• measuring TOOL-CHOICE CORRECTNESS as precision, recall and F1 over the calls, deliberately ORDER-FREE, so that a legal reordering never masquerades as a tool-selection error - and knowing why that decomposition is load-bearing rather than fussy
• measuring ORDER three separate ways on the same run - Kendall tau over first-occurrence positions, a Needleman-Wunsch edit distance over the tool sequence, and a dependency-graph grader - and knowing why computing tau over an alignment always returns exactly +1.000 and is therefore worthless
• writing a DEPENDENCY-GRAPH grader that accepts any topological order satisfying the real constraints, verifying with graphlib.TopologicalSorter that the graph is acyclic and the reference is one of its topological orders, and counting the linear extensions by subset dynamic programming - 192 legal paths for the beacon case, of which a strict grader accepts one
• quantifying how badly a naive STRICT grader under-reports: 9.9% pass against the partial-order grader 47.2%, a 37.4-point gap, with 897 of the strict grader 2,163 failures (41.5%) turning out to be correct paths - and a 63.7% false failure rate on an agent that is correct on every single run
• separating ARGUMENT CORRECTNESS from tool choice and catching the failure that actually causes outages - an agent with tool F1 1.000 and perfect ordering that isolated the wrong host on 107 calls across 54 runs
• classifying TERMINATION rather than assuming it - clean, looped, cap_hit and early_stop - and reading a policy that repeats enrichment until 86% of its runs loop and 14% hit the 16-step cap without ever closing the ticket
• scoring an explicit SAFETY dimension over irreversible actions by checking each precondition at the moment of the call, which found 420 precondition violations and 92 entirely unwarranted containments from one policy and 240 from another that simply skipped the criticality check
• putting an output-only eval and a trajectory eval on the identical runs and reporting the DISAGREEMENT - 57.88%, of which 1,192 runs were bad paths waved through and 197 were good paths failed for wording - plus the Spearman +0.143 between the two leaderboards
• the judgment to report a VECTOR rather than a scalar, having watched two runs with an identical aggregate score of 0.8286 differ by one isolate_host on unmet preconditions, and to justify every dependency edge in one sentence, having watched a single wrong edge fail a better-than-reference policy on 100% of its runs

Steps in this lab

1. The words first: agent, tool call, trajectory, span, reference trajectory, precision, recall, F1, DAG, topological order, partial order, edit distance, step cap, termination, irreversible action
2. Open a terminal and make a scratch folder (all three operating systems)
3. The corpus: 300 SOC cases, each with a reference trajectory AND a dependency graph
4. Eight agents, and every single one of them is a stub - say it out loud before you look at a number
5. The harness itself: seven dimensions, written out by hand, and why the decomposition is the hard part
6. Score all 2,400 runs: the four tables the whole vein has been missing
7. The headline: put both evals on the same runs and count where they disagree
8. Strict versus partial order: the named failure mode of naive trajectory eval, with a number on it
9. The three honest negatives, each with a number: the scalar trap, the over-strict edge, and when this whole technique bought nothing
10. Plug your own agent in - and make the harness unit-test itself before you believe a word of it
11. When NOT to reach for a trajectory eval
12. The way a pro does it differently

Open the full interactive lab →