Lion Cipher Lab
Trace a whole agent-team run and find the real bottleneck: build an OpenTelemetry-shaped tracer from scratch, compute self time, slack and the critical path, and prove the slowest span is not the problem
- AI Security
- Intermediate
- ~90 min
- Free · no account required
Why it matters
An agent team is a distributed system that its authors usually refuse to think of as one. The moment you have a planner, four workers and a critic, latency stops being a property of any single model call and becomes a property of the STRUCTURE - who was handed work first, who queued for a tool somebody else was holding, who retried three times, and who everybody had to wait for at the end. Part 1 of this pair taught you to trace one call and read p50, p95, tokens and cost from a pile of them. That is exactly the right instrument for one call and exactly the wrong instrument for six actors,…
What you'll learn
- building a tracer from scratch in pure Python, modelled on the OpenTelemetry span data model: trace id, span id, parent span id, name, kind, start/end, attributes, status and events - and naming the six things it deliberately leaves out (context propagation, links, resource/scope, the SDK pipeline, the other signals, and a real clock)
- instrumenting an agent TEAM rather than a single call: agent turns, tool calls, serialized handoffs, queue waits, retries with backoff, and a fan-in barrier
- reproducing a distributed run exactly with a seeded virtual clock, and knowing why sample-first-schedule-second is what makes a before/after comparison honest
- rebuilding a span tree from a flat file and computing SELF TIME correctly by unioning overlapping child intervals instead of summing them
- computing the CRITICAL PATH of a trace with a backwards walk that attributes every millisecond of the run to exactly one span, and proving it by summing the segments back to the total
- reading slack: knowing that the 9,268.8 ms span with 5,305.2 ms of slack cannot be the bottleneck no matter how big its bar is
- splitting a worker's time into queue wait, retry backoff and service time, and choosing a fix from that split
- measuring a fix instead of assuming it: 4,413.8 ms saved by one config change, and exactly 0.0 ms saved by an 80% speedup of the slowest span in the trace
- judging what head sampling costs you (0 of 15 slow runs retained at a 5% keep rate) and how attribute cardinality turns a trace bill into a liability
- the judgment to know when a hand-rolled tracer is the wrong answer and a real backend one config line away is the right one
Steps in this lab
- Part 1 traced one call. This lab traces a whole team.
- Open a terminal, check Python, make a folder
- Build the tracer, part 1: what a span actually is
- Build the tracer, part 2: the Tracer - and what it deliberately leaves out
- The incident, the team, and why the clock is deliberately fake
- Emit the trace: the run that produces 37 spans
- Read one span with your own eyes before you read 37
- Read the trace, part 1: rebuild the tree, compute self time, draw a waterfall
- Read the trace, part 2: the critical path, slack, and the per-worker table
- Run the analysis, and watch the slowest span turn out to be innocent
- Fix one thing, re-measure, and watch the bottleneck move
- Now do the obvious thing instead, and measure how little it buys
- Sampling and cardinality: the two things that decide whether you can afford this
- When NOT to build your own tracer
- What to take back to your own agent team