Lion Cipher Lab
See inside your LLM app: trace every call, its tokens, latency, and cost
- AI Security
- Intermediate
- ~60 min
- Free · no account required
Why it matters
A language model is the one piece of your app you cannot reason about by reading its code. It is non-deterministic — the same prompt can give different answers — so when it misbehaves you have nothing to step through. Worse, the two things most likely to hurt you are completely invisible by default: cost (every call burns tokens, and a bad prompt change or a runaway loop can quietly 10x your bill) and latency (a model that got slower will ruin a product before anyone can say why). Observability is the fix, and it is not optional at any real scale: you record a structured trace for every…
What you'll learn
- adding observability to an LLM app (tracing every model call)
- recording a structured span: model, tokens, latency, status, cost
- reading Ollama's response fields (prompt_eval_count, eval_count, total_duration)
- writing traces as JSON Lines and summarizing them later
- computing p50/p95 latency, total cost, and error rate from a trace file
- redacting PII and secrets from traces before they are stored
- judging when observability matters and how pros do it (OpenTelemetry GenAI, sampling)
Steps in this lab
- Why an LLM app is undebuggable without traces
- Install Ollama and a small model
- Build the tracer: capture a span for every call
- Add cost, redaction, and write the trace file
- Run the tracer and watch spans land
- Read the summary: p50/p95 latency, cost, and error rate
- Run the summary and read what it tells you
- When it matters, and how the pros do it for real