Lion Cipher Lab
Make your LLM app resilient: timeouts, retries with backoff, a circuit breaker, and a fallback model
- AI Security
- Intermediate
- ~60 min
- Free · no account required
Why it matters
A call to a language model is a call over the network to a slow, stateful, rate-limited backend -- and in production that call WILL fail sometimes: it times out, it gets rate-limited (HTTP 429), the server throws a 500, the connection drops. The naive app treats the model like a local function that always returns, so the first hiccup either crashes it or -- worse -- hangs it forever on a call that never comes back, taking your whole app down with it. Resilience is the difference between a demo and a product, and it is four concrete, learnable moves: put a hard TIMEOUT on every call so one…
What you'll learn
- telling transient errors (timeout, 429, 5xx, reset) apart from permanent ones (400, 401, context-length)
- putting a hard timeout on every model call so one hung call cannot block your whole app
- writing retries with exponential backoff AND full jitter, and knowing why jitter matters
- implementing a circuit breaker (closed/open/half-open) to stop hammering a dead backend
- adding a fallback model that degrades gracefully and flags when a fallback answer was served
- injecting real failures (dead port, bad model name, forced delay) and watching the app recover
- judging when NOT to retry (non-idempotent side effects, permanent errors, retrying forever)
Steps in this lab
- Why LLM calls fail, and the one decision that matters
- Install Ollama and a small model
- Step 1 - a hard timeout, and classifying the error
- Step 2 - retries with exponential backoff AND full jitter
- Step 3 - a circuit breaker so a dead backend can recover
- Step 4 - the fallback model, and putting it all together
- Prove it: run the happy path, then INJECT failures
- When NOT to retry, and how pros run this for real