Lion Cipher Lab
Force an LLM to return valid JSON — validate it, and build a repair loop for when it doesn't
- AI Security
- Intermediate
- ~45 min
- Free · no account required
Why it matters
The moment you ask an LLM for JSON so another program can use its answer, you have a new problem: sometimes it returns prose, sometimes it wraps the JSON in ```json fences, sometimes it adds a trailing comma or invents a field. In a chat window a human shrugs; in a harness that is a crash or, worse, a silently bad record flowing downstream. Every serious LLM app therefore validates the model's output and repairs it when it's wrong. This is the exact skill OWASP calls out as 'Improper Output Handling' — trusting model output as if it were correct. Learn the pattern once here and you can safely…
What you'll learn
- declaring the shape you require as a Pydantic model (and deriving its JSON Schema)
- constrained decoding with Ollama's format= parameter
- parsing and validating LLM output against a schema
- catching the three real failure modes (not JSON, wrong/missing fields, extra hallucinated fields)
- building a bounded repair loop that re-prompts with the error
- failing closed instead of passing bad data downstream
- judging constrained decoding vs validate-and-repair, and the cost of each
Steps in this lab
- The problem: why an LLM's JSON breaks your harness
- Two approaches, and when each is right (read before you build)
- Install Ollama, pull a small model, and set up Python
- See the problem for yourself: ask naively and look at the raw output
- Declare the shape you require: a Pydantic model
- Ask with constrained decoding (format=SCHEMA) — and why that is still not enough
- Validate: catch the three real failure modes
- Build the repair loop — and FAIL CLOSED when the budget runs out
- The judgment: what a repair loop costs, and when to cap it