Lion Cipher Lab
Render untrusted model chat templates in a Jinja sandbox, and audit them before you load
- AI Security
- Advanced
- ~75 min
- Free · no account required
Why it matters
You download a model from a hub, point your favourite inference library at it, and send it a chat. Before the model ever generates a token, the library has already read a Jinja2 template out of the model's own metadata and rendered it to build the prompt. If it rendered that template with a plain jinja2.Environment, the model's author - who may not be who the upload says - just ran template code in your process. That is not hypothetical: CVE-2024-34359, nicknamed 'Llama Drama', was exactly this in llama-cpp-python (versions 0.2.30 through 0.2.71), rated CVSS 9.6, and the fix in 0.2.72 was to…
What you'll learn
- explaining why a model's chat template (the Jinja string in tokenizer_config.json or a GGUF's tokenizer.chat_template) is attacker-controlled code that an inference library executes, and naming CVE-2024-34359 as the real case
- building a corpus of chat templates - five realistic legitimate ones (ChatML, Llama-style, one using raise_exception, one using loop.index, one using namespace()) and five benign-canary 'malicious' ones that reach dunder attributes or walk a function's __globals__ to a planted marker
- measuring the vulnerable path: in a plain jinja2.Environment all five canaries fire, including one that reads a planted secret string and one that calls a reached function
- measuring the fix: switching to ImmutableSandboxedEnvironment blocks the object-graph canaries with SecurityError while every legitimate template still renders byte-for-byte identically (zero false breakage)
- writing a static pre-load auditor that parses each template with env.parse and walks the AST for Getattr/Getitem on dunder names, plus a pinned sha256 allowlist, and measuring its false negatives and false positives
- proving three things neither control fixes: the sandbox does not stop resource exhaustion (you add a render time/size cap), a |attr filter with a concatenated name evades the AST check, and a hash allowlist breaks on every benign template update
Steps in this lab
- The words first: chat template, SSTI, the object graph, dunder, sandbox, SecurityError, AST, allowlist - and the fact that no model runs
- Open a terminal, check Python, make a scratch folder, and install jinja2 in a virtual environment
- The case you are reproducing: CVE-2024-34359, and why this is not the web-SSTI fix
- Build the corpus: five real templates and five benign-canary ones
- BEFORE: render the corpus with a plain jinja2.Environment and watch all five canaries fire
- AFTER: switch to ImmutableSandboxedEnvironment and measure what breaks and what is blocked
- Add a static pre-load auditor: parse the AST for dunder access, and pin a hash allowlist
- The honest negatives: resource exhaustion, the auditor's blind spot, and allowlist brittleness
- When to reach for which control - and when NOT to
- Ship it: re-run, break a constant on purpose, and clean up