Lion Cipher Lab

Stop your RAG app from leaking documents: enforce per-user authorization in retrieval

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

Retrieval-Augmented Generation is now the default way teams put a chatbot on top of their own documents: when a user asks a question, the app RETRIEVES the most similar chunks from a document store and pastes them into the model's prompt as context, so the answer is grounded in real files instead of the model's memory. It is genuinely useful, and it ships with a dangerous default flaw. A vector store ranks by SIMILARITY, and a language model reads whatever text you hand it — neither has any notion of WHO is asking. So the naive pipeline everyone builds first will happily retrieve an HR…

What you'll learn

• building a tiny MULTI-TENANT RAG pipeline from scratch with a pure-Python TF-IDF + cosine retriever (scikit-learn) — no API key, no GPU, no model needed on the graded path
• tagging every document/chunk with an ACL in its METADATA and writing a deny-by-default authorized(doc, user) predicate that is the one place access is decided
• DEMONSTRATING the classic RAG flaw (OWASP LLM02, Sensitive Information Disclosure): a naive top-k retriever ranks over the whole index and quotes a confidential document to a user who must not see it, because similarity search has no notion of who is asking
• proving that 'just tell the model not to reveal other users data' is NOT a control — the confidential text is already inside the assembled prompt, so a rephrase or a prompt injection surfaces it; access control must live below the model
• fixing it with DEFENSE IN DEPTH: (a) a PRE-RETRIEVAL metadata filter that scopes candidates to the caller ACL before ranking, and (b) a POST-RETRIEVAL authorization check that drops any chunk the caller cannot read before it reaches the prompt
• MEASURING the fix: a leakage counter (cross-tenant documents retrieved) computed over a small test set of queries, before and after, that drops from N to 0
• the judgment layer: pre-filter vs post-check trade-offs, why per-tenant index/namespace isolation is a stronger boundary than shared-index metadata filtering, 'authorize at the data layer not in the prompt', and the residual risks (embeddings, logs/caches, and prompt injection are separate controls)

Steps in this lab

1. The flaw, in one breath
2. Set up a free, key-free, offline environment
3. Build the multi-tenant corpus and the TF-IDF index (rag.py)
4. Demonstrate the LEAK with your own hands (leak_demo.py)
5. Why 'just tell the model not to reveal it' is NOT a control
6. Fix, part A: a PRE-retrieval ACL filter (deny-by-default)
7. Fix, part B: a POST-retrieval authorization check (belt-and-braces)
8. Re-run the exact attack: zero leakage (secure_demo.py)
9. MEASURE it: a leakage counter from N to 0 (measure.py)
10. OPTIONAL (not graded): a local model, and the real-vector-DB pattern
11. Judgment: pre vs post, index isolation, and the residual risks
12. What you built — and the one sentence to keep

Open the full interactive lab →