Lion Cipher Lab

Poison a RAG knowledge base (PoisonedRAG) and defend it: flip an answer with a few crafted documents, measure the attack success rate, then drive it back down (free, local, deterministic — no API key)

• AI Security
• Advanced
• ~90 min
• Free · no account required

Why it matters

The moment your app answers from retrieved text, the documents it retrieves become part of its trust boundary — and most teams never treat their corpus like the security-critical asset it now is. PoisonedRAG is the sharp version of that risk: an attacker who can add even a handful of documents to your knowledge base (a wiki anyone can edit, a scraped web index, a shared drive, a support-ticket store, a user-upload feature) can make your assistant confidently state an attacker-chosen 'fact' for a specific question — a wrong dosage, a malicious download URL described as official, a competitor…

What you'll learn

• the PoisonedRAG threat model: corrupting the ANSWER by poisoning the knowledge base, not by injecting instructions
• how it differs from indirect prompt injection (answer integrity vs control-flow hijack)
• the two conditions a poison document must satisfy: the RETRIEVAL condition and the GENERATION condition
• building a tiny, fully local, deterministic RAG: sentence-transformers embeddings + a numpy cosine index + an LLM-free reader
• crafting poison docs black-box (echo the target question) vs white-box (optimise against the known retriever)
• measuring Attack Success Rate (ASR) before vs after injection, and watching a single doc flip a naive reader
• self-consistency / majority voting across retrieved passages, and why raising k can help OR hurt
• why a keyword/blocklist filter misses knowledge poison, and how near-duplicate detection measurably lowers ASR
• the judgment: RAG moves the trust boundary onto your CORPUS — provenance and write-access become security controls, and retrieval relevance is not truth

Steps in this lab

1. Two different attacks on RAG — and which one this is
2. The two conditions a poison document must satisfy, and ASR
3. Why relevance is not truth — the taxonomy and the judgment
4. Set up a clean environment and install the local stack
5. Write the deterministic RAG: corpus, targets, embedder, cosine index, reader, and the attack
6. Measure the clean baseline: ASR = 0.00
7. Inject the poison and watch ASR jump to 1.00
8. Defense 1 — self-consistency (majority vote), and why bigger k cuts both ways
9. Defense 2 — why the keyword filter fails, and the near-duplicate filter that works (with its cost)
10. The judgment: named failure modes, and where the real boundary lives
11. Do it better — where a production version goes

Open the full interactive lab →