Lion Cipher Lab
Add semantic caching to your LLM app: cut cost and latency by reusing answers to similar questions (without leaking data)
- AI Security
- Intermediate
- ~60 min
- Free · no account required
Why it matters
The single biggest lever on an LLM app's cost and latency is not calling the model when you do not have to. Real traffic is enormously repetitive: hundreds of users ask the same FAQ, docs, or support question worded slightly differently, and a plain string cache misses every paraphrase because 'How do I reset my password' and 'I forgot my login' share almost no characters. A semantic cache fixes exactly this: it stores each answer keyed by the EMBEDDING of the question, and when a new question is close enough in meaning (cosine similarity above a threshold) it returns the stored answer…
What you'll learn
- building a semantic cache: keying past answers by the question's embedding
- running a local embedding model (nomic-embed-text) and a local chat model on Ollama
- computing cosine similarity by hand and using a similarity threshold
- measuring a cache hit rate, the latency drop, and the token cost saved
- tuning the similarity threshold and seeing the false-hit tradeoff first-hand
- keying a cache per-tenant/per-user and excluding PII to avoid a cross-user data leak
- judging when semantic caching helps (high-volume, repetitive, non-personalized Q&A) and when it is dangerous
Steps in this lab
- What a semantic cache is, in one breath
- When this helps -- and when it is the wrong tool
- Install Ollama and pull TWO models
- Build the plumbing: talk to both models from Python
- Add the cache: an embedding-keyed lookup with a threshold
- Baseline vs cache: run it and MEASURE the win
- Tune the threshold and watch it serve a WRONG answer
- The security caveat: a naive cache leaks data and spreads poison
- What you built, and how the pros run it