Your app asks the LLM the same question 500 times a day.
Here is the cache that fixes it in an afternoon.

I'm a senior backend tech lead and I run Belmo, a small European PaaS. I'll mention it once near the end with a clear disclosure. Everything before that works on any platform, any model provider, and any cache you already have lying around.

You ship an AI feature. It works. A few hundred people use it, and the OpenAI bill arrives looking like rent. You open the dashboard expecting heavy users and instead you find the same handful of requests, repeated. The same product description regenerated forty times because forty people viewed the same item. The same support question answered from scratch on every page load. The same summary recomputed because nothing told your app it had already done the work. You are not paying for intelligence. You are paying, over and over, for an answer you already had.

This is the most common waste I see in apps built fast with Claude Code, Cursor, or Lovable, and it is also the easiest to fix. The model call feels like the product, so people treat it as sacred and untouchable, when most of the time it is a pure function: same input, same output, and you are calling it as if the output were new every time. The fix is a cache, and not a clever one. You can get most of the win in an afternoon with tools you already run.

Why your bill scales with traffic instead of with users

A normal web request reads from a database in single-digit milliseconds and costs you effectively nothing. An LLM request takes one to ten seconds and costs real money per call, sometimes a meaningful fraction of a cent, sometimes several cents on the larger models with long prompts. So the economics of an AI app are inverted compared to a CRUD app. In a CRUD app, traffic is cheap and you optimize the database when you grow. In an AI app, every uncached call is a line item, and traffic maps directly onto spend.

The thing that makes this fixable is that a huge share of those calls are not unique. Think about what your prompt actually contains. If it is "summarize this article" and the article does not change, the answer does not change either. If it is "classify this support ticket into one of these five categories," the same ticket text produces the same category. The model is deterministic enough for these purposes, especially at low temperature, that calling it twice for the same input is just buying the same answer twice. Caching is not a trick here. It is recognizing that you built a pure function and then forgot to memoize it.

Layer one: the exact-match cache

Start with the boring version, because it covers more traffic than you expect. You hash the full input, you check a cache, and if the answer is there you return it without calling the model at all.

The important part is what goes into the hash. It is not just the user's text. It is everything that could change the output: the model name, the temperature and other parameters, the system prompt, and a version number you control. If any of those change, you want a different key, because the old cached answer is no longer the answer you would produce today.

import hashlib
import json
import redis

r = redis.Redis()

# Bump this when you change the prompt template or how you post-process output.
PROMPT_VERSION = "v3"

def cache_key(user_input: str, model: str, temperature: float, system: str) -> str:
    payload = {
        "v": PROMPT_VERSION,
        "model": model,
        "temp": temperature,
        "system": system,
        "input": user_input.strip(),
    }
    blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
    return "llm:" + hashlib.sha256(blob.encode("utf-8")).hexdigest()

def get_completion(user_input, model="gpt-4o-mini", temperature=0.0, system=""):
    key = cache_key(user_input, model, temperature, system)
    cached = r.get(key)
    if cached is not None:
        return cached.decode("utf-8")

    answer = call_the_model(user_input, model, temperature, system)  # your real call
    r.set(key, answer, ex=60 * 60 * 24 * 7)  # keep for a week
    return answer

That is the whole idea. Everything else is refinement. Note the PROMPT_VERSION constant, because it is the part people forget. The day you improve your system prompt, every cached answer becomes stale, and without a version bump you will serve the old behavior for a week and lose an afternoon wondering why your prompt changes did nothing. Bumping one string invalidates the entire cache cleanly.

A word on temperature. If you are caching, you almost certainly want temperature=0 or close to it for the cacheable calls. High temperature exists to give you variety, and variety is the opposite of cacheable. Decide per feature: a creative tagline generator might want randomness and should not be cached, while a classifier or a summarizer wants the same answer every time and should be. Do not cache a call whose entire value is that it is different each time.

What to actually use as the cache, and a note on TTL

If you already run Redis or Memcached, use it. If you do not, a database table with a unique index on the key column works fine and survives restarts, which an in-memory dict in your web process does not. The cache living in process memory is the trap I see most often: it works in testing, then you scale to two instances and your hit rate silently halves because each instance has its own copy, and it resets to zero on every deploy. Put the cache somewhere shared and persistent from the start.

On expiry, match the time-to-live to how often the underlying truth changes. A summary of a fixed document can live for weeks. A classification of static text can live effectively forever, so a long TTL is fine. Anything that depends on data that changes, prices, inventory, anything personalized, needs a short TTL or a real invalidation hook that deletes the key when the source changes. The failure mode of caching is not usually cost, it is serving a stale answer with full confidence. Be honest about which of your calls are pure and which only look pure.

Layer two: the semantic cache, and the trap inside it

Exact matching misses near-duplicates. "How do I reset my password" and "I forgot my password, how to reset" are the same question to a human and produce two different hashes. A semantic cache closes that gap by comparing meaning instead of bytes. You embed the incoming text, look for a stored embedding that is close enough, and if you find one you reuse its answer.

import numpy as np

SIMILARITY_THRESHOLD = 0.92  # tune this carefully, see below

def semantic_lookup(user_input, stored):  # stored: list of (embedding, answer)
    query = embed(user_input)  # your embedding call, also worth caching
    best_answer, best_score = None, 0.0
    for emb, answer in stored:
        score = np.dot(query, emb) / (np.linalg.norm(query) * np.linalg.norm(emb))
        if score > best_score:
            best_answer, best_score = answer, score
    if best_score >= SIMILARITY_THRESHOLD:
        return best_answer
    return None

In production you would not loop in Python over every stored vector. You would use a vector index, and if you already run Postgres you can add pgvector and let the database do the nearest-neighbor search, which keeps your stack small. The pseudocode above is to show the shape of the decision, not the data structure.

Here is the trap, and it is worth slowing down for. The threshold is doing something dangerous: it decides that two different inputs are close enough to share an answer. Set it too low and you will confidently serve the answer to a question nobody asked. "What is your refund policy" and "what is your privacy policy" are textually similar and semantically opposite, and a loose threshold will happily hand a user the wrong document. Exact-match caching can never be wrong, because identical input genuinely has identical correct output. Semantic caching can be wrong, because "similar" is a judgment call you are delegating to a number. Start strict, around 0.92 to 0.95 cosine similarity depending on your embedding model, log every semantic hit with both the query and the matched original so you can eyeball whether the matches are actually equivalent, and loosen only when the logs convince you. For a lot of apps, the exact-match layer alone is worth doing first and the semantic layer is worth adding only once you have measured that near-duplicates are a real share of your traffic.

The calls you must not cache

Caching is a statement that the same input deserves the same output, so the discipline is knowing where that statement is false. Do not cache anything personalized to a user unless the user identity is part of the key, or you will leak one person's answer to another, which is a privacy incident, not a bug. Do not cache anything that depends on current time or live data, like "what is trending today," without a TTL short enough to make it true. Do not cache deliberately creative output where variety is the feature. And be careful with anything that feeds a decision with money or safety attached, because a stale cached answer there is worse than a slow fresh one.

If you stream responses token by token to the user, you can still cache. Cache the full assembled answer once the stream completes, and on a cache hit replay the stored text, either instantly or with an artificial typing effect if the streaming feel matters to your product. You lose nothing the user can perceive and you skip the entire model call.

What I built

I run Belmo, a small European PaaS, and the reason caching is on my mind is that the AI log monitoring I built keeps surfacing the same shape of waste in the logs of customer apps: a burst of identical or near-identical model calls, the same prompt fingerprint firing hundreds of times an hour, the bill climbing while the actual unique work stays flat. That is the disclosure, that is the one mention. Belmo can show you the pattern in your logs and alert you to it over Telegram, but it cannot add the cache for you, because only you know which of your calls are pure functions and which are personalized or time-sensitive. The platform can point at the leak. Deciding what is safe to cache is product judgment, and it stays yours.

What to do tonight, regardless of your platform

1. Look at your model provider's usage logs and find your most repeated single prompt. Almost everyone has one call that dominates, and it is almost always cacheable. That one is your highest-leverage fix.
2. Add the exact-match cache around that one call. One hash function, one cache lookup, one write on miss. Resist doing all your calls at once. Prove the pattern on the worst offender first.
3. Put a version string in your cache key today, before you have a cache to invalidate, so that the day you change a prompt you bump one constant instead of flushing by hand or serving stale answers.
4. Set the temperature to zero on every call you intend to cache, and consciously leave the deliberately creative calls uncached. Write down which is which so the next person, including future you, does not cache the wrong thing.
5. Move any in-process cache to something shared, Redis or a database table, so it survives deploys and works across more than one instance.
6. Add a hit-rate counter. You want one number that tells you what fraction of calls the cache served, because that fraction is your bill reduction, and without measuring it you are guessing.
7. Only after exact match is live and measured, consider a semantic layer, start the threshold strict, and log every semantic hit so you can verify the matches are truly equivalent before you trust them.

One open question

The part I keep wrestling with is the semantic threshold, because it is a dial with money on one side and correctness on the other. Tighten it and you save less but you are never wrong. Loosen it and you save more but you start occasionally serving a confidently incorrect answer to a question that was merely similar, not the same. For a low-stakes feature like tag suggestions I will happily run it loose. For anything a user might act on I get nervous fast, and I tend to keep the semantic layer off entirely and live with the lower hit rate of exact matching.

So I will put the question to people who run these apps for real. Do you trust a semantic cache in front of user-facing answers, and if so, how did you decide the threshold was safe? Did you A/B it, audit the logs by hand, gate it to certain features? I suspect the honest answer for most small teams is that exact match is plenty and semantic is a premature optimization, but I would like to be talked out of that.

Related reads on this blog: Your AI app is silently burning $2,000/month and you don't know it. Here are the 5 patterns that bite founders. · Your /generate endpoint has no rate limit. One bored person with a for-loop turns your OpenAI bill into the punchline. · Your logs are useless at 2am. The 4-line change that makes every production bug findable (structured logging for founders who never set it up).