Your /generate endpoint has no rate limit.
One for-loop turns your OpenAI bill into the punchline.

I'm a senior backend tech lead and I run Belmo, a small European PaaS. I'll mention it once near the end with a clear disclosure. Everything before that works on any platform you deploy to.

Your demo did better than you expected. Someone posted it, the traffic came, and for about forty minutes it was the best day your side project ever had. Then one visitor opened the network tab, saw that your "Generate" button just POSTs to /api/generate with a JSON body, and wrote eight lines of Python to call it in a loop. No login. No limit. By the time you noticed, your provider dashboard showed a number with a comma in it that was not there this morning.

This is the failure mode nobody warns the new wave of AI builders about. You shipped something that works, and the thing that works is also a money printer pointed at your own card. I have helped clean this up more than once, and the cleanup is always the same handful of fixes. The good news is they are not hard. You can put the important ones in tonight.

Why an AI endpoint is a different animal

For fifteen years the standard advice about rate limiting was about protecting your server. Too many requests, your CPU spikes, your database connections run out, the site gets slow for everyone. Annoying, but the cost is bounded. Your server can only do so much, and when it maxes out it just gets slow. The bill at the end of the month looks roughly the same whether you served the traffic well or badly.

An AI endpoint breaks that assumption completely. Every call to /generate reaches out to an LLM provider and spends real money, per request, in real time. There is no natural ceiling. Your own server barely does any work, it is mostly waiting on the model, so the usual symptoms (high CPU, slow responses) never show up to warn you. The system feels healthy right up until the invoice. A scraper hitting a normal JSON endpoint costs you some bandwidth. A scraper hitting an LLM endpoint costs you forty cents a call, and a script can comfortably make a few thousand calls a minute.

So the thing you are protecting is not your server. It is your wallet. That changes what "rate limiting" even means here. You are not just smoothing traffic, you are putting a hard cap on spend, and you want that cap to hold even when something goes wrong with everything else.

The three layers, in the order you should add them

Think about it as three separate questions, each with its own answer. How many requests can come from one source in a short window? How much can a single known user consume in a day? And how much is the whole app allowed to spend before someone has to look at it? Most people who get burned had zero of these. You need all three eventually, but they go in fastest-first.

Layer one: a per-IP limit, because it takes five minutes

This is the floor. Before any auth, before anything clever, you put a basic limiter in front of the expensive route so that a single address cannot hammer it. On a single Node instance this is a few lines:

import rateLimit from "express-rate-limit";

const generateLimiter = rateLimit({
  windowMs: 60 * 1000,        // one minute
  max: 10,                    // 10 requests per IP per minute
  standardHeaders: true,      // send RateLimit-* headers
  legacyHeaders: false,
  message: { error: "Too many requests. Slow down and try again shortly." },
});

app.post("/api/generate", generateLimiter, handleGenerate);

Ten a minute is plenty for a real human clicking a button and a wall for a script. This will not stop a determined attacker with a pool of addresses, and it is not meant to. It stops the lazy 95 percent, the bored person with one for-loop, and it stops them tonight.

One catch that bites people behind a proxy or a CDN: the IP your app sees may be the proxy, not the visitor, so every request looks like it comes from one address. Set app.set("trust proxy", 1) (or the right hop count) so the limiter reads the forwarded client IP instead of rate-limiting the whole planet as a single user.

Layer two: per-user quotas, once you have accounts

In-memory limiters reset when your app restarts and do not share state across instances. The moment you run more than one process, or you want a limit that means "per account" instead of "per connection," you need a shared store. Redis is the usual answer, and the pattern is a counter with an expiry:

// Allow N generations per user per day. Returns true if allowed.
async function consumeQuota(userId, limit = 50) {
  const key = `quota:gen:${userId}:${todayUTC()}`; // e.g. quota:gen:abc:2026-06-20
  const used = await redis.incr(key);
  if (used === 1) {
    await redis.expire(key, 60 * 60 * 24); // first hit sets the daily TTL
  }
  return used <= limit;
}

app.post("/api/generate", generateLimiter, async (req, res) => {
  const ok = await consumeQuota(req.user.id, planLimit(req.user.plan));
  if (!ok) {
    return res.status(429).json({
      error: "You have hit today's generation limit.",
      upgrade: "/pricing",
    });
  }
  // ... do the expensive thing
});

Notice the limit comes from the user's plan. This is where rate limiting quietly turns into your pricing model. Free users get a small number, paying users get more, and the same line of code that protects you from abuse is the thing that makes the upgrade button mean something. A counter that resets at midnight UTC is crude but honest, and it is enough for a long time.

Layer three: a global spend cap, the one that actually saves you

The first two layers assume the abuse comes in a shape you predicted. The global cap is for the day it does not. It is a single number that says: across all users, all routes, everything, this app may make at most X expensive calls per hour, and past that it refuses and pages me. It is the circuit breaker.

async function withinGlobalBudget(maxPerHour = 500) {
  const key = `budget:gen:${currentHourUTC()}`;
  const used = await redis.incr(key);
  if (used === 1) await redis.expire(key, 3600);
  return used <= maxPerHour;
}

Set the ceiling a comfortable margin above a good real day and forget about it. On a normal week it never fires. On the one bad night, it is the difference between a forty dollar surprise and a four thousand dollar one. I treat this as non-negotiable for anything that spends money per request, because every other safeguard is something you reason about, and this is the one that holds when your reasoning was wrong.

Estimate cost before the call, not after

There is a subtler version of this problem that the request-counting layers miss. Not all calls cost the same. A user who pastes a 30,000-token document into your summarizer costs fifty times more than one who types a sentence, and a per-request counter treats them as equal. If your margins are thin, that is the gap someone drives a truck through.

So gate on size, not just count. Before you send anything to the model, look at the input and reject what is obviously out of bounds:

const estTokens = Math.ceil(inputText.length / 4); // rough chars-to-tokens
const MAX_INPUT_TOKENS = 8000;

if (estTokens > MAX_INPUT_TOKENS) {
  return res.status(413).json({
    error: "Input is too long. Trim it and try again.",
    limit_tokens: MAX_INPUT_TOKENS,
  });
}

Then set max_tokens on the output side of the model call so a single response cannot run away either. Capping both ends turns "one request could cost anything" into "one request costs at most this," and that is the number you actually want to reason about when you set the limits above.

Say no like a professional

When you reject a request, do it in a way a real client can recover from, because your own front end is one of those clients. The correct status is 429 Too Many Requests, and the kind thing to include is a Retry-After header telling the caller how long to wait:

res.set("Retry-After", "30");
res.status(429).json({ error: "Rate limit reached. Try again in 30 seconds." });

On the front end, catch the 429 and show "You are going a bit fast, try again in a moment" instead of a raw error or, worse, a spinner that never resolves. A surprising number of "the app is broken" reports are really an unhandled rate-limit response. Handling it well costs you one if statement and saves you a support thread.

You also want to see it coming

Caps stop the bleeding. They do not tell you it happened. The night your global budget fires, you want to know within minutes, not when you next happen to open a dashboard. So whatever you cap, also alert on. A spike in 429s is a signal: either a real user hit a wall and you should raise a limit, or someone is probing you and you should look. Either way you want a message, not a surprise.

This is the part I kept rebuilding by hand for clients, and it is what eventually pushed me to build Belmo, the European platform I run. Disclosure, since I said I would make it once: Belmo is my product, so I am not neutral about it. It watches your logs and flags patterns like a sudden run of rate-limit rejections or an anomalous burst on one route, and sends the alert to Telegram before the monthly total does the telling for you. You do not need Belmo for any of the code above, it runs anywhere. But the lesson it is built around is the honest one here: a cap you cannot see hit is a cap you will find out about late.

What to do tonight, whatever you deploy on

You can get most of the protection in under an hour. In rough order of how much each one saves you per minute of effort:

1. Put a per-IP limiter in front of every route that calls a model. Ten a minute is a fine starting point. This alone stops the casual for-loop, and it is five lines.
2. Set max_tokens on every model call right now. An uncapped output is an uncapped cost, and this is a one-line change with no downside.
3. Add a rough input-size check and reject anything absurdly large before it reaches the provider. Length over four is close enough for tokens.
4. Add one global hourly spend cap backed by a shared counter. Set it well above a good day. This is the circuit breaker, and it is the single highest-value thing on this list.
5. If you have user accounts, add a per-user daily quota and wire the number to the plan. You just built your pricing tiers and your abuse defense in the same five lines.
6. Make your front end handle a 429 gracefully, with a human message and a retry hint. Test it by setting your own limit to 1 for thirty seconds.
7. Send yourself an alert when 429s spike or the global cap trips. A counter you never look at is decoration.

One honest open question

The thing I still do not have a clean answer for is where to put the line between protecting yourself and quietly punishing your best users. Your most engaged free user and a scraper can look almost identical from the server's point of view: lots of requests, fast, no hesitation. Set the limits tight and you frustrate the person who loves the thing you made. Set them loose and you leave the door open. I tend to start strict and raise limits for specific accounts as I learn who they are, but that is a judgment call dressed up as a config value, and I am not sure it is the right one.

How do you draw that line in your own app? Do you rate-limit by request count, by cost, by some trust score that climbs as an account ages? I would like to hear what has actually worked once real users (and real abusers) showed up, because that is where the textbook answers tend to fall apart.

Related reads on this blog: Your AI app is silently burning $2,000/month and you don't know it. Here are the 5 patterns that bite founders. · Your app asks the LLM the same question 500 times a day and pays full price every time. Here is the cache that fixes it in an afternoon. · Your .env file is probably already in your Git history. The 15-minute audit (and the 5 habits that stop new leaks for good).