LLM API Rate Limits and Proxies: Why Changing Your IP Does Nothing

A developer building on an LLM API starts hitting 429 responses, finds our site while searching for a fix, and asks whether a rotating residential pool will spread the load across enough addresses to get past it.

It will not, and we would rather say so than sell a gigabyte that cannot possibly help. The reason is structural rather than a matter of degree, and once it is clear the real fixes are obvious.

The header is the identity, not the address

An LLM API request carries a credential in its headers. That key identifies the caller, and everything the provider accounts for, quota, spend, tier, usage history, is attached to it. The source address is transport detail: it tells the server where to send the response.

So a rotating pool changes the return path and nothing else. A thousand different exit addresses sending the same key produce a thousand requests billed and counted against one account, exactly as if they had all come from your laptop. The limiter was never looking at the address, so there is nothing for the address change to affect.

This is the opposite of web scraping, where the address genuinely is the identity because there is no credential and the site must guess who you are from what it can see. That difference is the whole reason proxies work for one and not the other, and it is why advice from the scraping world transfers badly to API work.

Reading the 429 properly

Before changing anything, read what the response is telling you. Providers typically meter several dimensions at once, and they fail differently.

Requests per minute. Too many calls in the window, regardless of size. Usually the easiest to fix, because it is caused by firing an unbounded loop at the API rather than by genuine volume.

Tokens per minute. Total input and output tokens in the window. This one surprises people, because a small number of very large requests can trip it while the request count looks fine.

Concurrency. How many requests are in flight at once. Common when a worker pool is sized for a database rather than for an API.

The response normally carries a Retry-After header, and it is the most ignored useful value in this entire subject. It is the provider telling you exactly how long to wait. Honour it before inventing a backoff schedule of your own.

Also check what your SDK already does. The official Anthropic SDKs, for instance, retry connection errors, 408, 409, 429 and 5xx automatically with exponential backoff, with the retry count configurable on the client. A great deal of hand-written retry code in the wild is duplicating that badly.

import anthropic

# The SDK already backs off on 429 and 5xx. Configure it rather than
# wrapping every call in your own loop.
client = anthropic.Anthropic(max_retries=5, timeout=60.0)

try:
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=16000,
        messages=[{"role": "user", "content": "..."}],
    )
except anthropic.RateLimitError as e:
    # Only reached after the built-in retries are exhausted.
    retry_after = int(e.response.headers.get("retry-after", "60"))
    print(f"Still limited. The provider asked for {retry_after}s.")

What actually moves the number

Six things, in rough order of how much they help.

Honour the header, then back off with jitter. Fixed-interval retries from many workers re-synchronise into a thundering herd that trips the limit again at the same instant. Randomising the delay spreads them out, and it is a one-line change.

Cache the stable prefix of your prompts. If every request re-sends the same long system prompt, instructions or document, prompt caching lets the provider serve that prefix from cache at a fraction of the input cost. Because token-per-minute limits count what you send, cutting repeated input reduces both the bill and the pressure on the limit. The catch is that caching is a prefix match, so anything volatile, a timestamp, a request id, an unsorted JSON blob, has to sit after the stable part or it silently invalidates everything behind it.

Move non-urgent work to a batch endpoint. Work that does not need an answer this second belongs in an asynchronous batch, which typically prices well below the synchronous path and takes the load off your per-minute limits entirely. Nightly enrichment, backfills and evaluation runs are all batch-shaped and frequently are not batched.

Stream long outputs. A request generating a very large response can sit close to an HTTP timeout, and a timeout that gets retried doubles the work you just spent quota on. Streaming removes that failure mode.

Queue at a deliberate concurrency. Pick a number, enforce it with a semaphore, and let work wait. An unbounded worker pool does not go faster once the provider starts refusing it; it just converts throughput into 429s and burns retry budget.

Then ask for more. Once your usage is real, providers raise limits. That conversation goes better with a week of usage data behind it, and it is a legitimate step rather than an admission of defeat.

Notice that every item on that list is about your request pattern. None of them is about the network path, which is the point.

Size the limit before you fight it

Most rate-limit firefighting happens without anyone working out what the application actually needs, which is why the fixes get applied in the wrong order. The arithmetic takes five minutes and usually names the problem outright.

Start with tokens rather than requests, because the token meter is the one that surprises people. For one representative call, count the input and the output. Providers expose a token-counting endpoint precisely so you do not have to estimate this, and using it beats guessing from character counts, which are wrong by a different factor for every language and every prompt shape.

Then multiply by the rate you intend to run at. Twenty calls a minute at six thousand input and one thousand output tokens is 140,000 tokens a minute, and if your ceiling sits below that no retry schedule in the world will save you: you are asking for more than you have, continuously, and the 429s are the system working correctly.

That framing sorts the fix immediately. If the number you need is far above your ceiling, the answer is a limit increase or a cheaper request, not better retries. If the number is comfortably under the ceiling and you still get 429s, the problem is burstiness rather than volume, and a concurrency cap will fix it. If the input side dominates the total, caching the stable prefix is the single highest-leverage change available. And if none of the work is time-sensitive, batching removes the question entirely.

Two things worth measuring while you are in there. Check whether your cached prefix is actually being served from cache, because providers report that in the usage figures and a silent invalidator makes the whole optimisation vanish without any error. And check whether retries are counted against you, since a retry storm can consume more quota than the original work and produce nothing.

The two cases where the address does matter

Neither is a rate limit, and both are worth separating out because they get filed under the same complaint.

Regional availability. Some providers do not serve some countries. That is a reachability problem, it usually presents as a refusal or a block rather than a 429, and it is governed by the provider's terms rather than by your quota. Read those terms before routing around them, because access rules generally exist for reasons the provider will enforce on the account rather than on the address.

Corporate egress. Your own network blocks the endpoint. This is the most ordinary problem in this article and the one people least expect, since nothing about the error mentions the network. If the same request works from a phone tether and fails on the office wifi, stop reading about rate limits.

Where proxies genuinely belong in an AI stack

There is real work here for the product we sell, and it sits on the other side of the model.

Agents that browse the open web. An agent following links, reading pages and pulling data is making ordinary HTTP requests to ordinary websites, and those sites run the same bot defences everyone else faces. That is exactly the job residential addresses exist for, and it is covered in proxies for AI agents.

Collecting training and retrieval data. Gathering a corpus at scale runs into rate limits and blocks that genuinely are per-address, because the sources have no idea who you are. Proxies for LLM scraping covers what that takes and how the sources are changing.

Testing your own AI product from elsewhere. If your application behaves differently by region, verifying that requires exiting in those regions. Ordinary geo-targeting, ordinary web traffic.

Checking what competitors expose. Pricing pages, model availability, documentation and public benchmarks are web pages, and reading them from several countries is the same problem as any other price monitoring job.

In every one of those the traffic is going to a website rather than to an authenticated model endpoint, and that is the line. Web traffic, proxies help. Keyed API traffic, they do not.

The workaround we will not help with

There is an obvious idea sitting behind a lot of these questions, so it is worth answering directly: many free-tier accounts, each with its own key, spread across addresses so they do not look connected.

We will not help build that, and separately it does not work well. Providers link accounts on payment methods, phone numbers, identity verification, device signals and behaviour, and those correlations are both stronger and cheaper to compute than an IP match. The address is the easiest signal to change and the least load-bearing one, which is why changing it rarely decides the outcome. What it does reliably achieve is putting the account you actually care about at risk when the cluster is found.

If the free tier is not enough, the honest answer is that the work has outgrown free, which is the same thing we say about free proxies on our own side of the fence.

The short version

A 429 from a model endpoint is a message about your key, so route the fix through your request pattern: honour Retry-After, cache your stable prefix, batch what can wait, stream what is long, and cap your concurrency deliberately. No amount of address rotation touches any of it.

When the traffic in question is aimed at websites rather than at a model, that is our half of the problem, and residential proxies start at $0.50/GB for a single gigabyte with no subscription. If you are not sure which half you are looking at, the test takes one second: does the request carry an API key? If it does, the proxy is not the answer.