Rate Limiting

The three provider limits

Most providers enforce rate limits along three independent axes. Hitting any of them returns HTTP 429 and (on some providers) a temporary account-level cooldown.

A value of 0 means unlimited for that axis on that provider/tier.

Why TPM is necessary (and why it bites before the context window does)

The most counter-intuitive limit is TPM. People often ask: "Gemini has a 1-million-token context window, so why did my 300K-token request get rate-limited?"

The answer is that the context window and the TPM cap measure different things:

• Context window = how many tokens fit in a single request (the model's working memory for one call).
• TPM = how many tokens your account may consume in total across all requests in a 60-second window.

These exist for an infrastructural reason: providers run shared GPU inference clusters. A single forward pass over a 300K-token prompt occupies VRAM and compute on GPUs that dozens of other tenants are sharing. The provider cannot let one account monopolise the cluster, so they cap the total tokens-per-minute per account tier. TPM is the fairest way to allocate shared compute: whoever you are, you get N tokens of GPU time per minute, no more.

The practical consequence:

So TPM is necessary because the context window is a per-request ceiling; TPM is a per-account-per-minute ceiling on shared hardware. The context window being large does not mean your minute-level token budget is large. When the two disagree, TPM wins and you get 429.

This is also why Amplify's rate-limit handling focuses so heavily on TPM rather than just RPM: the most common real-world 429 for someone using a long-context model is a TPM violation, not an RPM violation.

How Amplify adapts (it does not impose limits)

To be explicit: Amplify does not rate-limit you. It does not throttle your requests for its own sake, and it does not set ceilings lower than what your provider allows. What it does is read the limits you've configured for each provider (in Settings) and steer requests so they're more likely to succeed against your provider's limits.

There are three adaptation mechanisms, all of which react to provider limits rather than impose new ones:

• 1. Pre-Flight Check

  Before sending a request, Amplify estimates its token cost (prompt tokens + expected completion budget) and compares it against your remaining TPM for the current 60-second window.

  • If the request would fit under remaining TPM → send immediately.
  • If it would exceed remaining TPM → Amplify waits until enough TPM has replenished (the rolling window slides forward), then sends.

  This turns a hard 429 rejection into a short, predictable wait. The provider still enforces the limit; Amplify just avoids triggering it.

• 2. RPM Throttling

  For providers with a tight RPM cap, Amplify spaces out consecutive requests so you don't fire 60 calls in the first second of a minute and then sit idle for 59. This is a client-side pacing layer — it never sends more requests than your RPM allows, but it also doesn't artificially cap you below your plan.

• 3. Auto-Shrink to TPM

  When a request's estimated token cost would exceed remaining TPM and the window won't replenish in time, Amplify can auto-shrink the request: it trims non-essential context (older memory entries, lower-priority skill descriptions) to bring the request under the TPM ceiling. This is a last resort before waiting — it lets the conversation continue with slightly less context rather than blocking entirely.

Configuring limits per provider

You set TPM / RPM / RPD per provider in the Settings → Providers panel. Each provider entry has three numeric fields; leave a field at 0 for "unlimited".

# amplify.config.yaml — provider rate-limit configuration
providers:
  openai:
    apiKey: ${OPENAI_API_KEY}
    rateLimits:
      tpm: 150000      # tokens per minute (0 = unlimited)
      rpm: 60           # requests per minute (0 = unlimited)
      rpd: 0            # requests per day (0 = unlimited)
  anthropic:
    apiKey: ${ANTHROPIC_API_KEY}
    rateLimits:
      tpm: 120000
      rpm: 50
      rpd: 0
  google:
    apiKey: ${GOOGLE_API_KEY}
    rateLimits:
      tpm: 250000
      rpm: 100
      rpd: 0

Limits are persisted per provider in localStorage under the amplify:rate-limits key, and the wire format serialised for the server is produced by getRateLimitWire(). The Settings panel reads/writes the same store, so changes take effect on the next request without a restart.

• Suggested defaults by provider

  These are conservative mid-tier defaults — adjust to match your actual plan. 0 = unlimited.

  Provider	TPM	RPM	RPD
  OpenAI (GPT-4o tier)	150,000	60	0
  Anthropic (Claude 3.5)	120,000	50	0
  Google (Gemini 1.5 Pro)	250,000	100	0
  Groq	30,000	30	14400
  DeepSeek	60,000	60	0
  OpenRouter	200,000	200	0
  Mistral	150,000	50	0
  Cohere	100,000	100	0
  Together AI	80,000	60	0
  Fireworks AI	80,000	60	0
  Perplexity	50,000	50	0
  AI21	60,000	50	0
  Ollama (local)	0	0	0
  LM Studio (local)	0	0	0

Common issues