Rate Limits

Xantly enforces per-organization rate limits using a distributed sliding window algorithm. Limits are applied per endpoint category and per minute.

How rate limiting works

• Sliding window: enforced per organization per endpoint via Redis atomic Lua scripts
• Dual enforcement for inference: chat completions and embeddings check both RPM (requests per minute) and TPM (tokens per minute) simultaneously to prevent quota burn attacks
• Other endpoints: RPM only

Default limits by endpoint

Rate limits are operational guardrails set per organization, not pricing tiers. Every organization starts with sensible defaults, and limits can be raised from the dashboard or through an enterprise contract. Check your dashboard for your current limits.

Voice rate limits

Voice endpoints (/v1/voice/transcribe, /v1/voice/synthesize, /v1/voice/chat, /v1/voice/stream, /v1/voice/realtime, /v1/voice/turn) have three independent enforcement dimensions, all evaluated on every request:

1. Voice RPM: sliding-window requests-per-minute, identical mechanism to inference RPM but a separate counter
2. Monthly audio minutes quota: total voice_input_audio_ms summed across the calendar month
3. Concurrent voice sessions: distinct active sessions in the voice:concurrent:{org_id} Redis ZSET

If any one of these is exceeded, the request returns 429 Too Many Requests (or 402 Payment Required for budget/credit exhaustion).

Per-org defaults and overrides

Every organization gets default voice limits (RPM, monthly audio minutes, concurrent sessions, and an idle-session concurrency TTL), and each one can be overridden per organization via org_settings:

The concurrency TTL is how long an idle voice session remains in the active set before the reaper releases its slot. Long-running streaming sessions get heartbeats from the WebSocket handler every turn so they aren't reaped mid-call. Voice limits are guardrails, not pricing: raising them never changes the metered rate.

Enforcement order

When a voice request arrives, Xantly evaluates the checks in this order. The first failure returns immediately:

1. Monthly USD budget cap: same budget:usage:{org}:general:{YYYY-MM} Redis pool as inference. Returns 402 Payment Required.
2. Monthly voice minutes limit: voice:audio_mins:{org}:{YYYY-MM} Redis key vs the org limit. Returns 429.
3. Credit floor: accounts must have credit_balance_cents >= 5 ($0.05) to start a request. Returns 402.
4. Concurrent session limit: Lua script checks ZCARD against the org limit. Returns 429.
5. Voice RPM (sliding window): middleware-level RPM enforcement. Returns 429.

Cost-aware sub-pipeline

Inside the voice pipeline, the LLM stage uses BaRP (the existing routing fairness layer) with a VoiceOnly constraint that prefers models with avg_latency_ms < 300. This sub-pipeline does not consume the voice RPM bucket, only the outer voice request does.

Rate limit headers for voice

Voice endpoints return the same standard headers as other endpoints:

For per-request voice cost + model headers, see Voice Billing and Voice Agents.

Rate limit response headers

Every response includes headers showing your current rate limit status:

Every response carries two families of rate limit headers. They are not aliases of each other: the two reset headers use different units, and reading the wrong one will schedule your next attempt decades away or immediately.

The X- family is the older GitHub-style convention. The unprefixed family follows the IETF RateLimit header fields draft, which specifies delta seconds.

Which to use. Prefer RateLimit-Reset, because a delta needs no clock agreement between your machine and ours. Prefer Retry-After over both when it is present.

The window is rolling

The limiter uses a rolling 60 second window, not a fixed one that empties on a clock boundary. On a successful request RateLimit-Reset marks the end of the current window rather than counting down, so under sustained load it stays pinned at 60 while RateLimit-Remaining moves. That is expected. On a 429 it is a genuine countdown to when capacity returns.

When a limit is exceeded

X-Xantly-Throttle-Source is worth branching on. plan means you are over your own limit and backing off will help. upstream means a model provider is throttling us, and a different model may succeed immediately. gateway means the gateway itself is at capacity.

Concurrency

Requests per minute is not the only ceiling. There is also a limit on how many requests may be in flight across the gateway at one time:

The scope label matters. Every other X-RateLimit-* header describes your own quota; this one describes a shared resource, so spare capacity in it is not headroom you own. Exceeding it returns a 429 with X-Xantly-Throttle-Source: gateway.

If you are sizing a worker pool, size it by measuring where your own error rate starts rising, not by dividing a published number. Concurrency, not requests per minute, is usually what binds first on agent workloads.

Rate limit exceeded, 429 response

{
  "error": {
    "message": "Rate limit exceeded: 1000 requests per minute",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  }
}

Handling rate limits

Exponential backoff with jitter (recommended)

import time
import random
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["XANTLY_API_KEY"],
    base_url="https://api.xantly.com/v1",
)

def chat_with_backoff(messages, max_retries=6):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                model="auto",
                messages=messages,
            )
        except Exception as e:
            if "429" in str(e) or "rate_limit" in str(e):
                wait = (2 ** attempt) + random.uniform(0, 1)
                print(f"Rate limited. Waiting {wait:.1f}s...")
                time.sleep(wait)
            else:
                raise
    raise Exception("Max retries exceeded")

async function chatWithBackoff(messages, maxRetries = 6) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await client.chat.completions.create({ model: "auto", messages });
    } catch (err) {
      if (err?.status === 429) {
        const wait = Math.pow(2, attempt) * 1000 + Math.random() * 1000;
        console.log(`Rate limited. Waiting ${(wait / 1000).toFixed(1)}s...`);
        await new Promise((r) => setTimeout(r, wait));
      } else {
        throw err;
      }
    }
  }
  throw new Error("Max retries exceeded");
}

Inspect headers before retrying

import httpx

response = httpx.post(
    "https://api.xantly.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
    json={"model": "auto", "messages": [{"role": "user", "content": "Hello"}]},
)

remaining = int(response.headers.get("X-RateLimit-Remaining", 0))
if remaining < 10:
    print(f"Warning: only {remaining} requests remaining in this window")

Best practices

1. Use model: "auto": the gateway's intelligent routing can reuse cached responses and distribute load across providers, reducing your effective RPM.
2. Enable semantic caching (xantly.enable_cache: true, default), cache hits don't count against your rate limits and are billed at a flat $0.25 per million tokens instead of the provider price.
3. Batch when possible: combine multiple inputs into a single embeddings request.
4. Monitor headers: log X-RateLimit-Remaining in production to catch approaching limits before they hit.
5. Use service_tier: "batch": signals a cost/latency preference that may take advantage of off-peak capacity.

Next steps

• Billing & Credits, Prepaid credits and monthly budget caps
• Chat Completions, Main inference endpoint