Platform Overview

How Xantly fits in your stack — the AI gateway architecture, request lifecycle, and key differentiators.

Xantly is an AI infrastructure layer that sits between your application and LLM providers. One API call. Automatic routing. Built-in caching. Persistent memory. Zero lock-in.

How Xantly fits in your stack

You change one line of code, your base_url, and get intelligent routing, caching, memory, and cost optimization across every major LLM provider.

What happens when you make a request

Every call to /v1/chat/completions goes through six stages:

1. Authentication

Your API key is verified and mapped to your organization. Rate limits, budgets, and permissions are enforced.

2. Cache check

The gateway checks if an identical or semantically similar request has been answered recently. Cache hits return instantly and are billed at a flat $0.25 per million tokens instead of the provider price. Response headers tell you what happened: x-xantly-cache-hit: true and x-xantly-cache-type: exact or semantic.

3. Intelligent routing

If the request isn't cached, the routing engine analyzes it, complexity, intent, required capabilities, and selects the optimal model. Simple factual queries go to fast, cheap models. Complex reasoning goes to frontier models. The routing model improves continuously from feedback.

4. Provider call

The selected provider receives your request. The gateway handles retries, failover, and hedging automatically. If a provider is down, traffic waterfall to the next best option.

5. Response delivery

Responses stream back to your application in real-time. Response headers include the model used, cost, latency, and routing metadata, full transparency.

6. Async learning

After the response is delivered (never blocking your request), the gateway:

Provider ecosystem

Xantly routes across major LLM providers. You don't need API keys for each, Xantly manages provider relationships. Or bring your own keys (BYOK) for any provider.

Xantly serves models through its own provider relationships, so you need no provider accounts and no keys of your own. The families available today include Claude, Gemini, Llama, Qwen, DeepSeek, Mistral, GPT-OSS, Nova and Gemma, across several hundred regional deployments.

The authoritative list is always the gateway itself:

curl -s https://api.xantly.com/v1/models -H "Authorization: Bearer $XANTLY_API_KEY"

Use model: "auto" and the routing engine selects per request. Or pass any id from that list to pin one; the gateway normalises the interface either way, so your code does not change.

Use model: "auto" and let the gateway choose. Or specify a model directly, it works either way.

Key differentiators

Intelligent routing

Not all tasks need the most expensive model. Xantly's routing engine matches each request to the right model based on complexity, latency requirements, and cost constraints. The result: same quality output at 40-70% lower cost.

Multi-layer caching

Identical and similar requests are served from cache, no provider call, a flat $0.25 per million tokens, sub-millisecond latency. For agentic workflows with repetitive patterns, this eliminates redundant LLM calls entirely.

Persistent memory

Every conversation enriches an organization-level knowledge base. The gateway learns your domain, remembers context across sessions, and assembles relevant knowledge automatically. The longer you use Xantly, the smarter it gets.

Zero integration effort

Xantly is fully compatible with the OpenAI SDK. Migration is one line:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.xantly.com/v1",  # ← only change
    api_key="your-xantly-api-key"
)

response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hello!"}]
)

Every OpenAI SDK feature works: streaming, tool calling, structured output, function calling, vision. No code changes beyond base_url.

Quality & reliability

Xantly is continuously validated through a multi-layer testing framework:

What's next

Frequently Asked Questions

What is an AI gateway?

An AI gateway is an infrastructure layer that sits between your application and LLM providers, handling routing, authentication, caching, and observability in a single unified API. Xantly acts as this gateway, your app sends one standard OpenAI-format request, and the gateway authenticates, checks cache, selects the optimal model, executes the call with automatic failover, and streams the response back with full cost and latency metadata.

How does Xantly reduce costs?

Xantly reduces costs through three reinforcing mechanisms: intelligent routing selects the cheapest model capable of handling each request (saving 40-70% vs. always using a frontier model), semantic caching serves repeat and near-repeat queries at a flat $0.25 per million tokens instead of the provider price (62% hit rate, sub-5ms responses), and persistent memory reduces per-request token counts by curating only the relevant context. Combined, these deliver up to 80% cost reduction for production workloads.

What happens if a provider goes down?

Xantly's waterfall fallback mechanism automatically retries with the next best model when a provider fails, times out, or returns an error. The gateway maintains 36+ provider API keys and performs multi-key rotation, so a single provider outage never blocks your requests. Failover decisions happen in under 2ms, and circuit breakers proactively route around degraded providers before they fail completely.

How much latency does Xantly add?

Xantly adds a median overhead of 12ms for routed requests, covering authentication, cache check, task classification, and model selection. Cache hits are even faster: exact matches return in under 5ms and semantic matches in under 20ms, both billed at the flat cache rate rather than the provider price. The cache check itself adds less than 2ms of overhead on a miss, so there is no performance penalty for having caching enabled.