Xantly documentation

Reference for the Xantly AI gateway: a single OpenAI-compatible endpoint that routes to 10,000+ models with semantic caching, persistent memory and automatic failover. Start with the Quickstart if you have an API key, or the Introduction if you do not.

Every page here is also available as raw Markdown: append .md to the URL, or send Accept: text/markdown.

Getting Started

• Authentication: Learn how to authenticate requests to the Xantly API using API keys and JWT tokens.
• Introduction: Xantly routes to 10,000+ LLMs with 12ms median overhead, semantic caching (sub-5ms), and persistent memory — reducing API costs by up to 80%.
• Quickstart: Get up and running with Xantly in under 5 minutes. Change your base_url, send your first request, and see intelligent routing in action.

Architecture

• Caching & Performance: Multi-layer caching with 62% semantic hit rate — exact match (sub-5ms), semantic match, and cross-conversation deduplication for up to 80% cost reduction.
• Intelligent Routing: How Xantly analyzes each request across 15 parameters and selects the optimal model from 10,000+ options with 99.999% routing reliability.
• Memory & Context: Persistent per-organization memory — automatic session detection, knowledge extraction, and intelligent context assembly.
• Platform Overview: How Xantly fits in your stack — the AI gateway architecture, request lifecycle, and key differentiators.

API Reference

• Audio: Speech-to-text transcription (Whisper) and text-to-speech generation with BYOK support.
• Billing & Credits: Prepaid credits, the 3% top-up fee, usage at provider cost, cache pricing, budget caps, and cost visibility per response.
• Chat Completions: Chat Completions API — route to 10,000+ LLMs via /v1/chat/completions with 12ms median overhead, intelligent routing, and semantic caching.
• Completions (Legacy): Legacy prompt-based text completions. Translates to chat completions internally for full routing support.
• Compliant Mode: Enterprise data protection for regulated workloads: route only to a verified model catalog on certified platforms (SOC 2, ISO 27001, HIPAA BAA), with zero data retention, region pinning, and a tamper-evident audit trail.
• Embeddings: Create vector embeddings for semantic search, RAG, and clustering via the OpenAI-compatible Embeddings API with intelligent provider routing.
• Images: Generate images from text prompts using DALL-E through the OpenAI-compatible API with BYOK support.
• Intelligence Modes: Control pipeline stages per request — proxy (raw speed, ~0ms overhead), cache (40-60% cost savings), or full (memory + personalization).
• Models: List all available models in the Xantly catalog — chat, completion, and embedding models.
• Moderations: Classify text for potentially harmful content using the OpenAI-compatible moderation API with BYOK support.
• OpenAPI Spec: Download the full OpenAPI 3.1 specification and import it into your favourite HTTP client.
• Rate Limits: Per-organization RPM and TPM limits, voice limits, response headers, and recommended backoff patterns.
• Responses API: Modern OpenAI Responses API endpoint — drop-in compatible with newer SDK versions that default to /v1/responses.
• Voice Billing: How voice requests are priced and surfaced on invoices: provider passthrough per minute and per character, prepaid credits, limits, and metering.
• Voice Models Catalog: All 30+ voice models available via the Xantly Voice API, grouped by type (STT, TTS, Realtime, Audio LLM, Music) with pricing units and provider info.

Guides

• Bring Your Own Key (BYOK): Use your own provider API keys through Xantly for direct billing and full cost control while keeping routing and caching.
• Cost-Optimized Routing: Route requests to the cheapest qualifying model — achieve 40-70% cost savings with 5 routing modes and fine-grained preference dial control.
• Multi-Agent Orchestration: Build AI agent pipelines with tool use, shared persistent memory, automatic handoffs, and workflow type hints across 10,000+ models.
• Streaming Responses: Stream tokens in real-time using server-sent events from any model in your Xantly pipeline.
• Voice Agents: Build production voice agents that are 80-90% cheaper than direct API calls — with sub-300ms latency, semantic caching, and built-in memory.
• XRB-1000: Xantly Routing Benchmark v1, September 2026. 1,000 questions to xantly/auto with no model pinned, against the ten most accurate models pinned on the same questions. Every number measured on the platform.

Reference

• Glossary: Definitions of key AI infrastructure terms, AI Gateway, Intelligent Routing, Semantic Caching, BYOK, Waterfall Fallback, and more.
• Security & Compliance: How Xantly protects your data: encryption in transit and at rest, the tiered content retention model and its Zero-Retention option, the guarantee that prompts are never used to train a language model, access controls, 72 hour breach notification, the sub-processor list, and certification status.
• Versioning & Deprecation Policy: How the Xantly API is versioned, what counts as a breaking change, and how a deprecation is signalled: Deprecation and Sunset response headers, the OpenAPI deprecated flag, and the minimum notice period before removal.

Integrations

• Migrate from Antigravity to Xantly + OpenCode: Google cut Antigravity's free tier by 92% in December 2025. Replace it with OpenCode + Xantly in under 5 minutes and regain access to 10,000+ models.
• Migrate from Cursor to Xantly + Cline: Cursor's BYOK is chat-only, Agent, Edit, and Tab all blocked by proprietary models. Escape to Cline + Xantly for full-stack BYOK everywhere in 10 minutes.
• Use Xantly alongside llamafile: Mozilla's single-binary llamafile runtime pairs with Xantly for a hybrid local + hosted setup. Offline private inference + cloud reasoning for the hard stuff.
• Use Xantly alongside Ollama: Hybrid local + hosted pattern, Ollama for the keystroke firehose (autocomplete), Xantly for the hard stuff (agents, reasoning). The two systems split cleanly.
• Use Xantly with Aide (codestory.ai): Aide from codestory.ai uses a local Rust sidecar for LLM dispatch. Its provider-config block accepts OpenAI-compatible endpoints, point it at Xantly for routing on chat, Cmd+K, and agent mode.
• Use Xantly with Aider (CLI): Aider is a terminal pair-programmer with a clever main/editor/weak model split. One --openai-api-base flag points it at Xantly for 60-70% cost savings via role-split routing.
• Use Xantly with AutoGen: Microsoft AutoGen's OpenAIChatCompletionClient accepts base_url. Point it at Xantly for multi-agent routing, cache, and waterfall fallback.
• Use Xantly with Claude Code: Point Anthropic's Claude Code CLI at Xantly via ANTHROPIC_BASE_URL and get smart routing, semantic cache, memory, and waterfall fallback on every chat/edit/agent call. Full Anthropic Messages API compatibility.
• Use Xantly with Cline (VS Code): Cline is the most popular AI coding agent for VS Code. Drop in Xantly as the OpenAI-compatible provider for BaRP routing, L2 semantic cache, and persistent memory across agent sessions.
• Use Xantly with Continue.dev: Continue.dev's per-role model config pairs perfectly with Xantly, route chat/edit/apply to Xantly Quality, autocomplete to a free local Ollama. Best mix-and-match demo in the ecosystem.
• Use Xantly with CrewAI: CrewAI's multi-agent framework routes through LiteLLM, so any OpenAI-compatible endpoint plugs in. Xantly pairs perfectly, semantic cache is disproportionately valuable for multi-agent workloads.
• Use Xantly with Cursor (chat-only BYOK): Cursor's BYOK is chat-only, Agent, Edit, and Tab all bypass it. If you must stay in Cursor, here's the real-world config with the known base-URL override bug documented.
• Use Xantly with Deep Agents: LangChain's Deep Agents framework, planner + sub-agent delegation + virtual file system. Routes through Xantly for per-role tier routing across 50-200 calls per run. Quality planner, Value sub-agents, Speed summarizers.
• Use Xantly with DSPy: Stanford DSPy's programming-not-prompting framework routes via LiteLLM. Point dspy.LM at Xantly and optimizer runs hit semantic cache hard, 40-70% cost savings on BootstrapFewShot compilation.
• Use Xantly with Gemini CLI: Google's Gemini CLI doesn't yet support custom base URLs. Until it does, the best pattern is OpenCode routing to Gemini via Xantly, here's how.
• Use Xantly with GitHub Copilot CLI: Skip the $10/mo Copilot subscription. Four env vars and Copilot CLI routes through Xantly for smart model selection, semantic cache, and cost control, no GitHub auth required.
• Use Xantly with Guidance: Microsoft Guidance's constrained-generation library takes a base_url on models.OpenAI. Point it at Xantly for typed-slot templates with routing, cache, and waterfall.
• Use Xantly with Instructor: Instructor patches the OpenAI or Anthropic client to return typed Pydantic objects with auto-retry. Wrapping a Xantly-configured client gives you routing + cache on every structured extraction.
• Use Xantly with JetBrains AI Assistant + Junie: JetBrains AI Assistant (IntelliJ, PyCharm, WebStorm, etc.) and agent-mode Junie support custom OpenAI-compatible providers since 2024.3. Drop in Xantly for routing on chat, inline edit, and Junie agent runs.
• Use Xantly with Kilo Code: Kilo Code is the #1 IDE Extension on OpenRouter by token volume. Routing through Xantly adds bandit routing, semantic cache, and USD budget caps for heavy-usage patterns.
• Use Xantly with LangChain (Python): LangChain's ChatOpenAI and ChatAnthropic accept custom base URLs. Point them at Xantly and every chain, agent, and RAG pipeline gets routing, cache, and memory.
• Use Xantly with LangChain.js: @langchain/openai and @langchain/anthropic both accept configuration.baseURL, route every chain, agent, and RAG pipeline through Xantly.
• Use Xantly with LangGraph: LangGraph's graph-based agent runtime uses LangChain ChatOpenAI/ChatAnthropic under the hood. Route every node through Xantly and get per-node tier routing with cache and waterfall.
• Use Xantly with LlamaIndex: LlamaIndex's OpenAI integration takes an api_base argument. Point it at Xantly for smart routing on every RAG pipeline, query engine, and agent.
• Use Xantly with n8n: n8n's OpenAI Chat Model node accepts a custom base URL. Every workflow that touches LLMs, AI Agent, RAG, HTTP Request, routes through Xantly for BaRP routing, cache, and budget caps.
• Use Xantly with OpenCode: sst/opencode is the cleanest BYO story in the AI coding CLI ecosystem. One opencode.jsonc block pointed at Xantly and every feature, chat, edit, agent, streaming, tools, just works.
• Use Xantly with OpenHands: OpenHands (formerly OpenDevin) is an autonomous coding agent. LiteLLM-backed, any OpenAI-compatible gateway just works. Three env vars and it routes through Xantly.
• Use Xantly with PydanticAI: PydanticAI's OpenAIModel accepts a base_url via OpenAIProvider. Point it at Xantly for typed agents with smart routing, cache, and memory.
• Use Xantly with Raycast (AI extensions): Raycast AI itself is hosted, but open-source extensions that call LLMs via the Node SDK can route through Xantly. Works for both extension users and authors.
• Use Xantly with Roo Code: Roo Code is a Cline fork with custom per-mode models. Pair it with Xantly to route Code mode to Claude, Architect to GPT, and commit-message to fast speed-tier models.
• Use Xantly with the Anthropic Python SDK: Xantly ships a byte-accurate /v1/messages endpoint. The official anthropic Python SDK points at it with base_url, messages.create, streaming, tool_use, prompt caching all work.
• Use Xantly with the Anthropic TypeScript SDK: The @anthropic-ai/sdk npm package accepts a baseURL option. Point it at Xantly's /v1/messages for smart routing, semantic cache, streaming, tool use, and prompt caching.
• Use Xantly with the OpenAI Python SDK: Point the official openai Python SDK at Xantly with base_url. Chat, streaming, function calling, structured outputs, and async all work untouched.
• Use Xantly with the OpenAI TypeScript SDK: The openai npm package accepts a baseURL option. Point it at Xantly for smart routing, semantic cache, streaming, and tool calling, works in Node, Bun, Deno, and Edge.
• Use Xantly with the Vercel AI SDK: The Vercel AI SDK's @ai-sdk/openai-compatible provider plugs into Xantly in one line. generateText, streamText, generateObject, agents, useChat, all routed.
• Use Xantly with Void Editor: Void is the OSS alternative to Cursor, same editor fork, every API call goes to whatever provider you configure. Works with Xantly via env vars or the OpenAI Compatible settings.
• Use Xantly with Zapier (OpenAI connector): Zapier's OpenAI app supports a Custom Base URL. Route every Zap through Xantly for massive cache savings on high-volume classifier and templated workflows.
• Use Xantly with Zed: Zed's assistant panel supports an openai_compatible provider block in settings.json. Drop in Xantly for BaRP routing on inline chat, edit, and the inline assistant.
• Windsurf BYOK, use Xantly + Cline instead: Windsurf (Codeium) hard-codes its AI endpoint, no custom base URL, no full BYOK. This page explains why Xantly can't plug in directly, and points you at working alternatives (Cline, Zed).
• Xantly MCP Server, an MCP Surface Over the Gateway: A Model Context Protocol surface over the Xantly gateway routing, cache, memory, cost, and control internals. 15 tools, hosted at https://api.xantly.com/mcp over Streamable HTTP. Connect a client with your Xantly API key as a bearer token.