Use Xantly with the OpenAI Python SDK

The official openai Python library ships a base_url argument on every client. Point it at Xantly once and every downstream call, chat completions, streaming, tools, structured outputs, async, routes through smart routing, semantic cache, and memory. Zero code changes beyond the two kwargs.

Prerequisites

• Python 3.8+
• pip install openai (version 1.0.0+)
• A Xantly API key, create one

Setup

from openai import OpenAI

client = OpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)

That's the entire migration. Every existing OpenAI call keeps working, client.chat.completions.create(...), client.embeddings.create(...), all of it.

Persist in env vars if you don't want to edit your code:

export OPENAI_BASE_URL=https://api.xantly.com/v1
export OPENAI_API_KEY=xantly_sk_...

Then OpenAI() with no arguments picks them up.

Chat completions

resp = client.chat.completions.create(
    model="xantly/auto-quality",
    messages=[{"role": "user", "content": "Write a bubble sort in Rust."}],
)
print(resp.choices[0].message.content)

Streaming

Works exactly like OpenAI's SSE shape, Xantly normalizes every provider back into OpenAI events:

stream = client.chat.completions.create(
    model="xantly/auto-quality",
    messages=[{"role": "user", "content": "Stream a haiku."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

Function / tool calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

resp = client.chat.completions.create(
    model="anthropic/claude-sonnet-4.6",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
)
tool_call = resp.choices[0].message.tool_calls[0]
print(tool_call.function.name, tool_call.function.arguments)

Xantly translates tool-call format across providers, so you can swap model= from OpenAI to Claude to Groq and the same tool schema keeps working.

Structured outputs (response_format)

from pydantic import BaseModel

class ExtractedUser(BaseModel):
    name: str
    email: str
    role: str

resp = client.beta.chat.completions.parse(
    model="openai/gpt-5.4",
    messages=[{"role": "user", "content": "Extract: Jane Doe, [email protected], CTO"}],
    response_format=ExtractedUser,
)
print(resp.choices[0].message.parsed)

Async client

import asyncio
from openai import AsyncOpenAI

aclient = AsyncOpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)

async def main():
    resp = await aclient.chat.completions.create(
        model="xantly/auto-value",
        messages=[{"role": "user", "content": "Ping"}],
    )
    print(resp.choices[0].message.content)

asyncio.run(main())

Model choice

Verify

resp = client.chat.completions.create(
    model="xantly/auto-speed",
    messages=[{"role": "user", "content": "say pong"}],
)
print(resp.choices[0].message.content)
# Check response headers for routing info:
print(resp._response_headers.get("x-xantly-model-id"))
print(resp._response_headers.get("x-xantly-cache-status"))

What you get

• Drop-in compatibility. No new SDK, no new mental model.
• Waterfall fallback. Provider outages don't propagate up, Xantly retries on the next-best model in the same tier.
• Semantic cache. Repeat prompts hit L2 cache at near-zero cost.
• Memory. Opt in per request via extra_headers={"X-Xantly-Memory-Mode": "recall"}.
• Cost transparency. Every call logs model + cache status + USD cost to your dashboard.

Gotchas

base_url must include /v1. The SDK appends paths like /chat/completions to it. Bare https://api.xantly.com won't work.

The openai library v0.x is EOL. You need v1.0.0+ (pip install -U openai). The old openai.api_base global was removed.

Azure-style clients. If you use AzureOpenAI(...), switch back to OpenAI(base_url=...). Xantly isn't Azure-compatible (yet).

Retries. The SDK retries on 5xx. Xantly's own waterfall already does this smarter, set max_retries=0 on the client to avoid double retries.

Next steps

• OpenAI SDK (TypeScript), the Node.js equivalent.
• Streaming Responses, SSE format details.
• Cost-Optimized Routing, how to keep bills down.
• Bring Your Own Key, route through your own OpenAI credits.