Use Xantly with Guidance

Guidance is Microsoft's constrained-generation library, write templates with typed slots and loops, let the model fill them while respecting the structure. Its models.OpenAI class accepts a base_url kwarg, so pointing it at Xantly gives every guided generation the benefits of routing + cache + memory.

Prerequisites

• Python 3.9+
• pip install guidance
• A Xantly API key, create one

Setup

import guidance
from guidance import models

lm = models.OpenAI(
    "xantly/auto-quality",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)

Structured template

from guidance import gen, select

lm += f"""\
Here's a recipe for {{{{ dish }}}}:

Ingredients:
- {gen('ingredient_1', max_tokens=20)}
- {gen('ingredient_2', max_tokens=20)}
- {gen('ingredient_3', max_tokens=20)}

Difficulty: {select(['easy', 'medium', 'hard'], name='difficulty')}
"""

Guidance constrains the model to pick from the options in select() and limits token counts per slot.

Function / tool use

from guidance import assistant, user, system, gen

with system():
    lm += "You're a concise assistant."

with user():
    lm += "What's 17 * 23?"

with assistant():
    lm += gen("response", max_tokens=100)

print(lm["response"])

Regex-constrained generation

import guidance

@guidance
def phone_number(lm):
    lm += "Phone: " + guidance.gen(regex=r"\d{3}-\d{3}-\d{4}", name="phone")
    return lm

lm += phone_number()
print(lm["phone"])  # always matches the regex

JSON-schema-constrained

import json
from guidance import json as guidance_json

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
    },
    "required": ["name", "age"],
}

lm += "Describe a person: " + guidance_json(schema=schema, name="person")
print(json.loads(lm["person"]))

Per-block model selection

fast = models.OpenAI(
    "xantly/auto-speed",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
quality = models.OpenAI(
    "xantly/auto-quality",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)

# Use fast model for classifiers, quality for reasoning

Model choice

Verify

from guidance import gen, models

lm = models.OpenAI(
    "xantly/auto-speed",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
lm += "Say: " + gen("word", max_tokens=3)
print(lm["word"])

What you get

• Cache hits on template re-fills. Same template with different slot inputs → partial cache overlap via semantic matching.
• Waterfall fallback. Guidance templates don't crash mid-generation on provider outages.
• Cost visibility per template block. Each gen() call logs separately.
• Cross-provider consistency. Xantly translates constrained-generation semantics where the underlying provider supports it (OpenAI JSON mode, Anthropic structured output).

Gotchas

Guidance's full constraint features require models.Transformers (local). The remote models.OpenAI is limited to what OpenAI-compatible endpoints support, JSON mode, max_tokens, stop sequences, logit_bias. Complex regex/CFG constraints fall back to retry loops over the model.

base_url required with /v1.

Xantly doesn't support logit_bias override for non-OpenAI providers. If you use Guidance's constraints that rely on logit_bias, pin the model to openai/gpt-5.4 or another OpenAI model.

Streaming. Guidance streams natively; Xantly passes SSE through unchanged.

Next steps

• Instructor, simpler Pydantic-based structured output.
• DSPy, programmatic prompt optimization.
• PydanticAI, Pydantic's agent framework.
• OpenAI SDK (Python), lower-level control.