# Use Xantly with Guidance

[Guidance](https://github.com/guidance-ai/guidance) is Microsoft's constrained-generation library, write templates with typed slots and loops, let the model fill them while respecting the structure. Its `models.OpenAI` class accepts a `base_url` kwarg, so pointing it at Xantly gives every guided generation the benefits of routing + cache + memory.

## Prerequisites

- Python 3.9+
- `pip install guidance`
- A Xantly API key, [create one](/dashboard/api-keys)

## Setup

```python
import guidance
from guidance import models

lm = models.OpenAI(
    "xantly/auto-quality",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
```

## Structured template

```python
from guidance import gen, select

lm += f"""\
Here's a recipe for {{{{ dish }}}}:

Ingredients:
- {gen('ingredient_1', max_tokens=20)}
- {gen('ingredient_2', max_tokens=20)}
- {gen('ingredient_3', max_tokens=20)}

Difficulty: {select(['easy', 'medium', 'hard'], name='difficulty')}
"""
```

Guidance constrains the model to pick from the options in `select()` and limits token counts per slot.

## Function / tool use

```python
from guidance import assistant, user, system, gen

with system():
    lm += "You're a concise assistant."

with user():
    lm += "What's 17 * 23?"

with assistant():
    lm += gen("response", max_tokens=100)

print(lm["response"])
```

## Regex-constrained generation

```python
import guidance

@guidance
def phone_number(lm):
    lm += "Phone: " + guidance.gen(regex=r"\d{3}-\d{3}-\d{4}", name="phone")
    return lm

lm += phone_number()
print(lm["phone"])  # always matches the regex
```

## JSON-schema-constrained

```python
import json
from guidance import json as guidance_json

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
    },
    "required": ["name", "age"],
}

lm += "Describe a person: " + guidance_json(schema=schema, name="person")
print(json.loads(lm["person"]))
```

## Per-block model selection

```python
fast = models.OpenAI(
    "xantly/auto-speed",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
quality = models.OpenAI(
    "xantly/auto-quality",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)

# Use fast model for classifiers, quality for reasoning
```

## Model choice

| Model ID | When |
|---|---|
| `xantly/auto-quality` | Constrained reasoning, complex templates. |
| `xantly/auto-value` | High-volume template filling. |
| `openai/gpt-5.4` | Strict schema adherence. |
| `xantly/auto-speed` | Simple `select()` classifiers. |

## Verify

```python
from guidance import gen, models

lm = models.OpenAI(
    "xantly/auto-speed",
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
lm += "Say: " + gen("word", max_tokens=3)
print(lm["word"])
```

## What you get

- **Cache hits on template re-fills.** Same template with different slot inputs → partial cache overlap via semantic matching.
- **Waterfall fallback.** Guidance templates don't crash mid-generation on provider outages.
- **Cost visibility per template block.** Each `gen()` call logs separately.
- **Cross-provider consistency.** Xantly translates constrained-generation semantics where the underlying provider supports it (OpenAI JSON mode, Anthropic structured output).

## Gotchas

**Guidance's full constraint features require `models.Transformers` (local).** The remote `models.OpenAI` is limited to what OpenAI-compatible endpoints support, JSON mode, max_tokens, stop sequences, logit_bias. Complex regex/CFG constraints fall back to retry loops over the model.

**`base_url` required with `/v1`.**

**Xantly doesn't support logit_bias override for non-OpenAI providers.** If you use Guidance's constraints that rely on logit_bias, pin the model to `openai/gpt-5.4` or another OpenAI model.

**Streaming.** Guidance streams natively; Xantly passes SSE through unchanged.

## Next steps

- [Instructor](/docs/use-with-instructor), simpler Pydantic-based structured output.
- [DSPy](/docs/use-with-dspy), programmatic prompt optimization.
- [PydanticAI](/docs/use-with-pydantic-ai), Pydantic's agent framework.
- [OpenAI SDK (Python)](/docs/use-with-openai-sdk-python), lower-level control.