Intelligent Routing

Xantly's routing engine automatically selects the optimal model for each request based on task complexity, latency requirements, cost constraints, and historical performance. No configuration required, but full control when you want it.

Why routing matters

LLMs vary dramatically in cost, speed, and capability:

Sending every request to a frontier model wastes money on simple tasks. Sending everything to a cheap model degrades quality on hard ones. Intelligent routing solves this by matching each request to the right model automatically, and it re-measures the fleet continuously rather than relying on a fixed table.

How auto-routing works

When you send model: "auto", the gateway runs a three-step process:

Step 1: Task analysis

The request is classified by type and complexity:

• Task type: Code generation, factual QA, creative writing, structured extraction, tool calling, SQL, reasoning, or general-purpose
• Complexity signals: Message length, conversation depth, schema requirements, tool definitions, required capabilities

This classification happens in under 2ms and never adds perceptible latency.

Step 2: Model selection

A machine learning model evaluates the classified request against the available model pool. It considers:

• Task-model fit: Which models perform best for this task type
• Latency budget: How fast the response needs to be
• Cost efficiency: The cheapest model that meets quality requirements
• Memory context: Whether cached context reduces the need for a frontier model
• Provider health: Current availability and latency of each provider

The selection model improves continuously, it learns from every request which model performs best for which task type and tenant workload.

Step 3: Execution with fallback

The selected model receives the request. If the primary provider fails or times out, the gateway automatically falls back to the next best option. This waterfall mechanism ensures high availability without any client-side retry logic.

Request → Primary model → ✓ Response
                        → ✗ Timeout → Fallback model → ✓ Response
                                                      → ✗ Error → Second fallback → Response

How a model gets chosen

Two things decide every request, and they are kept separate on purpose.

Constraints are hard. Whether a model can take your tools, honour your JSON schema, read your images, hold your context, and satisfy your organization's compliance policy. A model either can or it cannot. Constraints are never traded away for price or speed.

The objective is what to optimize among the models that survive those constraints. That is what a routing mode sets.

Quality is not part of the objective. Every request carries a quality bar derived from how hard the request actually is, and no objective may serve below it. That is what keeps "cheapest" and "fastest" from ever meaning "worse than your request needs".

Controlling routing

Auto-routing works out of the box, but you can influence or override it.

Presets (simplest)

{
  "model": "auto",
  "routing_hints": { "mode": "cost_optimized" }
}

Fine-grained hints

{
  "model": "auto",
  "routing_hints": {
    "preference_dial": 0.2,
    "max_latency_ms": 500,
    "task_complexity": "standard"
  }
}

• preference_dial (0.0-1.0): Lower = cheaper/faster, higher = quality. Default: 0.5.
• max_latency_ms: Soft latency budget. Values under 500ms switch the objective to latency.
• task_complexity: trivial, standard, complex, expert. Overrides automatic complexity detection.

Direct model selection

Skip routing entirely by specifying a model:

{
  "model": "bedrock/gpt-oss-120b-1/us-east-1",
  "messages": [...]
}

This bypasses the routing engine and sends directly to the specified model.

Continuous learning

The routing engine improves with every request:

1. Each response generates a feedback signal: latency, cost, success/failure, and quality indicators
2. The model selection algorithm updates online: no batch retraining, no downtime
3. Per-tenant adaptation: the system learns your specific workload patterns
4. Drift detection: if a model's performance changes, the router adapts automatically

After processing a few hundred requests, the routing engine is typically 30-50% more cost-efficient than static model selection, while maintaining equivalent or better output quality.

Workflow routing for agent loops

Most LLM gateways route every API call in isolation. Agent frameworks like Claude Code, deepagents, and Cline fire dozens of related calls for a single user task, and routing each one without context wastes money and produces inconsistent quality.

Xantly's routing engine supports workflow-level routing: tell the gateway which calls belong to one agent task, and it will route them as a coordinated unit.

How it works

Add three optional headers to mark a call as part of a workflow:

x-xantly-agent-framework: deepagents          # any string identifier
x-xantly-agent-session:   sess-abc-123        # client-generated, stable across calls
x-xantly-workflow-budget-usd: 1.00            # optional per-workflow budget

You can also signal the workflow phase explicitly so the router biases tier selection accordingly:

x-xantly-workflow-phase: exploration          # cheap exploration calls
x-xantly-workflow-phase: refinement           # mid-workflow refinement
x-xantly-workflow-phase: synthesis            # final answer / synthesis turn
x-xantly-workflow-end: true                   # finalize after this call

When the gateway sees these headers, it:

• Tracks per-workflow state: call count, cumulative cost, phase signal, keyed by (tenant + framework + session)
• Couples per-workflow budget: soft warning at 70%, hard cap at 95% (returns an error so your agent knows to stop)
• Biases tier selection by phase: exploration → speed-tier, synthesis → quality-tier
• Issues retroactive credit at workflow end: the router learns which models contributed to the workflow's overall success, not just per-call latency, so cheap exploration turns that set up successful synthesis turns get rewarded for it

Why this matters

A typical agent task (50+ calls, $0.40 budget) routed call-by-call wastes 30-40% on overspecified models for cheap exploration steps. Workflow routing flips this: cheap-fast for the legwork, frontier-quality for the synthesis. Equal or better task completion at 60-70% of the cost.

No SDK changes required, just three HTTP headers from your agent framework.

Self-improving model selection

The routing engine continuously updates an internal capability matrix that estimates each model's strengths across multiple dimensions, reasoning, coding, tool use, structured output, knowledge, and others. The matrix refreshes weekly from public benchmark scores plus operator-curated data, and updates instantly when new models land in your catalog.

What this means in practice:

• Cold-start works from real evidence. A brand-new tenant's very first request is routed using benchmark-informed priors, not blind exploration.
• Per-query specialization. If a query is structurally similar to past queries that succeeded with a specific model, the router prefers that model, even if it's not the tenant's "usual" choice.
• Workload adaptation. A tenant whose traffic is 90% code generation learns within ~50 requests to prefer code-tuned models for code calls, while still routing reasoning calls to reasoning-tuned models.
• Drift protection. If a model silently degrades (a provider pushes a bad update), the router notices via online performance monitoring and routes away from it, typically within 20 observations.

You don't configure any of this, it happens in the background. The dashboard surfaces what the router learned about your traffic if you want to inspect it.

Decision audit and observability

Every routing decision generates a structured audit record visible in the dashboard:

• Selected model and tier with provider attribution
• Per-term score breakdown: capability match, cost penalty, latency penalty, reliability, policy compliance, so you can see why a model was chosen
• Alternatives considered: the top runners-up with their scores
• Filtered models: anything excluded by data-class, region, DPA, SLO, or budget policies, with the specific reason
• Matrix version stamp: which capability snapshot was active at decision time

The Admin Portal → Routing Intelligence dashboard surfaces this end-to-end:

• Aggregate KPIs (decision count, fallback rate, distinct tenants/models)
• Searchable per-decision drill-down with the full breakdown chart
• Capability matrix health (benchmarks, latent axes, version history)
• Policy compliance trends (which rules fired, when)
• Workflow + drift + regression-gate status

Enterprise auditors can trace any individual call from request → decision → score breakdown → outcome without leaving the dashboard.

Response headers

Every response includes routing metadata:

Use these headers to monitor routing decisions and tune your hints.

What's next

• Caching & Performance: How caching eliminates redundant LLM calls
• Cost-Optimized Routing Guide: Advanced routing patterns and controls
• Memory & Context: How memory makes routing smarter over time

Frequently Asked Questions

How does intelligent routing work?

When you send model: "auto", the routing engine analyzes 15 request parameters, including task type, complexity, message length, conversation depth, schema requirements, tool definitions, and required capabilities, classifies the request in under 2ms, and selects the optimal model from the available pool based on task-model fit, latency budget, cost efficiency, memory context, and real-time provider health.

What routing modes are available?

Xantly offers six routing modes: auto (the engine decides based on request analysis), fast (lowest measured latency among models that clear the request's quality bar), balanced (default, best quality per dollar), quality (the most capable models, cheapest within that frontier), cost_optimized (lowest cost among models that clear the quality bar), and free_models_only (biases hard toward the cheapest models available). You can also fine-tune with a preference_dial (0.0-1.0) for continuous control between cost and quality. Every mode is bounded by the request's quality bar, so none of them serves a model below what the request needs.

Can I force a specific model?

Yes. Set the model field to any provider/model identifier (e.g., bedrock/gpt-oss-120b-1/us-east-1 or anthropic/claude-3.5-sonnet) and the request bypasses the routing engine entirely, going directly to that specific model. You can also use routing_override.model for hard overrides while keeping other routing features like failover and caching active.

How reliable is the routing?

Xantly's routing engine achieves 99.999% routing reliability through waterfall fallback with 36+ API keys, automatic circuit breakers that proactively route around degraded providers, and multi-key rotation that handles billing exhaustion without request failures. If the primary model fails or times out, the gateway automatically falls back to the next best option, ensuring high availability with zero client-side retry logic.

How do I route an agent workflow as a unit instead of one call at a time?

Add the x-xantly-agent-framework, x-xantly-agent-session, and (optionally) x-xantly-workflow-budget-usd headers from your agent framework. The gateway tracks per-workflow state, call count, cumulative cost, phase signal, keyed by (tenant + framework + session), biases tier selection based on workflow phase (exploration/refinement/synthesis), and issues retroactive credit at workflow end so cheap exploration calls that lead to successful synthesis turns get rewarded. Typical agent workloads see 30-40% cost reduction at equal or better task completion. No SDK changes, just three HTTP headers.

Can I see why a specific request was routed where it was?

Yes. The Admin Portal → Routing Intelligence dashboard shows every routing decision with a per-term score breakdown, capability match, cost penalty, latency penalty, reliability, policy compliance, plus the alternatives the router considered, the models it filtered out (and why), and the capability matrix version active at decision time. Every call is auditable end-to-end without leaving the dashboard.

Does the routing engine adapt to my specific workload?

Yes, automatically. The selection algorithm updates online with every response, no batch retraining, no downtime. A tenant whose traffic is 90% code generation learns within roughly 50 requests to prefer code-tuned models for code calls. The internal capability matrix refreshes weekly to incorporate new public benchmark scores and any updates to your model catalog. If a model silently degrades (a provider pushes a bad update), the router notices via online performance monitoring and routes away from it within ~20 observations.