Caching & Performance

Xantly's multi-layer caching infrastructure eliminates redundant LLM calls. Identical requests return instantly. Similar requests return from semantic cache. Agentic workflows with repetitive patterns see 40-70% cost reduction, automatically.

Why caching matters for AI

LLM calls are expensive and slow. A single frontier-model request costs $5-15 per million tokens and takes 1-3 seconds. In production agentic workflows, the same questions get asked repeatedly, customer support bots answering FAQ variations, code assistants generating similar patterns, data pipelines running identical extractions.

Without caching, you pay the provider's full price every time. With Xantly's caching, repeat and near-repeat requests return in under 5ms for a flat $0.25 per million tokens instead of the provider price.

Two types of cache hits

Exact match

When the same request (same messages, same parameters) is sent again, the cached response is returned immediately.

• Latency: Under 5ms
• Cost: Zero tokens consumed
• Response header: x-xantly-cache-type: exact

This catches identical requests from retry loops, page refreshes, and duplicate webhook triggers.

Semantic match

When a similar request is sent, same intent, slightly different wording, the gateway detects the similarity and returns the cached response.

• Latency: Under 20ms
• Cost: Zero tokens consumed
• Response header: x-xantly-cache-type: semantic

This catches rephrased questions, agentic loops with minor prompt variations, and template-based queries where only small details change.

# These would be semantic cache hits for each other:
"What is the capital of France?"
"Tell me the capital city of France"
"Capital of France?"

Multi-layer cache architecture

The caching infrastructure uses multiple layers optimized for different access patterns. Each layer is checked in order, the fastest layers are checked first.

If all cache layers miss, the request proceeds to the routing engine and LLM provider. After a successful response, the result is stored in all applicable cache layers for future hits.

Cross-conversation deduplication

In agentic workflows, the same question often appears across different conversation sessions. For example:

• A customer support agent handles 100 similar password reset requests per day
• A code assistant generates the same boilerplate across different projects
• A data pipeline extracts the same fields from similar documents

Xantly automatically detects these cross-conversation patterns. Even without explicit conversation_id linking, the system normalizes request intents and deduplicates across sessions. This means your 100th password-reset question is served from cache, not sent to the LLM.

Cache controls

Enabling/disabling cache

Cache is enabled by default. To disable for a specific request:

{
  "model": "auto",
  "messages": [...],
  "xantly": {
    "enable_cache": false
  }
}

Or use a header. Each one only tightens what your organization allows; none loosens it:

How long an entry lives

Seven days by default, set per organization (cache_ttl_seconds), up to thirty. A workflow that runs once a day or a few times a week hits on its next run; an hour would have missed every time. The semantic tier follows the same clock: a stored answer older than your lifetime is never served, and expired entries are swept out daily.

Checking cache status

Every response includes cache headers:

What's not cached

Some responses are never stored, because replaying them would be wrong rather than merely stale:

• A response that contains tool calls. A cached tool-call plan would be executed again against state that has since moved, which fails quietly rather than loudly. A request that merely declares a tools array is cached normally; the tool definitions are part of the cache key, so changing them is a different entry.
• JSON the caller did not ask for, which is a symptom of a mis-serving model rather than an answer worth reusing. A request that explicitly sets response_format.type: "json_schema" is cached normally.
• Single-value answers (one integer, one choice letter), where near-identical questions have different right answers.
• Requests carrying explicit cache-skip headers, and everything under intelligence_mode: "proxy".

Streaming is cached. A streamed response is stored once it completes and replayed as SSE on a later hit; only the synchronous read is skipped mid-stream.

Cost impact

Cached responses consume zero provider tokens. Instead of the provider's price, you pay a flat $0.25 per million tokens served from cache, on every cache layer. Cache hits don't count against your rate limits either.

For typical production workloads:

Cache hit rates improve over time as more responses are stored. The first few hundred requests build the cache; subsequent requests increasingly benefit.

Performance characteristics

Cache misses add negligible overhead, the cache check itself is extremely fast. There is no performance penalty for having caching enabled.

Response cache and prompt caching are different things

This page is Xantly's own response cache: model agnostic, it returns a stored answer without calling any model, and it is billed at the flat cache rate. Prompt caching is something the model providers do inside their own inference stacks: the processed prefix of a prompt (a long system message, a tool list, the conversation so far) is kept so the next call that repeats it is cheaper and faster, and the model is still called. It is per provider and per model by nature; Xantly turns it on wherever the upstream supports it and passes the discount through. See Prompt Caching.

What's next

• Prompt Caching: The provider-side prefix cache, and what Xantly does with it
• Memory & Context: How persistent memory goes beyond caching
• Intelligent Routing: How routing works with cache signals
• Cost-Optimized Routing Guide: Advanced cost reduction strategies

Frequently Asked Questions

What is semantic caching?

Semantic caching matches requests by meaning rather than exact text, so "What is the capital of France?" and "Tell me the capital city of France" return the same cached response. Xantly's semantic cache achieves a 62% hit rate for typical production workloads, returning results in under 20ms for a flat $0.25 per million tokens. It uses cosine similarity with Jaccard safety thresholds to ensure only genuinely equivalent queries are matched.

How does caching reduce costs?

Cache hits skip the provider entirely: you pay a flat $0.25 per million tokens instead of the provider's price, and they don't count against your rate limits. For production workloads with repetitive patterns (customer support bots, code assistants, data pipelines), caching delivers 40-80% cost reduction automatically. Cache hit rates improve over time as more responses are stored, and cross-conversation deduplication means your 100th similar question is served from cache, not sent to the LLM.

Can I disable caching?

Yes. You can disable caching per-request by setting xantly.enable_cache: false in the request body, or by sending the header x-xantly-cache: off. You can also use intelligence_mode: proxy (the X-Intelligence-Mode: proxy header, or xantly.intelligence_mode in the body) to bypass the entire caching and memory pipeline for maximum raw speed with zero overhead. Response headers (x-xantly-cache-hit, x-xantly-cache-type) always tell you whether a response was served from cache.

What's the difference between exact and semantic cache?

Exact cache matches identical requests (same messages, same parameters) and returns in under 1ms from the in-process layer or under 5ms from the distributed layer, it catches retry loops, page refreshes, and duplicate webhook triggers. Semantic cache matches requests with the same intent but different wording and returns in under 20ms, it catches rephrased questions, FAQ variations, and template-based queries where only small details change.