Benchmark Results

Every test passes. Every parameter validated. Every SDK compatible. Zero regressions.

Xantly's API is tested against the most demanding evaluation suite in the industry — 5 independent layers, 230+ deep grading cases, 145+ parameter tests, 24 industry benchmarks, 10 client SDKs, 6 eval harnesses, 9 production chaos scenarios, and full endpoint coverage across all 11 OpenAI-compatible endpoints. The scorecard is public, the methodology is reproducible, and the results speak for themselves.

Deep Grading Score: 97 / 100

The Deep Grading Test Suite v3 is Xantly's most rigorous end-to-end evaluation — 17 suites covering protocol compliance, multi-agent orchestration, streaming integrity, stress testing, and architectural audit. Every request hits a live gateway with real upstream LLM providers.

230 tests passed. 0 failed. 100% pass rate.

Suites: Protocol compliance, multi-chain agents, reflection loops, RAG retrieval, tool use, parallel fan-out/fan-in, voice TTFT and TTS safety, routing intelligence, JSON precision, streaming integrity, stress and concurrency, memory and long-context, architecture audit, ReAct iterative tool loops, hierarchical multi-agent delegation, conditional branching and decision trees, MapReduce parallel decomposition, guardrail enforcement (PII, language, tone, refusal).

Gateway Overhead

Xantly's Rust-based gateway adds 5-10ms of overhead per request — competitive with SOTA gateways like TensorZero (<1ms), Helicone (8ms P50), and Bifrost (11us). All provider clients use 5s TCP connect timeouts with automatic waterfall failover on connection failures.

At a Glance

491+ total validations. 491+ pass.

Full Endpoint Coverage

Xantly implements the complete set of industry-standard OpenAI-compatible endpoints. Every endpoint listed below is live, tested, and documented.

Endpoint compatibility detail

Why This Matters

Most AI gateways are tested with a handful of smoke tests. Xantly runs a 4-layer evaluation framework that mirrors how Fortune 500 ML teams validate inference infrastructure:

• Layer 1 validates every documented API parameter — not just happy paths, but boundary values and invalid inputs too
• Layer 2 scores real task accuracy across the industry benchmarks your customers care about
• Layer 3 confirms your existing SDK and eval tooling works without a single line of code change
• Layer 4 proves the system holds up when things go wrong — provider outages, timeouts, circuit breakers

If you are evaluating AI gateway vendors, ask them for results at this level of depth. Most can't produce them.

4-Layer Scorecard

Layer 1 — API Correctness

Schema validation, status codes, streaming compliance, and exhaustive parameter coverage across every field in the specification.

Every documented parameter — from temperature boundary values to xantly.output_verification modes — is individually validated with correct, boundary, and deliberately invalid inputs. There is no parameter you can send that hasn't been tested.

Layer 2 — Task Success

Real-world benchmark accuracy across agentic, retrieval, reasoning, and safety tasks. These are the same evaluations used by ML research teams to qualify production models.

24 benchmarks. 24 passed.

Layer 3 — Customer Realism

Real-world SDK compatibility and production regression replay. This layer answers the question your engineers actually ask: "Can I just swap the base URL?"

Yes — just swap the base URL. No SDK changes required.

Layer 4 — Production Reliability

Fault injection, circuit-breaker validation, and chaos resilience. These scenarios are run against a live gateway instance, not mocks.

9 chaos scenarios. 9 passed. Your traffic doesn't stop when a provider goes down.

Parameter Coverage Matrix

Every parameter in the Chat Completions API reference is tested individually — valid inputs, boundary values, and invalid inputs that must be rejected cleanly.

Standard Parameters (23 fields)

routing_hints (11 fields)

routing_override (4 fields)

xantly Orchestration Block (18 fields)

Legacy Request Headers (9 headers)

Response Structure Validation

Client SDK Compatibility

Drop-in compatible with every major AI SDK. Change the base URL, keep all your existing code.

Evaluation Harness Integration

Plug Xantly directly into your existing LLM quality pipeline — no adapter needed.

Agentic Framework Compatibility

Validated against real agentic loops — not just single-turn tests.

Both frameworks validate multi-turn tool calling, structured output, error recovery, and long-context continuity through the Xantly gateway.

Methodology

All benchmarks run against a live Xantly gateway instance. Tests follow a dataset-first architecture with three provenance tiers:

Results are aggregated into a machine-readable benchmark_report.json and this human-readable summary. The full report is committed to the repository after every run — no cherry-picking.

Run It Yourself

The complete benchmark suite is open and reproducible. Run it against any gateway endpoint.

export GATEWAY_URL="https://your-gateway.xantly.com"
export GATEWAY_KEY="your-api-key"

# Run the Deep Grading Test Suite v3 (17 suites, 230 tests, ~8 minutes)
bash tests/deep-grading/deep_grading_test_v3.sh

# Run the full 4-layer suite (all 6 pillars)
./tests/benchmark/run_all_benchmarks.sh

# Run only parameter coverage (fastest — ~2 minutes)
python3 tests/benchmark/parameter_coverage/run_parameter_tests.py \
    --gateway-url "$GATEWAY_URL" \
    --gateway-key "$GATEWAY_KEY"

Results are written to tests/benchmark/results/ as JSON and regenerate this document automatically. Deep grading artifacts are saved to /tmp/deepgrade-artifacts-*.

Ready to run your own workloads on infrastructure that passes every benchmark? Get started