XRB-1000

XRB-1000 is the Xantly Routing Benchmark: 1,000 questions from eight public datasets, sent once each to xantly/auto with no model pinned, and answered by every model in the fleet pinned on the same questions. Every number is measured on the Xantly platform; nothing is a published score or an estimate.

Accuracy and cost against the top models

Xantly auto answered 86.3% for $1.72. Claude Opus 4.6 reached the same accuracy for 4.6 times the cost. Gemini 3.5 Flash scored 1.1 points higher for 5.3 times the cost, and Gemini 3.1 Pro scored 0.4 points lower for 6.6 times the cost.

The ten most accurate pinned models and the router, on the same 1,000 questions. Cost is the provider cost for all 1,000 answers; the multiple is that cost divided by the router's.

Setup

The suite

1,000 questions drawn from eight public datasets, fixed once and reused for every model. Difficulty labels come from the suite, not from the router.

How each system was run

• The router. Each question was sent once to xantly/auto through the public API, exactly as a customer sends it: no model pinned, no routing hints, streaming on, one call per question. The gateway chose the model per request.
• The pinned models. Each comparison model answered the same 1,000 questions through the same API with the model pinned. The ten most accurate are shown above.
• Grading. One automatic grader scored every answer for every system: exact match for numeric and multiple-choice answers, test execution for code, and the dataset's own answer key throughout.
• Cost. Provider list price at the tokens each model actually used, taken from the gateway's billing records. The router's cost is the sum of what its chosen models charged, including any call the gateway made on the caller's behalf.

Results by part of the suite

Reliability

• 1,000 of 1,000 requests returned an answer. No request timed out, failed over on an error, or returned empty.
• Twice, a thinking model spent its whole output budget reasoning and produced no answer. The gateway detected it, did not serve the empty reply, and asked another model; the caller received a normal answer. Both requests are counted at their full cost.

Reproducibility

• The suite, the grader and the cost accounting are fixed; the same questions and the same grader scored every system on this page.
• The run's records (every request, the served model, the graded answer and the billed cost) are retained, so the page can be regenerated from them.
• A new run replaces this page when the platform changes. The revision history records each one.

Revision history

Every model the router used

All 20 models the router chose across the 1,000 requests. Accuracy per row is on the questions that model received and is not comparable between rows: the router sends the hard questions to the strong models and the easy ones to the cheap models, so the strong models carry the hard misses by design.