Use Xantly alongside llamafile

Mozilla's single-binary llamafile runtime pairs with Xantly for a hybrid local + hosted setup. Offline private inference + cloud reasoning for the hard stuff.

llamafile is Mozilla's single-binary LLM runtime, download one file, run any GGUF model with a built-in OpenAI-compatible server. Good fit for offline / privacy-first local inference. This guide shows how to run llamafile for local models and route hosted requests through Xantly, same hybrid pattern as Ollama bridge but llamafile-specific.

Why pair llamafile with Xantly?

WorkloadRun where
Private / air-gapped workllamafile (local): no network, zero cloud.
Small classification / quick completionsllamafile: qwen / llama / mistral GGUFs.
Big reasoning / agent workXantly: BaRP picks T1 (Claude, GPT).
Heavy tool-callingXantly: local GGUFs often fail tool schemas.
Embedding a huge corpusXantly: cheaper + higher quality at scale.

Prerequisites

Run llamafile

./qwen2.5-coder-7b-instruct-q6_k.llamafile --server --port 8080

llamafile exposes an OpenAI-compatible server at http://localhost:8080/v1.

Pattern 1, Continue.dev hybrid config

~/.continue/config.yaml:

models:
  - name: Xantly Quality
    provider: openai
    model: xantly/auto-quality
    apiBase: https://api.xantly.com/v1
    apiKey: xantly_sk_...
    roles: [chat, edit, apply]

  - name: Local llamafile (autocomplete)
    provider: openai
    model: qwen2.5-coder-7b
    apiBase: http://localhost:8080/v1
    apiKey: dummy
    roles: [autocomplete]

Autocomplete runs on the local llamafile (zero cost, offline). Chat/edit/apply hit Xantly for smart routing.

Pattern 2, Aider role split

aider \
  --openai-api-base https://api.xantly.com/v1 \
  --openai-api-key xantly_sk_... \
  --model xantly/auto-quality \
  --editor-model openai/qwen2.5-coder-7b \
  --weak-model openai/qwen2.5-coder-7b

Configure the editor + weak model to hit llamafile's http://localhost:8080/v1 by setting an OPENAI_API_BASE override via an .aider.conf.yml per role, or just run llamafile through a local LiteLLM proxy for clean routing.

Pattern 3, Manual switching per task

Use llamafile for the task. Hit Xantly when you need quality. Two quick aliases:

alias ai-local='export OPENAI_API_BASE=http://localhost:8080/v1 && export OPENAI_API_KEY=dummy'
alias ai-xantly='export OPENAI_API_BASE=https://api.xantly.com/v1 && export OPENAI_API_KEY=xantly_sk_...'

ai-local for private chat, ai-xantly for production-grade reasoning. Any OpenAI-compatible tool picks up the env vars.

Cost example

Heavy developer workflow:

Plus: offline work becomes possible. Plane wifi or power outage? llamafile keeps working.

What you get

Gotchas

llamafile's built-in server doesn't do tool-calling reliably. Most GGUFs need careful prompting for tools. Push agent/tool work to Xantly.

Context window limits. A 7B GGUF is typically 8k-32k tokens. If you paste a 100k-token file, llamafile truncates. Xantly auto-escalates to a bigger-context model.

First-call warmup. llamafile mmaps the model; first call after idle is slower. Keep it running.

Port conflicts. If port 8080 is taken, use --port 8090.

Cold-starting Xantly for escalation. If a local call fails and you need Xantly, there's a brief context-switch cost. Negligible in practice.

Next steps