# Use Xantly alongside llamafile

[llamafile](https://github.com/Mozilla-Ocho/llamafile) is Mozilla's single-binary LLM runtime, download one file, run any GGUF model with a built-in OpenAI-compatible server. Good fit for offline / privacy-first local inference. This guide shows how to run llamafile for local models **and** route hosted requests through Xantly, same hybrid pattern as [Ollama bridge](/docs/use-with-ollama-bridge) but llamafile-specific.

## Why pair llamafile with Xantly?

| Workload | Run where |
|---|---|
| Private / air-gapped work | **llamafile (local)**: no network, zero cloud. |
| Small classification / quick completions | **llamafile**: qwen / llama / mistral GGUFs. |
| Big reasoning / agent work | **Xantly**: BaRP picks T1 (Claude, GPT). |
| Heavy tool-calling | **Xantly**: local GGUFs often fail tool schemas. |
| Embedding a huge corpus | **Xantly**: cheaper + higher quality at scale. |

## Prerequisites

- [llamafile](https://github.com/Mozilla-Ocho/llamafile) binary + your chosen GGUF model
- A Xantly API key, [create one](/dashboard/api-keys)
- A tool that supports configuring two providers

## Run llamafile

```bash
./qwen2.5-coder-7b-instruct-q6_k.llamafile --server --port 8080
```

llamafile exposes an OpenAI-compatible server at `http://localhost:8080/v1`.

## Pattern 1, Continue.dev hybrid config

`~/.continue/config.yaml`:

```yaml
models:
  - name: Xantly Quality
    provider: openai
    model: xantly/auto-quality
    apiBase: https://api.xantly.com/v1
    apiKey: xantly_sk_...
    roles: [chat, edit, apply]

  - name: Local llamafile (autocomplete)
    provider: openai
    model: qwen2.5-coder-7b
    apiBase: http://localhost:8080/v1
    apiKey: dummy
    roles: [autocomplete]
```

Autocomplete runs on the local llamafile (zero cost, offline). Chat/edit/apply hit Xantly for smart routing.

## Pattern 2, Aider role split

```bash
aider \
  --openai-api-base https://api.xantly.com/v1 \
  --openai-api-key xantly_sk_... \
  --model xantly/auto-quality \
  --editor-model openai/qwen2.5-coder-7b \
  --weak-model openai/qwen2.5-coder-7b
```

Configure the editor + weak model to hit llamafile's `http://localhost:8080/v1` by setting an `OPENAI_API_BASE` override via an `.aider.conf.yml` per role, or just run llamafile through a local LiteLLM proxy for clean routing.

## Pattern 3, Manual switching per task

Use llamafile for the task. Hit Xantly when you need quality. Two quick aliases:

```bash
alias ai-local='export OPENAI_API_BASE=http://localhost:8080/v1 && export OPENAI_API_KEY=dummy'
alias ai-xantly='export OPENAI_API_BASE=https://api.xantly.com/v1 && export OPENAI_API_KEY=xantly_sk_...'
```

`ai-local` for private chat, `ai-xantly` for production-grade reasoning. Any OpenAI-compatible tool picks up the env vars.

## Cost example

Heavy developer workflow:

- llamafile running 24/7 on an M3 Mac → effectively $0 marginal.
- Xantly hit only for big reasoning jobs (~15/day) → ~$1.50/day.
- Net: **~$45/month** down from $150+ all-cloud.

Plus: offline work becomes possible. Plane wifi or power outage? llamafile keeps working.

## What you get

- **True offline** when needed, llamafile takes no network.
- **Pay only for reasoning**: Xantly handles the expensive calls, llamafile covers the bulk.
- **Same OpenAI shape** on both sides, tools switch seamlessly.
- **Waterfall from local to hosted.** If a local model can't handle a request (context window, tool complexity), escalate to Xantly.

## Gotchas

**llamafile's built-in server doesn't do tool-calling reliably.** Most GGUFs need careful prompting for tools. Push agent/tool work to Xantly.

**Context window limits.** A 7B GGUF is typically 8k-32k tokens. If you paste a 100k-token file, llamafile truncates. Xantly auto-escalates to a bigger-context model.

**First-call warmup.** llamafile mmaps the model; first call after idle is slower. Keep it running.

**Port conflicts.** If port 8080 is taken, use `--port 8090`.

**Cold-starting Xantly for escalation.** If a local call fails and you need Xantly, there's a brief context-switch cost. Negligible in practice.

## Next steps

- [Ollama bridge](/docs/use-with-ollama-bridge), same hybrid pattern with Ollama.
- [Continue.dev](/docs/use-with-continue-dev), the best per-role hybrid config.
- [Aider](/docs/use-with-aider), main/editor/weak role split.
- [Cost-Optimized Routing](/docs/cost-optimized-routing), when local isn't viable.