Why local models should be first-class citizens in AI agents

I asked a local model a question and waited nine minutes for the first word of the answer. The model was a 27B Qwen, quantized to 4 bits, served by mlx_lm.server on an M3 Pro. The server hadn't hung. It was working through a prompt of 50,111 tokens at about 93 tokens per second. When I finally counted, the conversation itself — my question and the history before it — came to less than a hundred of those tokens. Everything else was my agent harness.

That is this article in small. A local model in an agent is not a cloud model with a different URL. It punishes everything a harness sends without thinking, and it forgives nothing a large model would. For most agent tools, that is a reason to leave local models in a settings field for custom endpoints. I think it is the best reason to do the opposite.

Here is what I learned wiring Ollama, LM Studio and MLX into an agent as equals of OpenAI, Anthropic and Google: what is different, what went wrong, how a small model copes with tool calling, and what still doesn't work.

Why local models matter for agents in particular

The usual reasons for running a model locally — privacy, cost, control — all apply. With an agent, each of them gets sharper.

• An agent reads your files. That is the point of it: you hand it a folder, and it opens what it needs. With a cloud model, every file it reads becomes part of a request to someone else's server. With a local model, the folder stays on your disk. Tools that fetch web pages or call outside services are a separate question; the model itself doesn't need the network.
• An agent spends tokens in rounds. One question can take a dozen tool rounds, and each round sends everything again: the system prompt, the tool definitions, the history, every tool result so far. Per-token pricing charges most for exactly the open-ended, try-it-and-see work an agent is good at. Locally, the price is time, and you can see it.
• A small model is an honest test. A large cloud model forgives a vague tool description, an ambiguous schema and a system prompt three times longer than it needs to be. A small local model doesn't. It shows you where your harness quietly depends on a big model's goodwill.
• The model stays the model. No account, no deprecation notice, no silent update between Tuesday and Wednesday. A prompt you tune against a local model is tuned against the same model next month.

What first-class means

Supporting local models can mean as little as a field for a custom base URL: a request goes out, and the harness behind it neither knows nor cares that the model is running on your laptop. That is support. It isn't first-class. For me, first-class meant four things:

• One list. Local and cloud models sit in the same picker, and you switch per conversation. The model belongs to the chat: an old conversation comes back with the model it was held with.
• The same agent. The same tools, the same approval rules, the same prompt, whichever model answers. Where something does differ — image attachments, for instance — the app says so instead of quietly dropping them.
• No key where none is needed. A local server usually has no API key. An empty key field then means no Authorization header at all, not an empty one.
• A harness that knows the model is local, and changes its behaviour where that matters.

The last point is where it gets interesting.

What is different from a cloud API

Prefill is what you wait for

With a cloud API, reading the prompt is fast and invisible; you notice the prompt's size on the invoice. On a laptop it is the other way round. Generating the answer is bearable; reading the prompt is what you wait for. At 93 tokens per second, every thousand tokens of system prompt and tool definitions cost about eleven seconds whenever the server has to read them.

Every server speaks a slightly different dialect

“OpenAI-compatible” describes a family resemblance, not a standard. This is what I had to handle:

• Ollama gets its native API, not its /v1 layer: /api/tags for the model list, /api/chat with a stream of JSON lines. Tool calls arrive whole rather than in fragments, with the arguments as a JSON object instead of a string, and without call IDs. The harness turns the arguments into a string and makes up an ID for each call, because the rest of the pipeline expects both. Going through /v1 instead would have cost the native model list.
• LM Studio, llama.cpp, vLLM and mlx_lm.server speak Chat Completions. Tool calls come in streamed pieces, and even the name can arrive in several chunks, so a call is announced only once its first argument fragment is there. Some servers leave out a call's ID or its index; the harness assigns them. Reasoning models on llama.cpp or vLLM stream their thinking in a field of its own (reasoning_content or reasoning), which is kept apart from the answer.
• Some servers choke on tool schemas. For those there is a switch per entry, “Send tools along”, that turns the model into a plain chat partner. I considered detecting tool support automatically and dropped the idea: the same provider also talks to paid gateways, where every probe costs money.
• A model list is a convenience. If GET /models fails or comes back empty, that is not an error. You type the model name and carry on.

MLX-LM started out as a provider of its own. When I built the generic OpenAI-compatible one, the MLX provider turned out to be functionally identical — there is nothing MLX-specific about the endpoint mlx_lm.server offers. It is now a template of the generic provider. One well-tested path for every OpenAI-shaped server is worth more than a special case per server.

Local is a property of the address

Once one provider can point at LM Studio on localhost and at a gateway across the internet, the provider's name can no longer tell you whether a model is local. The host of the address can: localhost, the whole 127.0.0.0/8 range (llama.cpp sometimes binds to 127.0.0.2), ::1, 0.0.0.0 and *.local. Ollama counts as local whatever its host.

A private network — 10.x, 192.168.x — deliberately doesn't count. There is usually a different, often much faster machine on the other end, and treating it as local would cost history for nothing.

What “local” changes is small and specific. Listing the models may take 30 seconds instead of 15 before it times out. The history budget is 40,000 characters instead of 200,000. And the estimate of what a prompt costs uses a divisor measured against a local tokenizer. Tools, approvals and the prompt itself stay the same.

Nine minutes, and the wrong suspect

Back to the nine minutes. The first suspect was obvious. The history budget stood at 200,000 characters, and at the usual four characters per token that is almost exactly 50,000 tokens. The prompt looked as if it was sitting right at the cap. So I gave local models the tighter budget and trimmed the built-in part of the system prompt from 8,108 to 5,956 characters. The prompt went from 50,111 tokens to 49,535.

The history had never been the problem; the coincidence with the budget had sent me the wrong way. So I did what I should have done first: I captured the real request with a small server that counts, and counted it with the model's own tokenizer.

The biggest single block was one MCP server. It brought 73 tools for an issue tracker and a wiki, with 86,210 characters of schema, sent in the tools field of every request and counted by no budget. I needed a handful of them. Deselecting the rest should have been easy, and here the second bug surfaced: the selection was silently lost on saving whenever the server wasn't connected — which, for a server running in Docker, is the normal case when you open the settings. With that fixed and 63 of the 73 tools deselected, the prompt was down to 27,487 tokens. The next count split what was left:

Almost the entire system prompt was skills, and eight of them were ones I had switched on for other work, none relevant to the question at hand. Each went into every request in full. Switching them off brought the prompt to 12,831 tokens: from about nine minutes to just over two.

Switching things off by hand isn't a fix, though; it is a chore you have to remember every time you change models. The structural fix was to put only each skill's name and description into the prompt and let the model load the rest when it needs it, through a load_skill tool. With the eighteen skills I had enabled, the base load fell from 36,619 tokens to 2,604 — 93 percent less, the new tool's own schema included.

And because it had taken a counting server to find any of this, the prompt's composition is now visible. Below the input field, the size of the last request expands into what it consisted of: every skill, the built-in tools, each MCP server, the rest of the system prompt, the history. The total is the server's own figure; the breakdown is an estimate.

Even the estimate had a lesson in it. Measured against Qwen's tokenizer, JSON schemas packed into about 2.3 characters per token. Against OpenAI's o200k it was about 4.1. A single divisor tuned on the local model overstated OpenAI's tool definitions by around 80 percent, so each tokenizer family now has a profile of its own.

Here is the part I keep coming back to: every one of these fixes made the prompt smaller for the cloud models too. With a cloud model, 50,000 tokens per request is a line on an invoice nobody reads closely. On a laptop it is nine minutes, and that is what finally made me look. Most of what the local model needed wasn't special treatment. It was a harness that stops wasting tokens — and the local model was the only one that made the waste impossible to ignore.

How a small model copes with tool calling

I wish I had a ranking for you. I don't; a paired benchmark for tool-call accuracy is still on the list. What I have is one small model, measured carefully, failing in an instructive way.

The model was llama3.1:8b on Ollama, at temperature 0, with eighteen skills enabled. Five questions, each of which should make the model call load_skill with the right skill's name — given as free text or as an enum of the enabled skills:

Without the enum, the model invented skill names and loaded nothing. With it, the model picked only real names, and its remaining mistakes were understandable ones — a Markdown-to-PDF skill where a Markdown-to-Word one was meant. The enum stayed in: for a small model, a closed list beats any description.

The second finding is the one that stuck with me. With only the file-reading tool in the schema, “Read package.json” produced exactly the right call. With all 13 tools in the schema, the same request got an answer in prose and no tool call at all. The model could do the task; it just couldn't find it among thirteen.

One model is not a study. But it changed how I think about tool sets. For a large model, one more tool costs a few hundred tokens. For a small one, it can cost the ability to call any tool at all. The arguments show a milder version of the same thing: weak models readily pass a parameter that belongs to a different tool, a path to the search tool, say. My harness now counts those instead of silently skipping them, because what they cost isn't the wrong parameter but the extra round it triggers.

The 27B Qwen from the beginning I measured for prefill, not for tool-call accuracy, so I won't pretend to numbers I don't have.

What doesn't work yet

• The budget is still reactive. The system prompt and the tool definitions don't count towards the history budget, and tool results are appended in full and only cut once the total overflows. A budget that plans the whole prompt up front is the real fix for prefill, and it isn't built yet.
• Many tools, small models. Loading MCP tools on demand, the way skills are loaded, is the obvious next step. I did the arithmetic: it only pays off above about fourteen active tools, because the loading step produces a tool result of its own that rides along in every later round. For the built-in tools, the same scheme came out negative.
• No context size for Ollama. The harness doesn't set num_ctx, so whatever your Ollama installation defaults to applies.
• No tool-support detection on Ollama, and no switch to leave the tools out there; only the OpenAI-compatible entries have one.
• Tool calls written as text aren't parsed. If a server leaves a model's tool call in the text of the answer instead of turning it into a structured call, the agent sees prose.
• Ollama's thinking is dropped. Its separate thinking field isn't read yet.
• Images on Ollama. A local vision model would keep images on the machine as well, but attachments to Ollama are still blocked, with a message that says so.

Where this ended up

I ended up building this into an open-source project called Snotra. It is a desktop agent that works in a folder you keep in view, asks before it acts, and treats cloud and local models the same: OpenAI, Anthropic and Google next to Ollama and anything OpenAI-compatible, with templates for LM Studio, MLX-LM, llama.cpp and vLLM. No account, no telemetry, Apache 2.0. The longer story of why it exists is in Why I built an open-source AI agent desktop.

The open ends above are issues there, if you want to follow one or push it along: the prompt budget, MCP tools on demand, the tool-call benchmark and images for Ollama.

And since I couldn't give you a ranking, I'd rather ask for one. If you run local models in an agent: which server, which model, and where did it fall over? The Discussions are open for exactly that.