Model families

Small language models.
The case for smaller.

Most production AI work is not hard. It is classification, extraction, routing and rewriting, at volume, against a fixed schema. Sending that to a frontier model is the most common expensive default in the field, and it is often a short piece of work to fix.

SPECIFICATION

Size range: Roughly 1B to 14B parameters — the class that fits on one accelerator, often with room to spare
Families: Llama, Qwen, Mistral, Gemma, Phi, and distillations of larger reasoning models
Hardware: A single GPU serves a substantial number of concurrent users; the lower sizes run on CPU at usable speeds for batch work
Cost against a frontier API: One to two orders of magnitude lower per token, and fixed rather than usage-linked once self-hosted
Suited to: Fixed-schema extraction, classification, routing, reranking, and any path where latency is felt by a user
Not suited to: Multi-step agentic work, open-ended reasoning over long context, and holding many simultaneous constraints

Provider-published specifications as of 1 September 2026, not figures we measured. Model families move quickly — check the provider's own documentation before committing to a number.

WHAT COUNTS AS SMALL

Roughly, anything from one to fourteen billion parameters — the range that fits on a single accelerator, often with room to spare, and in the lower half runs acceptably on a CPU. Every major family ships in this class: Llama, Qwen, Mistral, Gemma, Phi and the distilled variants of larger reasoning models.

The interesting property is not that they are cheap, though they are, by one to two orders of magnitude per token against a frontier API. It is that they are local. A model that fits on one GPU can run inside your building, at the edge, on a laptop, or in a container next to the service that calls it — with no network hop, no per-token bill and no third party in the data path. That changes what is architecturally possible, not just what it costs.

WHERE THEY WIN OUTRIGHT

Fixed-schema extraction

Pulling defined fields out of a document into a known structure. The task is bounded, the output is validated downstream, and a small model fine-tuned on a few thousand of your own examples can match a frontier model on it.

Classification and routing

Deciding which queue, which team, which template, which model. High volume, narrow decision space, and a place where latency is felt directly by a user. This is the single highest-value small-model deployment in most organisations.

Reranking and filtering

Sitting between retrieval and generation to reorder candidates. Called many times per request, which makes unit cost dominate and makes a frontier model structurally unaffordable in the slot.

Anywhere latency is the product

Autocomplete, inline suggestion, real-time moderation. A small local model answers in the time a network round trip to an API would still be in flight, and users experience that difference as quality.

WHERE THEY STOP BEING ENOUGH

Multi-step agentic work. Holding a plan across many tool calls, recovering sensibly when one fails, and not losing the thread halfway through is where the capability gap is still wide and obvious. Small models can execute a step; they are much less reliable at owning a sequence.

Open-ended reasoning over long context. Synthesis across a large document set, or any task where the shape of the answer is not known in advance. The gap narrows every release and it has not closed.

Instruction adherence under many simultaneous constraints. Hold this schema, in this tone, within this policy, across a long interaction. Frontier models are noticeably better at not dropping one of the four.

The practical architecture that follows is a ladder, not a choice: a small model handles the bulk, a gate detects the requests it should not have taken, and those escalate. Building that gate well is the actual engineering, and it is covered on hybrid model routing.

QUESTIONS

QUESTIONS — 4

You build an evaluation set from real examples of that task — a few hundred is usually enough to be decisive — and run both. It is a day or two of work and it settles the question with evidence instead of instinct. In our experience, more of the workload typically moves down than teams expect.

Often not. A well-designed prompt plus a few examples closes most of the gap on a bounded task. Fine-tune when you have a stable task, a few thousand labelled examples, and a measured gap that prompting has not closed — in that order.

A single accelerator will serve a small model to a substantial number of concurrent users, and the smallest run on CPU for batch work. The cost conversation moves from per-token to per-hour, which is exactly what makes high-volume workloads viable — the bill stops scaling with usage.

Cost is the visible benefit and rarely the important one. The important one is that a model small enough to run on your own hardware removes the third party from the data path entirely, which removes the third-party-processor part of the governance conversation.

Keep reading

Fine-Tuning & Adaptation

30-minute technical call · no deck