Technology

Inference optimisation.
What serving costs.

On a self-hosted deployment, inference efficiency is the difference between one GPU and four. These are the levers, in roughly the order of return we see when we apply them.

THE LEVERS, IN ORDER

Continuous batching

The single largest throughput win, and mostly free — a modern serving stack processes many requests together and admits new ones as slots free rather than waiting for a batch to complete. If you are running a naive one-request-at-a-time loop, this alone is typically a multiple.

Quantisation

Running weights at eight or four bits rather than sixteen cuts memory proportionally, which often means fitting on smaller or fewer accelerators. Quality cost is task-dependent and frequently unmeasurable — but it must be measured, not assumed.

KV cache management

The per-request attention cache dominates memory at high concurrency. Paged attention, prefix sharing across requests with a common system prompt, and eviction policy decide how many concurrent users a node supports far more than raw compute does.

Speculative decoding

A small model drafts tokens that the large model verifies in parallel, cutting latency without changing the output distribution. Real gains on latency-sensitive paths; extra complexity, and the draft model has to be well matched to be worth it.

MEASURE THE RIGHT THING

Throughput and latency pull against each other and optimising one blindly damages the other. Larger batches raise tokens per second and raise the time an individual user waits. Which matters depends on whether a person is watching, and that answer differs per endpoint in the same system.

For interactive work the metric users actually feel is time to first token, not total generation time — streaming means they start reading immediately, and a fast first token with a moderate rate feels better than a slow first token with a fast rate. For batch work, only total throughput matters and latency is irrelevant.

Measure at realistic concurrency. Single-request benchmarks are close to useless for capacity planning, because the whole point of continuous batching is that per-request cost falls as concurrency rises. A node that looks marginal on a single-request test frequently serves the whole department.

THE CHEAPEST OPTIMISATION

None of the above. It is sending fewer tokens. Context that is re-sent unchanged on every turn, retrieved passages nobody needed, verbose system prompts, and reasoning enabled on requests that did not require it are all pure cost — and they are typically a larger share of the bill than any serving inefficiency.

Before tuning a serving stack, look at what is actually in the requests. In our experience context hygiene alone has often produced a material reduction before any serving change, and it requires no infrastructure change at all.

QUESTIONS

QUESTIONS — 3

Task-dependent, sometimes nothing measurable at eight bits, more noticeable at four and more so on reasoning-heavy work. It is a measurement rather than a rule: run your evaluation set at each precision and decide with the numbers in front of you.

For production GPU serving, vLLM is the usual answer — mature continuous batching, paged attention and broad model support. Ollama is designed for local and development use rather than high-concurrency serving. The choice is less consequential than configuring whichever you pick properly.

The serving levers are the provider's problem, but prompt caching and context hygiene are yours and behave the same way. On a hosted API, designing context so the stable part is a genuine prefix is the single largest cost lever available to you.

Keep reading

Tokens & Context Windows

30-minute technical call · no deck