Technology Llama · Mistral · Qwen vLLM · Ollama On-prem or your VPC

Open-weight LLMs.
Own the model.

Self-hosting removes an entire class of data-residency and vendor problems, and introduces a different class of cost and capability ones. The trade is real in both directions.

SPECIFICATION

Families in scope: Llama, Mistral, Qwen, DeepSeek, Gemma, Phi and the distilled variants of larger reasoning models
Licence range: Apache-2.0 at the permissive end, bespoke community licences with attribution and user-count conditions in the middle, research-only at the restrictive end. Never uniform across a family — check the specific model card.
Hardware floor: A single accelerator for the small and mid sizes; a multi-GPU node for the largest. Accelerator memory, not compute, is the binding constraint.
Cost shape: Fixed infrastructure rather than per token, so marginal per-request cost falls toward power and operations cost once the hardware is paid for
What you give up: The absolute frontier on the hardest agentic and reasoning work, and operational simplicity
What you gain: No data egress, no per-token bill, no vendor-driven deprecation, and — where the licence permits — the right to fine-tune and keep the artefact

Provider-published specifications as of 1 September 2026, not figures we measured. Model families move quickly — check the provider's own documentation before committing to a number.

WHEN SELF-HOSTING IS RIGHT

Data genuinely cannot leave

Air-gapped environments, or a regulator or contract that forbids third-party processing outright. Here it is not a preference — no hosted API is available at any capability level, so the question is which open model, not whether.

Very high volume on a narrow task

At sufficient scale, amortised GPU cost beats per-token pricing — particularly for classification or extraction where a small fine-tuned model matches a much larger general one. The crossover is workload-specific and worth calculating rather than assuming.

Latency floors an API cannot meet

Local inference removes network round-trips. For interactive systems with tight budgets, or edge deployments with poor connectivity, this can be decisive.

Version stability you control

A model you host does not get deprecated on someone else's schedule. For systems that must behave identically for years — regulated decisioning especially — this is a genuine architectural advantage.

WHAT IT ACTUALLY COSTS

The GPU bill is the visible cost and rarely the largest one. Serving infrastructure needs building and operating: batching, queueing, autoscaling, health checks, model loading, version management. Somebody has to own it at 3am.

Utilisation is the number that decides the economics. A reserved GPU costs the same whether it is saturated or idle, so bursty workloads on dedicated hardware are frequently more expensive per request than an API — sometimes dramatically. Self-hosting wins on steady, predictable, high volume; it loses on spiky traffic.

And the capability gap is real. Open-weight models have closed much of the distance on many tasks and remain behind frontier models on the hardest reasoning and the most complex agentic tool use. For extraction, classification, summarisation and straightforward retrieval-augmented answering, that gap is often irrelevant. For multi-step reasoning under ambiguity, it is not.

Our usual recommendation is a hybrid: open weights for the high-volume, well-defined majority; a hosted frontier model for the small share of genuinely hard requests — assuming your data-residency rules permit anything hosted at all.

QUESTIONS

QUESTIONS — 4

It depends on model size and concurrency. A quantised mid-size model serves a small team comfortably on a single modern data-centre GPU. Larger models at production concurrency need multiple GPUs and real serving infrastructure. We size this against your actual expected throughput during discovery, because the difference between “a few people internally” and “customer-facing at scale” is one or two orders of magnitude.

Benchmark on your task rather than trusting general rankings. Llama, Mistral and Qwen families all have strong entries and they trade places regularly. Licence terms differ meaningfully and are worth reading — some carry usage restrictions that matter commercially, which is a constraint most technical evaluations skip.

Yes, and this is where self-hosting often pays for itself. Parameter-efficient fine-tuning on a well-defined task can bring a small model to or past a much larger general one on that task specifically. Note it teaches behaviour and format, not facts — facts still belong in retrieval.

It changes the threat model rather than strictly improving it. You remove third-party data processing, and you take on responsibility for patching, access control and isolation of a new piece of infrastructure. A well-configured enterprise API path is often more secure in practice than a hastily deployed internal GPU cluster. Choose it for the residency and control reasons, and then do the security work properly.

Keep reading

Architecture

Secure Data Pipelines

→

Trust

GDPR & Data Residency

→

Reference

Document Intelligence

→

30-minute technical call · no deck