Infrastructure

GPU sizing and cost.

In our experience GPU quotes are more often too large than too small. Sizing is not guesswork — it comes out of three numbers you already have, and getting it right is usually the difference between a project that clears its business case and one that does not.

THE THREE NUMBERS THAT DECIDE IT

Model memory footprint

Parameter count multiplied by bytes per parameter, plus overhead. At half precision that is roughly two bytes per parameter; quantised to eight or four bits it drops proportionally, at a quality cost you should measure rather than assume. This sets the floor: a model that does not fit does not run.

Context and concurrency

Every concurrent request holds a KV cache whose size scales with context length. This is the number teams forget, and it is frequently larger than the model itself. Twenty users on long documents need materially more memory than twenty users on short questions.

Throughput target

Tokens per second at your peak, derived from requests per hour, output length and how long a user will wait. Batching means throughput scales far better than intuition suggests — one node serves more people than most quotes assume.

WHERE QUOTES GO WRONG

Sizing for peak concurrency that never happens. Internal tools have usage patterns shaped like a working day, not like a consumer product. Sizing for simultaneous use by every licensed user produces a machine that is idle ninety per cent of the time. Size for realistic peak, and queue gracefully beyond it.

Ignoring quantisation. Running at eight-bit or four-bit precision cuts the memory requirement substantially, and on many tasks the quality difference is often not measurable on a real evaluation set. It is not free and it is not universal — but testing it before buying twice the hardware is an afternoon's work with a large payoff.

Choosing a model larger than the task needs. The most expensive sizing error happens before any hardware is discussed. If a small model handles eighty per cent of your requests identically — which is testable in a day — then the hardware conversation is about a much smaller machine, and the remaining twenty per cent has a routing answer rather than a procurement one.

Forgetting that batching exists. A serving stack like vLLM batches continuously, so aggregate throughput on one node is far higher than a single-request benchmark implies. Sizing from single-request latency alone can overshoot by a large factor.

BUY, RENT OR CALL AN API

	Public API	Rented cloud GPU	Owned hardware
Upfront cost	None	None	Significant
Cost shape	Per token	Per hour, running or not	Depreciation + power
Cost at high sustained volume	Worst	Middle	Best
Cost at low or spiky volume	Best	Middle	Worst
Data stays on your hardware	—	—	✓
You operate it	No	Partly	Yes
Sensible for	Prototypes, low volume	Bursty or uncertain load	Steady production volume

The structural cost shapes of the three options. Which one wins depends on your volume, and the crossover is arithmetic rather than opinion.

The crossover calculation is not complicated and we do it in the open: take your projected monthly token volume, price it at the API rate you would actually pay, and compare it against the monthly cost of ownership — hardware amortised over three years, plus power, plus a realistic fraction of an engineer. The number that decides it is usually the last one, and it is the one most comparisons leave out.

Two honest observations from doing this repeatedly. First, the arithmetic favours ownership sooner than most people expect once volume is genuinely sustained, because API pricing is linear and hardware is not. Second, it favours the API far longer than vendors of hardware will tell you when volume is spiky, because an idle GPU costs the same as a busy one.

QUESTIONS

QUESTIONS — 4

Very often, yes. A single well-specified accelerator runs a capable mid-size open model for a meaningful number of internal users, and it is the right way to prove the deployment before committing to a cluster. Scaling out later is a known quantity; scaling down after over-buying is a write-off.

Not usually. Memory capacity matters more than raw compute for most inference workloads, and previous-generation cards with generous memory are frequently better value than current-generation cards with less. That is a specification question we work through with your actual model, not a brand preference.

Sometimes nothing measurable, sometimes a lot — it depends on the model and the task, which is why it is a measurement rather than a rule. We run your evaluation set at full and reduced precision and show you both numbers before anyone signs a purchase order.

Real and worth including, and usually a smaller line than people fear relative to the hardware amortisation. It matters most for the office-based on-premise case, where it can also be a physical constraint on what the room can actually host.

Keep reading

Inference Optimisation

→

Models

Small Language Models

→

30-minute technical call · no deck