Resource

How to choose a model.
Constraints first.

Model selection is usually approached as a capability comparison and is usually settled by something else. Working the constraints first eliminates most candidates in an afternoon and leaves a shortlist worth testing properly.

THE PROCEDURE

STEP 01

Where may the data live?

The hardest constraint and the one that eliminates the most candidates. If nothing may leave your infrastructure, every closed model is out and the decision is among open weights. If a region is required, that determines which hosted paths are available. Settle this before anything else, because no capability argument overturns it.
STEP 02

What does the licence have to permit?

Commercial use is table stakes. Beyond that: do you need to fine-tune, distil, redistribute, or embed the model in a product you sell? Some open licences restrict these. Some procurement policies require a recognised open-source licence rather than a bespoke one. Cheap to check and expensive to discover late.
STEP 03

What latency can the interface absorb?

A user waiting inside your product is a different constraint from a nightly batch. This eliminates the largest and slowest models from interactive paths, and it frequently means a system needs two models rather than one.
STEP 04

What does it cost at your volume?

Project realistically, including retries, evaluation traffic and conversational growth. At this point the shortlist is usually three or four candidates, which is a tractable number to test properly.
STEP 05

Now benchmark — on your task

Build an evaluation set from real examples of your work, with correct answers. Run the shortlist. Compare on quality, cost and latency together. This is where public leaderboards are finally useful: as a source of candidates, not as the answer.
STEP 06

Design for substitution

Whatever wins, put it behind an interface, keep the prompts portable and keep the evaluation set. You will run this procedure again within a year, and the second time should cost days rather than a rewrite.

WHY LEADERBOARDS MISLEAD

They measure general capability on tasks designed by other people. Your task is narrower and its distribution is different, and relative performance reorders on domain-specific work with a regularity that ought to be better known. A model leading on general reasoning can be beaten on your extraction task by something a twentieth of the price.

They also do not measure the things that decide deployments: cost at your volume, latency at your concurrency, whether the licence permits what you need, and whether the data may go there at all. A benchmark score is one input among six and it is the last one to consider, not the first.

The one thing leaderboards are genuinely good for is generating a shortlist — knowing which models are broadly in contention this quarter. Used for that and no more, they save time.

QUESTIONS

QUESTIONS — 3

A hundred to a few hundred real examples with correct answers is usually enough to separate candidates decisively. Realism matters more than size: examples drawn from actual work, including the awkward ones, beat a larger set of clean invented cases every time.

When something changes — a new release, a price change, a deprecation notice, or a shift in your own workload. Roughly twice a year in practice. If the system was built for substitution it is a short exercise; if it was not, that is the real finding.

You can, and you will usually overpay for it. Most systems benefit from at least two: a small one for routine work and a larger one for the hard minority. That is a routing decision rather than a selection one, and it is where a great deal of cost saving lives.

Keep reading

Evaluating AI Systems

30-minute technical call · no deck