Model selection is usually approached as a capability comparison and is usually settled by something else. Working the constraints first eliminates most candidates in an afternoon and leaves a shortlist worth testing properly.
All Resources →
The hardest constraint and the one that eliminates the most candidates. If nothing may leave your infrastructure, every closed model is out and the decision is among open weights. If a region is required, that determines which hosted paths are available. Settle this before anything else, because no capability argument overturns it.
Commercial use is table stakes. Beyond that: do you need to fine-tune, distil, redistribute, or embed the model in a product you sell? Some open licences restrict these. Some procurement policies require a recognised open-source licence rather than a bespoke one. Cheap to check and expensive to discover late.
A user waiting inside your product is a different constraint from a nightly batch. This eliminates the largest and slowest models from interactive paths, and it frequently means a system needs two models rather than one.
Project realistically, including retries, evaluation traffic and conversational growth. At this point the shortlist is usually three or four candidates, which is a tractable number to test properly.
Build an evaluation set from real examples of your work, with correct answers. Run the shortlist. Compare on quality, cost and latency together. This is where public leaderboards are finally useful: as a source of candidates, not as the answer.
Whatever wins, put it behind an interface, keep the prompts portable and keep the evaluation set. You will run this procedure again within a year, and the second time should cost days rather than a rewrite.
They measure general capability on tasks designed by other people. Your task is narrower and its distribution is different, and relative performance reorders on domain-specific work with a regularity that ought to be better known. A model leading on general reasoning can be beaten on your extraction task by something a twentieth of the price.
They also do not measure the things that decide deployments: cost at your volume, latency at your concurrency, whether the licence permits what you need, and whether the data may go there at all. A benchmark score is one input among six and it is the last one to consider, not the first.
The one thing leaderboards are genuinely good for is generating a shortlist — knowing which models are broadly in contention this quarter. Used for that and no more, they save time.
A hundred to a few hundred real examples with correct answers is usually enough to separate candidates decisively. Realism matters more than size: examples drawn from actual work, including the awkward ones, beat a larger set of clean invented cases every time.
When something changes — a new release, a price change, a deprecation notice, or a shift in your own workload. Roughly twice a year in practice. If the system was built for substitution it is a short exercise; if it was not, that is the real finding.
You can, and you will usually overpay for it. Most systems benefit from at least two: a small one for routine work and a larger one for the hard minority. That is a routing decision rather than a selection one, and it is where a great deal of cost saving lives.
The shortlist is usually three candidates and the test takes two days. It is not the hard part.
30-minute technical call · no deck
What we built, what stage it is at, and a link to the live thing wherever there is one.
yourOS is an operating system with an AI at its heart. You talk to it like a person, out loud or typed. It makes your calls, plans your days and works on your projects with a crew of agents, on your own private server. Our own product, built from the first commit; the waitlist is open.
Website, backend and app assets for Sonora, a free AI sound-therapy app on iOS and Android published by Cognitive AI.
Brand, website, prototype, apps on five platforms, the AI backend and the algorithm-intelligence pipeline for Podswap, a social growth engine that runs without a human in the loop.
In closed beta: Roxy, a private AI assistant with permanent memory running on infrastructure the user owns. Brand, site, product, backend and conversational design.
Live at coherence.com: brand identity generated with AI agents we directed, the motion system and website for the group behind Sonora, AboutAI, WakeUp and MudShadows. Agent work remains in build.
Live at domainbooth.com for DomainBooth, a premium domain brokerage: a new animated mark, every page redesigned and rebuilt with designed mobile layouts, and a custom CMS for its sales record, featured names, press and enquiries.
Apps, platforms and campaigns for these teams, long before the AI work. Every one is written up on the work page — what we did, and where our part ended.
