Resource

LLM orchestration.
Worth abstracting.

Hardcoding one provider’s API into your application is a dependency worth abstracting. The frontier moves quickly, prices change, models get retired. An orchestration layer is the piece of software that absorbs those changes so the rest of your system does not have to.

WHAT IT IS

An orchestration layer sits between your application and the models it uses. The application sends a request with a task type and a set of constraints; the layer decides which model handles it, on what infrastructure, with what fallback if that fails. Your code talks to one interface. The model behind it can change without a deploy.

That is the whole idea. The value is in three things it lets you do.

WHAT IT BUYS YOU

Routing by task

Routine work — formatting, extraction, classification — goes to a small, cheap model (the current small open-weight class or a provider’s “mini” tier). Work that needs reasoning goes to a frontier model. Anything touching sensitive data goes to a model you host. Routing routine traffic to smaller models is usually the largest single cost lever; the size of the saving depends entirely on your traffic mix, which is why we measure it before quoting it.

Fallback

Providers have outages and rate limits. When the primary model times out or errors, the layer retries against a secondary model automatically; the end user sees a slower response rather than an error. The secondary needs to have been evaluated on the same task, or the fallback is a different product.

Substitution

When a better or cheaper model appears, you evaluate it against your existing test set and add it to the routing table. The application does not change. This reduces vendor dependence; it does not remove it, because prompts tuned for one model rarely transfer untouched.

WHAT IT COSTS

Latency. The routing decision is a classification. A small classifier adds tens of milliseconds; a model-based one adds more. Either is usually recovered by sending routine traffic to faster models — but measure it on your own traffic rather than assuming.

Evaluation. Every model in the routing table is a model you have to keep evaluated. Adding a route is cheap; keeping it honest is a standing cost that belongs in the cost model.

Complexity. One more component, with its own failure modes. For a single-model prototype it is premature. For anything with real traffic and a budget, it is the component you wish you had built first.

QUESTIONS

QUESTIONS — 3

Your application talks to one interface, so swapping the model behind it is a configuration change rather than a rewrite. The remaining lock-in is in the prompts and the evaluation set, both of which need re-checking on a new model. Reduced, not removed.

Yes, a little. A small classifier adds tens of milliseconds; a model-based router adds more. Routing routine requests to faster models usually gives that back and more, but it is a thing to measure, not to assume.

Several open-source gateways cover routing and fallback well. Build only the part that encodes your own policy — which task goes where, what may leave the perimeter — and keep it small enough to read in one sitting.

Keep reading

Architecture

Custom LLM Orchestration

→

Resource

How to Choose a Model

→

Resource

Modelling the Cost of an AI System

→

30-minute technical call · no deck