Model Routing — Open-Source vs Premium for Cost Control

The default configuration for a new agentic system is to use the most capable model available for every call. This is the safe choice when you have not measured what the system actually needs — premium models are more reliable on ambiguous tasks, less likely to produce malformed outputs, and require less prompt engineering to get consistent results. But it is also the expensive choice, often by an order of magnitude.

The cost ratio between a fast, inexpensive model tier and a full-reasoning premium tier typically ranges from 10× to 50× on input token pricing, and varies further by provider and by model generation. Check current rates on provider pricing pages — for Anthropic at anthropic.com/pricing, for OpenAI at openai.com/api/pricing. The ratio changes with each model release; pin to specific model versions, not tier aliases, so your cost model does not silently shift when a new model replaces a tier default.

In a well-instrumented agentic system, most agent operations are not equally hard. Classifying a log line, extracting function signatures from source code, or reformatting a data structure are tasks where correctness is verifiable and the output space is bounded. These do not need the same model as designing a migration strategy for a 200-module monolith. Model routing is the practice of matching task difficulty to model tier — systematically, at the orchestration layer, rather than by defaulting to premium and hoping for the best.

Task-difficulty taxonomy

The most useful taxonomy for agentic code engineering work has three tiers, defined by the nature of the reasoning required rather than by the surface-level complexity of the prompt.

Tier 1 — Classification and extraction (~60–70% of calls in a typical agentic workflow). This tier covers tasks where the output is structured, bounded, and verifiable. Examples: does this function signature match a known anti-pattern? Extract all import statements from this file. Classify this log line as one of: error, warning, info, or debug. Parse this JSON schema and return the required fields. Reformat this diff into a standard commit message. The outputs of Tier 1 tasks are checkable by a deterministic test — either the function signature matches the pattern or it does not, either the JSON is valid or it is not. Tier 1 tasks also tolerate occasional model errors well: if the classification is wrong, the pipeline catches it at the next validation step rather than propagating an incorrect decision through multiple downstream stages.

Tier 2 — Synthesis and generation (~25–30% of calls). This tier covers tasks that require producing coherent new content from multiple inputs, where correctness is less mechanically verifiable but there is still a clear right answer. Examples: write a refactor for this 40-line function that addresses these three constraints. Draft a migration for this schema change, preserving all existing foreign key relationships. Summarise the key behaviour changes in this diff. Translate this module from Go to Python, preserving the public API contract. Tier 2 tasks require the model to make judgment calls about ambiguity and produce output that another agent or human will use directly. They need a model with stronger generation capability than Tier 1, but do not require the full extended-reasoning capabilities of the premium tier.

Tier 3 — Reasoning and design (~5–10% of calls). This tier covers tasks where the answer is genuinely uncertain and depends on cross-cutting context that does not fit cleanly into a single prompt. Examples: propose an incremental migration path for this 80k-line framework transition, identifying the slice ordering that minimises regression risk. Identify the root cause of a failure mode that occurs across five interacting services. Evaluate whether this proposed architecture change creates a circular dependency given these 200 module relationships. Tier 3 tasks are the ones where model capability makes a measurable difference in output quality — and they are a small fraction of the total operation count in most systems.

Routing decision tree

The routing decision happens at the task dispatch layer, before a model is selected. The key question is: does this task require extended reasoning over uncertain inputs, or is there a verifiable correct answer?

The routing signal is the key: if you can describe what "correct" looks like without using the LLM's judgment to verify it — a regex, a schema, a CI test, an expected output value — the task is Tier 1 or Tier 2. If verifying correctness requires the same level of reasoning as producing the answer, the task is Tier 3.

Tier-based architecture at the gateway level

Routing logic belongs at the orchestration layer, not inside individual agents. Each agent receives a task type label as part of its dispatch envelope. The orchestration layer maps task types to model identifiers before dispatching to the agent. This separation means that when a new model tier is introduced (a new fast model at lower cost, or a new premium reasoning tier), the routing table changes in one place — the orchestration config — without touching any agent code.

ROUTING_TABLE = {
    # Tier 1 — fast, cheap; classification + extraction
    "classify":       "fast-model-v1",
    "extract":        "fast-model-v1",
    "parse":          "fast-model-v1",
    "lint":           "fast-model-v1",
    "diff-label":     "fast-model-v1",

    # Tier 2 — balanced; generation + synthesis
    "generate":       "balanced-model-v1",
    "refactor":       "balanced-model-v1",
    "migrate-fn":     "balanced-model-v1",
    "summarise":      "fast-model-v1",  # most summaries are Tier 1

    # Tier 3 — full reasoning; design + root-cause
    "architecture":   "reasoning-model-v1",
    "root-cause":     "reasoning-model-v1",
    "migrate-system": "reasoning-model-v1",
}

def route(task_type: str) -> str:
    model = ROUTING_TABLE.get(task_type)
    if model is None:
        # Unknown task type: default to Tier 2 (not Tier 3)
        # Fail-safe upward routing happens at retry, not at initial dispatch
        return "balanced-model-v1"
    return model

Replace the placeholder model identifiers with the specific version strings for your provider. Do not use floating aliases like "claude-sonnet-latest" — pin to a specific version so routing behaviour does not change silently when a provider updates the alias. Update the routing table and redeploy when you intentionally change model versions after evaluation.

Real cost-saving math on a sample workload

To make the cost impact concrete: consider a code-review agentic workflow that processes 100 pull requests per day. Each PR review is decomposed into five sub-tasks: (1) lint-check the diff for style violations; (2) extract function signatures changed in the diff; (3) classify the change type (feature / fix / refactor / chore); (4) draft a review summary; (5) flag any security or performance concerns that require senior attention.

Task mapping: (1) lint-check = Tier 1; (2) extract signatures = Tier 1; (3) classify change type = Tier 1; (4) draft summary = Tier 2; (5) security/performance flag = Tier 2–3 depending on findings.

Average token cost per task (illustrative, mid-2026 rates from public pricing pages; check current rates before using for planning):

At 100 PRs per day, the routed workflow costs approximately $1.03/day vs $4.90/day all-premium — a 79% reduction. Over a month (21 working days), that is $21.63 routed vs $102.90 all-premium. The savings from routing outpace the implementation cost within the first billing cycle for any team running more than a few dozen automated reviews per day.

These numbers are illustrative. Token counts per sub-task depend on your diff size, context included, and prompt structure. The cost-saving direction is consistent across workloads, but measure your own token distribution before committing to a routing architecture design. The right instrumentation to build before routing: log the exact token count and model used for every sub-task type for two weeks on your existing workload, then use that distribution to size the routing table.

Failure mode: the downgrade-cascade pattern

The most common failure mode in model routing is not bad routing decisions — it is good routing decisions that generate retry loops back to premium models. The pattern looks like this: a Tier 1 task is routed to a fast model; the fast model produces output that fails the downstream validation check; the orchestration retries with the next tier up; the Tier 2 model also fails the check on a particularly ambiguous input; the orchestration escalates to Tier 3. The final successful call is at premium cost, plus the cost of the two failed attempts at lower tiers. If this escalation pattern fires on 20% of Tier 1 tasks, the routing savings are substantially eroded.

The root cause is almost always one of two things: the task type was misclassified (a Tier 2 task was labeled Tier 1 and routed to a fast model that cannot handle it), or the validation check is too strict (the Tier 1 output is correct but fails a downstream schema that was designed for Tier 3 output verbosity).

Prevention:

• Measure escalation rate by task type. If lint-check tasks escalate more than 3% of the time, the fast model may be under-performing on your specific diff format. Either adjust the model selection for that task type or simplify the prompt to work within the fast model's capabilities.
• Design output schemas for the model tier, not for the human reader. A Tier 1 task that produces a JSON object with three fields should have a downstream schema that accepts those three fields. If the downstream consumer was designed expecting the verbose, explanatory output of a premium model, the validation will fail on the structured output of a fast model even when the fast model is technically correct.
• Retry policy: up one tier, once. If a Tier 1 task fails, retry with a Tier 2 model and log the escalation. If the Tier 2 model fails, do not retry with Tier 3 — escalate to a human-review queue. Cascades to Tier 3 on automated retry are rarely worth the cost; they should be operator-reviewed decisions.

When to consider open-source self-hosted models

Open-source models (such as those from the Llama family at ai.meta.com/llama, Mistral at mistral.ai, and others) introduce a different cost structure: infrastructure cost instead of per-token API cost. For Tier 1 tasks at high volume — millions of calls per day — self-hosted open-source models can reduce per-call cost substantially compared to API-billed calls. The trade-off is operational overhead: you run the inference infrastructure, you manage model updates, and you bear the cost of the GPU or accelerator hardware.

Self-hosting is worth evaluating when: (a) your Tier 1 call volume is high enough that the GPU rental cost is lower than the API cost for the same volume — typically above 5 million Tier 1 calls per month for a mid-range GPU setup; (b) your Tier 1 tasks are well-defined enough that a smaller open-source model can handle them without frequent escalation; (c) your data governance requirements prevent sending code or data to external API providers.

Self-hosting is not worth the complexity when call volume is moderate, when Tier 1 tasks require more instruction-following capability than smaller open-source models reliably provide, or when your team does not have infrastructure expertise for GPU cluster management. The API-based tier model described in this guide is the right starting point; consider self-hosting for Tier 1 only when you have measured that the API cost at your actual call volume exceeds the infrastructure cost of a self-hosted alternative.

A routing-governance layer makes these decisions auditable across a team — routing changes get reviewed before deployment rather than discovered in the next billing cycle. Treat routing config as code: PR'd, reviewed, version-controlled, and observable in your telemetry alongside the per-call costs it produces.