The AI Model Router: Stop Paying Your Best Model to Do Simple Work

Before you start

You have an application that calls an LLM API. You are using GPT-4o or Claude Sonnet for everything: classifying support tickets, extracting names from emails, answering customer questions, and also for complex code generation and multi-step reasoning. Your monthly API bill is higher than it should be, and you suspect a lot of that spend is wasted on trivial tasks that a cheaper model could handle.

This article shows you how to build a model router: a small service that inspects each request, estimates its complexity, and sends it to the cheapest model that can do the job. You will write real code, run it against live APIs, and measure the cost savings. By the end, you will have a working router you can drop into your own stack.

You need a few things before we start: an OpenAI API key (or Anthropic, but the examples use OpenAI), Node.js 18 or later, and a terminal. The code is TypeScript, but the concepts apply to any language.

• An OpenAI API key with billing enabled (you will spend a few cents during testing).
• Node.js 18+ installed on your machine.
• A code editor and a terminal open in a fresh project directory.
• Basic familiarity with fetch and async/await in JavaScript.

Step 1: Understand the cost difference

For a single request with 5K input and 1K output tokens, GPT-4o costs about $0.0225, while GPT-4o mini costs about $0.00135. That is a 16x difference. Multiply that across thousands of requests a day, and the savings are real.

But again, you cannot just pick the cheapest model for everything. You need a routing strategy.

cat << 'EOF' > cost-comparison.js
const models = [
  { name: 'gpt-4o', input: 2.50, output: 10.00 },
  { name: 'gpt-4o-mini', input: 0.15, output: 0.60 },
];

function cost(model, inputTokens, outputTokens) {
  return (inputTokens / 1e6 * model.input) + (outputTokens / 1e6 * model.output);
}

const inputTokens = 5000;
const outputTokens = 1000;

for (const m of models) {
  console.log(`${m.name}: $${cost(m, inputTokens, outputTokens).toFixed(4)}`);
}
EOF
node cost-comparison.js

Step 2: Design the router

A model router is a function that takes a prompt (and optionally metadata like the task type) and returns a model name. The decision can be based on rules, a classifier, or a heuristic. In this article, we build a rule-based router with a fallback: if the router is unsure, it sends the request to the larger model.

Here is the architecture: your application calls the router's route() function. The router inspects the prompt and decides. If the prompt is short, has no code, and looks like a simple classification or extraction, it uses gpt-4o-mini. Otherwise, it uses gpt-4o. You can extend this with more models and more nuanced rules.

We also add a confidence threshold. If the router is not confident, it defaults to the powerful model. This prevents quality regressions on edge cases.

Let's implement the router in TypeScript.

// router.ts
type ModelName = 'gpt-4o-mini' | 'gpt-4o';

interface RouteDecision {
  model: ModelName;
  reason: string;
}

const SIMPLE_PATTERNS = [
  /^classify/i,
  /^extract/i,
  /^summarize/i,
  /^translate/i,
  /^parse/i,
];

const CODE_INDICATORS = [
  '```',
  'function',
  'const ',
  'def ',
  'import ',
  'class ',
];

export function route(prompt: string): RouteDecision {
  const lower = prompt.toLowerCase();
  const isSimple = SIMPLE_PATTERNS.some((re) => re.test(lower));
  const hasCode = CODE_INDICATORS.some((ind) => prompt.includes(ind));
  const isShort = prompt.length < 300;

  if (isSimple && !hasCode && isShort) {
    return { model: 'gpt-4o-mini', reason: 'simple task, no code, short prompt' };
  }

  if (hasCode || prompt.length > 2000) {
    return { model: 'gpt-4o', reason: 'contains code or long prompt' };
  }

  // Default to the powerful model when unsure.
  return { model: 'gpt-4o', reason: 'not confident, fallback' };
}

Step 3: Call the OpenAI API with the chosen model

Now that we have a router, we need a function that sends the prompt to the selected model and returns the response. We will use the OpenAI chat completions API. You will need your API key in an environment variable.

Here is the function that makes the API call. It takes the prompt and the model name, and returns the assistant's message content.

// callModel.ts
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export async function callModel(prompt: string, model: string): Promise<string> {
  const response = await openai.chat.completions.create({
    model: model,
    messages: [{ role: 'user', content: prompt }],
    temperature: 0.2,
  });

  return response.choices[0]?.message?.content ?? '';
}

Step 4: Put it together with a cost tracker

Run this script with your API key set. You will see that the first two prompts go to gpt-4o-mini, and the last two go to gpt-4o. The cost difference is clear in the logs.

// router.ts (complete)
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

type ModelName = 'gpt-4o-mini' | 'gpt-4o';

const PRICING: Record<ModelName, { input: number; output: number }> = {
  'gpt-4o-mini': { input: 0.15, output: 0.60 },
  'gpt-4o': { input: 2.50, output: 10.00 },
};

const SIMPLE_PATTERNS = [
  /^classify/i,
  /^extract/i,
  /^summarize/i,
  /^translate/i,
  /^parse/i,
];

const CODE_INDICATORS = ['```', 'function', 'const ', 'def ', 'import ', 'class '];

export function route(prompt: string): { model: ModelName; reason: string } {
  const lower = prompt.toLowerCase();
  const isSimple = SIMPLE_PATTERNS.some((re) => re.test(lower));
  const hasCode = CODE_INDICATORS.some((ind) => prompt.includes(ind));
  const isShort = prompt.length < 300;

  if (isSimple && !hasCode && isShort) {
    return { model: 'gpt-4o-mini', reason: 'simple task' };
  }
  if (hasCode || prompt.length > 2000) {
    return { model: 'gpt-4o', reason: 'complex task' };
  }
  return { model: 'gpt-4o', reason: 'fallback' };
}

function estimateTokens(text: string): number {
  return Math.ceil(text.length / 4);
}

async function run(prompt: string) {
  const { model, reason } = route(prompt);
  const inputTokens = estimateTokens(prompt);
  const start = Date.now();
  const response = await openai.chat.completions.create({
    model,
    messages: [{ role: 'user', content: prompt }],
    temperature: 0.2,
  });
  const outputText = response.choices[0]?.message?.content ?? '';
  const outputTokens = estimateTokens(outputText);
  const latency = Date.now() - start;
  const cost = (inputTokens / 1e6 * PRICING[model].input) + (outputTokens / 1e6 * PRICING[model].output);

  console.log(`Prompt: ${prompt.slice(0, 50)}...`);
  console.log(`Model: ${model} (${reason})`);
  console.log(`Input tokens: ${inputTokens}, Output tokens: ${outputTokens}`);
  console.log(`Latency: ${latency} ms`);
  console.log(`Estimated cost: $${cost.toFixed(6)}\n`);
}

async function main() {
  const prompts = [
    'Classify this review: "The product is great but shipping was slow."',
    'Extract the name and email from this text: "Contact John Doe at john@example.com."',
    'Write a TypeScript function to debounce an input.',
    'Explain the difference between TCP and UDP in detail.',
  ];

  for (const p of prompts) {
    await run(p);
  }
}

main().catch(console.error);

Step 5: Verify it worked

After running the script, check the output. You should see that simple classification and extraction tasks are sent to gpt-4o-mini, while code generation and complex explanations go to gpt-4o. The estimated cost for the simple tasks should be a fraction of a cent, while the complex tasks cost more.

To verify the quality is acceptable, read the responses for the simple tasks. They should be accurate and complete. If you find that your simple tasks sometimes need the larger model, adjust the rules. For example, you might require that the prompt be shorter, or you might add more patterns.

Here is a checklist to confirm your router is working correctly.

• Simple classification prompts are routed to gpt-4o-mini.
• Code generation prompts are routed to gpt-4o.
• Long prompts (over 2000 characters) go to gpt-4o.
• The cost per simple request is under $0.001.
• The responses for simple tasks are accurate and complete.

Troubleshooting

If you run into issues, here are common problems and fixes.

• If you get an authentication error, check that OPENAI_API_KEY is set correctly in your environment.
• If you get a rate limit error, wait a few seconds and retry. The script makes four calls in quick succession.
• If the router sends everything to gpt-4o, check your prompt lengths and patterns. The fallback intentionally defaults to the larger model when unsure.
• If you want to use Anthropic models, replace the OpenAI client with the Anthropic SDK and adjust the pricing constants.

Step 6: Add a confidence score

This approach is more flexible because it does not rely on hard-coded patterns. The trade-off is an extra API call, but the mini model call is cheap. You can cache the complexity score for repeated prompts to avoid the overhead.

// confidenceRouter.ts
import OpenAI from 'openai';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function rateComplexity(prompt: string): Promise<number> {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [
      { role: 'system', content: 'Rate the complexity of the following user request on a scale from 1 (trivial) to 5 (very complex). Respond with only the number.' },
      { role: 'user', content: prompt },
    ],
    temperature: 0,
  });
  const score = parseInt(response.choices[0]?.message?.content ?? '3', 10);
  return isNaN(score) ? 3 : score;
}

export async function smartRoute(prompt: string): Promise<'gpt-4o-mini' | 'gpt-4o'> {
  const score = await rateComplexity(prompt);
  return score <= 3 ? 'gpt-4o-mini' : 'gpt-4o';
}

What I would do: Recommended setup

This setup gives you a good balance of cost and quality. You can adjust the threshold and the cache size based on your traffic.

// productionRouter.ts
import OpenAI from 'openai';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const cache = new Map<string, number>();

async function getComplexity(prompt: string): Promise<number> {
  if (cache.has(prompt)) return cache.get(prompt)!;
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [
      { role: 'system', content: 'Rate complexity 1-5. Reply with number only.' },
      { role: 'user', content: prompt },
    ],
    temperature: 0,
  });
  const score = parseInt(response.choices[0]?.message?.content ?? '3', 10);
  cache.set(prompt, score);
  return score;
}

export async function route(prompt: string): Promise<string> {
  // Quick rule-based check
  if (prompt.length < 100 && !prompt.includes('```')) {
    return 'gpt-4o-mini';
  }
  const score = await getComplexity(prompt);
  return score <= 3 ? 'gpt-4o-mini' : 'gpt-4o';
}

FAQ

Answers to the questions that come up most often on this topic.

• Q: Will the router work with other LLM providers? A: Yes, the same logic applies to Anthropic, Google, or open-source models. You only need to change the API client and pricing table.
• Q: How do I know the cheap model is good enough? A: Test on your own data. Run a sample of your tasks through both models and compare outputs. If the cheap model fails more than a few percent, adjust the routing threshold.
• Q: Can I use this router for agentic workflows? A: Absolutely. In an agent loop, you can route each tool call or sub-task independently. Simple tool calls go to the cheap model, complex reasoning goes to the big model.
• Q: What about latency? A: The mini model is often faster, so routing to it can reduce latency. The classifier call adds a small overhead, but it is usually negligible compared to the main call.

Next action

Now that you have a working model router, the next step is to integrate it into your application. Replace your direct API calls with the router, and monitor the cost and quality over a week. You will likely see a significant drop in your API bill without a noticeable change in user experience.

Start by running the complete router script from Step 4 with your own prompts. Then adapt the rules to your specific tasks. The key is to measure the impact, not just assume it works.