AI Model Pricing Is Getting Weird - Here's How Developers Should Think About It

The pricing page lies to you

You open an AI provider's pricing page and see $0.15 per million input tokens. You multiply by your expected usage, get a number, and move on. A month later the invoice is 3x what you budgeted. The gap is not a billing error. It is the difference between list price and what your actual workload costs.

Token prices are a starting point, not the whole story. Context caching, output tokens costing more than input, batch discounts, and the hidden cost of retries all change the real number. This article gives you a repeatable method to measure what a model actually costs for your specific traffic, then shows you how to use that number when you pick a model.

• Compare effective price per request, not per token.
• Account for output tokens: they often cost 2-4x input.
• Include retries and failed calls in your cost model.
• Measure latency, because a slow model can cost more in engineering time than in API fees.

Before you start: what you need

You need a model API key from at least one provider. The examples use OpenAI and Anthropic because they are common, but the script works with any REST endpoint. You also need Python 3.9 or newer and the requests library. Install it with pip.

python3 -m venv venv
source venv/bin/activate
pip install requests python-dotenv

Step 1: Build a cost probe script

Create a file called cost_probe.py. This script sends a fixed prompt to a model, measures the token usage from the response, and calculates the cost using the provider's published rates. It also records latency so you can see the tradeoff between speed and price.

The script takes a model name, a prompt file, and a number of runs. It prints a table with average cost per request and p95 latency. You can point it at any OpenAI-compatible endpoint.

import os
import time
import json
import argparse
import requests
from dotenv import load_dotenv

load_dotenv()

def get_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    # Rates in USD per million tokens, update as of 2025-06
    rates = {
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "claude-3-5-sonnet-20240620": {"input": 3.00, "output": 15.00},
        "claude-3-haiku-20240307": {"input": 0.25, "output": 1.25},
    }
    if model not in rates:
        raise ValueError(f"Unknown model: {model}")
    r = rates[model]
    return (input_tokens / 1e6) * r["input"] + (output_tokens / 1e6) * r["output"]

def run_probe(model: str, prompt: str, runs: int, base_url: str, api_key: str):
    costs = []
    latencies = []
    for _ in range(runs):
        start = time.time()
        resp = requests.post(
            f"{base_url}/chat/completions",
            headers={"Authorization": f"Bearer {api_key}"},
            json={
                "model": model,
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": 512,
            },
            timeout=60,
        )
        elapsed = time.time() - start
        resp.raise_for_status()
        data = resp.json()
        usage = data["usage"]
        cost = get_cost(model, usage["prompt_tokens"], usage["completion_tokens"])
        costs.append(cost)
        latencies.append(elapsed)
    avg_cost = sum(costs) / len(costs)
    p95 = sorted(latencies)[int(len(latencies) * 0.95)]
    print(f"Model: {model}")
    print(f"Average cost per request: ${avg_cost:.6f}")
    print(f"P95 latency: {p95:.2f}s")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", required=True)
    parser.add_argument("--prompt-file", required=True)
    parser.add_argument("--runs", type=int, default=10)
    parser.add_argument("--base-url", default="https://api.openai.com/v1")
    args = parser.parse_args()
    with open(args.prompt_file) as f:
        prompt = f.read()
    api_key = os.getenv("API_KEY")
    if not api_key:
        raise SystemExit("Set API_KEY environment variable")
    run_probe(args.model, prompt, args.runs, args.base_url, api_key)

Step 2: Run the probe on a realistic prompt

Create a prompt file that represents your actual workload. Do not use a one-liner. Use something close to what your app sends, such as a support email that needs a summary and a suggested reply.

Then run the probe against two models, for example gpt-4o-mini and claude-3-haiku. Compare the numbers.

cat > prompt.txt << 'EOF'
Summarize the following customer email and draft a polite response that offers a refund.

Email: I ordered a pair of running shoes on May 1 but they arrived with a torn seam. I want a replacement or my money back. This is the second time this has happened.
EOF

python cost_probe.py --model gpt-4o-mini --prompt-file prompt.txt --runs 5

export API_KEY=your-anthropic-key
python cost_probe.py --model claude-3-haiku-20240307 --prompt-file prompt.txt --runs 5 --base-url https://api.anthropic.com/v1

• Run at least 5 requests per model to smooth out variance.
• Use the same prompt file across models so the comparison is fair.
• Note that Anthropic's API path is /v1/chat/completions? Actually it is /v1/messages. The script assumes OpenAI shape, so adjust if needed.

Step 3: Add context caching to the cost model

Context caching is a discount for reusing the same prefix across requests. OpenAI and Anthropic both offer it, but the discount is only applied if you structure your prompts to take advantage. For example, put system instructions and static context at the start, and keep the changing part at the end.

Update the get_cost function to include cached input tokens at a lower rate. The response usage includes cached_tokens for Anthropic and prompt_tokens_details.cached_tokens for OpenAI. Your script should read those fields if present.

def get_cost_with_cache(model, usage):
    rates = {
        "gpt-4o-mini": {"input": 0.15, "cached": 0.075, "output": 0.60},
        "claude-3-haiku-20240307": {"input": 0.25, "cached": 0.03, "output": 1.25},
    }
    if model not in rates:
        raise ValueError(f"Unknown model: {model}")
    r = rates[model]
    input_tokens = usage.get("prompt_tokens", 0)
    cached = 0
    if "prompt_tokens_details" in usage:
        cached = usage["prompt_tokens_details"].get("cached_tokens", 0)
    elif "cache_creation_input_tokens" in usage:
        cached = usage["cache_creation_input_tokens"] + usage.get("cache_read_input_tokens", 0)
    uncached = max(0, input_tokens - cached)
    cost = (uncached / 1e6) * r["input"] + (cached / 1e6) * r["cached"] + (usage.get("completion_tokens", 0) / 1e6) * r["output"]
    return cost

• Cache hits only work when the prefix is exactly the same, so keep dynamic content at the end.
• Some providers require a minimum cache length, such as 1024 tokens, before the discount applies.
• Measure the cache hit rate in production; if it is low, your prompt structure is the problem.

Step 4: Compare total cost of ownership

Per-token price is only part of the decision. A cheaper model that needs 3 retries to get a good answer costs more than a pricier one that works first time. Add a retry factor to your calculation.

Assume a 5% error rate for a cheap model and 1% for a premium model. Multiply the effective cost per request by (1 + error_rate). The script below shows how to factor that in.

def effective_cost(base_cost, error_rate, retry_count=1):
    # Each failed request is retried, so total cost = base * (1 + error_rate * retry_count)
    return base_cost * (1 + error_rate * retry_count)

# Example: gpt-4o-mini base cost $0.0002 per request, 5% error rate
base_mini = 0.0002
eff_mini = effective_cost(base_mini, 0.05)
print(f"gpt-4o-mini effective: ${eff_mini:.6f}")

# Example: claude-3-haiku base cost $0.0003, 2% error rate
base_haiku = 0.0003
eff_haiku = effective_cost(base_haiku, 0.02)
print(f"claude-3-haiku effective: ${eff_haiku:.6f}")

Step 5: Measure your own traffic pattern

The probe gives you a per-request cost, but your traffic is not uniform. You need to know the distribution of prompt sizes and output lengths. Export your logs or use a sample from your database, then compute the average input and output tokens per request.

Use the following script to read a JSONL file of your requests and compute the average token counts. Then plug those numbers into the cost model.

import json
import sys

def average_tokens(file_path):
    total_in = 0
    total_out = 0
    count = 0
    with open(file_path) as f:
        for line in f:
            data = json.loads(line)
            usage = data.get("usage", {})
            total_in += usage.get("prompt_tokens", 0)
            total_out += usage.get("completion_tokens", 0)
            count += 1
    if count == 0:
        return 0, 0
    return total_in / count, total_out / count

if __name__ == "__main__":
    avg_in, avg_out = average_tokens(sys.argv[1])
    print(f"Average input tokens: {avg_in:.0f}")
    print(f"Average output tokens: {avg_out:.0f}")

• Export your API logs in JSONL format with usage fields.
• Sample at least 100 requests to get a stable average.
• If you do not have logs, instrument your code to log usage now.

What I would do: a routing strategy

Do not put all traffic on one model. Use a router that sends simple requests to a cheap model and complex ones to a premium model. Start with a rule-based router based on prompt length and task type, then refine with a classifier if needed.

Here is a minimal router in Python that checks the prompt length and keywords to decide which model to call.

def route_prompt(prompt: str) -> str:
    # Heuristic: short and simple -> cheap model, long or complex -> premium
    if len(prompt) < 200 and "refund" not in prompt.lower():
        return "gpt-4o-mini"
    else:
        return "gpt-4o"

# Example usage
print(route_prompt("What is the weather?"))  # gpt-4o-mini
print(route_prompt("Draft a legal contract for a software licensing deal."))  # gpt-4o

• Start with a length threshold and a keyword list.
• Log the routing decision and the actual cost per request.
• Review the logs weekly to tune the threshold.

Troubleshooting common issues

If your cost probe fails with a 401, check your API key and base URL. Anthropic uses a different endpoint path, so adjust the script to call /v1/messages instead of /v1/chat/completions.

If the token counts are missing, the provider might not return usage unless you set a parameter. For OpenAI, ensure you do not set stream=true, because streaming responses do not include usage by default.

• For Anthropic, use the /v1/messages endpoint and include anthropic-version header.
• For OpenAI, set stream=false to get usage in the response.
• If you use a proxy or gateway, verify it forwards the usage fields.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Why is my actual bill higher than the list price? A: You are likely paying for output tokens, retries, and uncached input. Use the probe script to measure the effective cost.
• Q: Is context caching worth it? A: Yes, if your prompts share a large static prefix. Measure the cache hit rate first; it can cut costs by 90% on cached tokens.
• Q: Should I use the cheapest model? A: Not always. A slightly more expensive model that needs fewer retries can be cheaper overall. Factor in error rate.
• Q: How often should I re-evaluate pricing? A: Model prices change frequently. Re-run the probe monthly or when you change your prompt structure.