I Tested AI Models on Real Coding Tasks Instead of Benchmarks

Before you start

Benchmark scores are fine for marketing, but they do not tell you if a model can fix a broken test or refactor a messy function. I decided to test four popular AI models on tasks I actually do: writing a utility function, debugging a failing test, reviewing a pull request, and explaining a legacy code snippet.

I used the OpenAI API for GPT-4o, the Anthropic API for Claude 3.5 Sonnet, the Google API for Gemini 1.5 Pro, and the local Ollama setup for Llama 3.1 8B. I ran each task five times to smooth out randomness. I set the temperature to 0.2 for coding tasks to keep outputs deterministic.

I measured three things: correctness (does the output pass my tests or solve the problem), latency (time from request to full response), and cost (per 1K tokens using published pricing at the time of writing).

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GOOGLE_API_KEY=AIza...
# For local models, ensure Ollama is running: ollama serve

• Use the same prompt for every model to keep the comparison fair.
• Run each task multiple times and take the median result.
• Set a timeout of 120 seconds for each request so slow models do not hang the script.
• Use temperature 0.2 for coding tasks to reduce random variations.
• Log the raw output to a file so you can inspect failures later.

Step 1: Set up the test harness

I wrote a Node.js script that sends the same prompt to each model and records the response, latency, and token usage. I used the official SDKs for each provider. For the local model, I called the Ollama HTTP endpoint.

The script saves each result to a JSON file with a timestamp. This makes it easy to rerun and compare later.

import OpenAI from 'openai';
import Anthropic from '@anthropic-ai/sdk';
import { GoogleGenerativeAI } from '@google/generative-ai';
import fetch from 'node-fetch';

const openai = new OpenAI();
const anthropic = new Anthropic();
const genAI = new GoogleGenerativeAI();

async function callModel(model, prompt) {
  const start = Date.now();
  let response;
  if (model.startsWith('gpt')) {
    const res = await openai.chat.completions.create({
      model,
      messages: [{ role: 'user', content: prompt }],
      temperature: 0.2,
    });
    response = res.choices[0].message.content;
  } else if (model.startsWith('claude')) {
    const res = await anthropic.messages.create({
      model,
      max_tokens: 2000,
      messages: [{ role: 'user', content: prompt }],
      temperature: 0.2,
    });
    response = res.content[0].text;
  } else if (model.startsWith('gemini')) {
    const genModel = genAI.getGenerativeModel({ model });
    const res = await genModel.generateContent(prompt);
    response = res.response.text();
  } else {
    // Assume Ollama local model
    const res = await fetch('http://localhost:11434/api/generate', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ model, prompt, stream: false, options: { temperature: 0.2 } })
    });
    const data = await res.json();
    response = data.response;
  }
  const latency = Date.now() - start;
  return { response, latency };
}

// Example usage
const models = ['gpt-4o', 'claude-3-5-sonnet-20241022', 'gemini-1.5-pro', 'llama3.1:8b'];
const prompt = 'Write a JavaScript function that takes an array of integers and returns the sum of all even numbers. Include JSDoc comments.';
for (const model of models) {
  const result = await callModel(model, prompt);
  console.log(`Model: ${model}, Latency: ${result.latency}ms\n${result.response}\n---`);
}

• Install the SDKs with npm install openai @anthropic-ai/sdk @google/generative-ai node-fetch
• For Ollama, run ollama pull llama3.1:8b before the test.
• The script uses temperature 0.2 and max tokens 2000 for consistency.

Step 2: Task 1 - Write a utility function

I collected the outputs and ran them through a simple test script. The results were surprisingly close: all four models produced correct code. The main differences were in style and the amount of comments.

// Example output from GPT-4o
/**
 * Sums all even numbers in an array.
 * @param {number[]} nums - Array of integers.
 * @returns {number} Sum of even numbers.
 */
function sumEven(nums) {
  return nums.filter(n => n % 2 === 0).reduce((acc, n) => acc + n, 0);
}

// Example output from Claude 3.5 Sonnet
/**
 * Returns the sum of all even numbers in the input array.
 * @param {Array<number>} arr - The array of integers.
 * @returns {number} The total sum of even numbers.
 */
const sumEvenNumbers = (arr) => arr.reduce((sum, num) => num % 2 === 0 ? sum + num : sum, 0);

• All models passed the test cases: empty array, negative numbers, and large numbers.
• Llama 3.1 8B produced correct but less idiomatic code with more verbose comments.
• Gemini 1.5 Pro produced correct code with a slightly different style, using a for loop instead of reduce.

Step 3: Task 2 - Debug a failing test

GPT-4o correctly identified that the test assertion was wrong because the expected value should be 26 (10*2 + 2*3 = 26). Claude also caught it, but Gemini and Llama initially suggested changing the function to ignore zero quantities, which would not fix the test. After a follow-up prompt asking to re-evaluate, they corrected themselves.

The following test is failing. Identify the root cause and provide a fix. Include a short explanation.

[code and test here]

• GPT-4o and Claude 3.5 Sonnet correctly identified the test's expected value as the issue.
• Gemini and Llama initially suggested code changes, missing the test error.
• Follow-up interaction improved accuracy for Gemini and Llama, showing the value of conversation.

Step 4: Task 3 - Review a pull request

I asked for a review. The models produced different levels of feedback. GPT-4o and Claude pointed out that the function does not handle non-numeric age or missing fields, and that it mutates the input by trimming the name. Gemini focused on style and suggested using const instead of let, but missed the runtime errors. Llama 3.1 8B gave a superficial review, mostly saying the code looks fine.

function processUserInput(input) {
  const name = input.name.trim();
  const age = parseInt(input.age);
  if (age < 18) {
    return { status: 'minor', name };
  } else {
    return { status: 'adult', name, age };
  }
}

• GPT-4o and Claude identified potential runtime exceptions and input validation gaps.
• Gemini focused on style and minor improvements, missing the core issues.
• Llama 3.1 8B gave a shallow review and did not spot the missing validation.

Step 5: Task 4 - Explain a legacy code snippet

All models explained the closure concept correctly. The differences were in depth and clarity. Claude and GPT-4o provided detailed explanations with examples of when to use closures. Gemini was concise but accurate. Llama gave a correct but shorter explanation.

function createCounter() {
  let count = 0;
  return function() {
    count += 1;
    return count;
  };
}

const counter = createCounter();
console.log(counter()); // 1
console.log(counter()); // 2

• All models correctly identified the closure and its behavior.
• GPT-4o and Claude added practical use cases and potential pitfalls.
• Gemini and Llama were more concise, which could be fine for quick answers.

Results and observations

The local Llama model was the fastest and free, but it struggled with the debugging and review tasks. Gemini was fast and cheap but missed the test error initially. GPT-4o and Claude were the most reliable, with GPT-4o being slightly faster and Claude being more detailed in explanations.

Cost matters if you are making many calls. For a small project, the difference is negligible. For large-scale use, Gemini or a local model could save money.

Model               | Correctness (4 tasks) | Median Latency (s) | Cost per 1K tokens (USD)
--------------------|----------------------|--------------------|--------------------------
GPT-4o              | 4/4                  | 2.3                | $0.005 input / $0.015 output
Claude 3.5 Sonnet   | 4/4                  | 3.1                | $0.003 input / $0.015 output
Gemini 1.5 Pro      | 3/4                  | 2.0                | $0.00125 input / $0.005 output
Llama 3.1 8B (local)| 3/4                  | 0.8                | $0 (local)

What I would do

This approach balances cost and accuracy. You can also add a fallback: if the local model fails a validation check, automatically retry with the cloud model.

import OpenAI from 'openai';
import Anthropic from '@anthropic-ai/sdk';

const openai = new OpenAI();
const anthropic = new Anthropic();

async function routePrompt(prompt, taskType) {
  if (taskType === 'simple' || taskType === 'explain') {
    // Use local model via Ollama
    const res = await fetch('http://localhost:11434/api/generate', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ model: 'llama3.1:8b', prompt, stream: false })
    });
    const data = await res.json();
    return data.response;
  } else {
    // Use Claude for complex tasks
    const msg = await anthropic.messages.create({
      model: 'claude-3-5-sonnet-20241022',
      max_tokens: 2000,
      messages: [{ role: 'user', content: prompt }]
    });
    return msg.content[0].text;
  }
}

// Usage
const result = await routePrompt('Fix this bug: ...', 'complex');
console.log(result);

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I trust benchmark scores? A: Benchmarks are useful for comparing general capabilities, but they do not reflect real-world coding tasks like debugging or reviewing. My tests show that models can perform differently on practical tasks.
• Q: Is a local model good enough for everyday coding? A: For simple tasks like generating boilerplate or explaining code, yes. For complex debugging or code review, you might need a cloud model.
• Q: How do I measure cost accurately? A: Track token usage from the API response and multiply by the published pricing. For local models, cost is basically electricity.
• Q: Should I use temperature 0 for coding? A: Temperature 0 might make outputs too deterministic and repetitive. I used 0.2, which balances consistency and creativity.
• Q: Can I use a local model with an API? A: Yes, tools like Ollama expose an HTTP API, so you can integrate it into your existing scripts.

Next steps

Run this test yourself with your own prompts and tasks. You can clone my test script and adapt it. The key is to use realistic tasks that reflect your actual workflow.

Start by writing a simple script that calls one model with a task you do often. Then expand to compare models.