Build an AI Agent That Fixes Failing Tests

Before you start

You have a failing test. The stack trace points to a function you wrote three weeks ago, and the fix is not obvious. You could spend an hour reading code, but what if a script could do that for you?

In this tutorial, you will build an AI agent that reads a failing test, inspects the relevant source files, proposes a fix, and applies it. You will use TypeScript, the OpenAI API, and a simple tool-use loop. The agent will run a test command, parse the output, read files, and write patches.

• Node.js 18 or later installed
• An OpenAI API key with access to gpt-4o-mini or similar
• A project with a failing test you can experiment on (or use the sample project below)
• Basic familiarity with TypeScript and async/await

Step 1: Set up the project

Create a new directory and initialize a TypeScript project. You will use tsx to run TypeScript directly, and the official OpenAI Node SDK for API calls.

mkdir ai-test-fixer
cd ai-test-fixer
npm init -y
npm install typescript tsx @types/node openai
npx tsc --init --target es2022 --module nodenext --outDir dist --rootDir . --strict

Step 2: Create a sample failing test

Install vitest and add a test script to package.json. Run the test to confirm it fails.

npm install -D vitest
# add "test": "vitest run" to package.json scripts
npm test

Step 3: Define the agent's tools

The agent needs tools to interact with the filesystem and run commands. Define a set of functions that the model can call: run_test, read_file, write_file, and list_files. Each tool has a schema that describes its parameters.

// src/tools.ts
import { execSync } from 'node:child_process';
import { readFileSync, writeFileSync, readdirSync } from 'node:fs';
import { join } from 'node:path';

export const tools = [
  {
    type: 'function',
    function: {
      name: 'run_test',
      description: 'Run the test suite and return the output.',
      parameters: { type: 'object', properties: {} }
    }
  },
  {
    type: 'function',
    function: {
      name: 'read_file',
      description: 'Read the contents of a file.',
      parameters: {
        type: 'object',
        properties: { path: { type: 'string' } },
        required: ['path']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'write_file',
      description: 'Write content to a file.',
      parameters: {
        type: 'object',
        properties: {
          path: { type: 'string' },
          content: { type: 'string' }
        },
        required: ['path', 'content']
      }
    }
  }
] as const;

export function runTool(name: string, args: any): string {
  switch (name) {
    case 'run_test':
      try {
        return execSync('npm test -- --reporter=verbose', { encoding: 'utf-8' });
      } catch (e: any) {
        return e.stdout?.toString() || e.message;
      }
    case 'read_file':
      return readFileSync(args.path, 'utf-8');
    case 'write_file':
      writeFileSync(args.path, args.content);
      return 'File written successfully.';
    default:
      return 'Unknown tool.';
  }
}

Step 4: Build the agent loop

The core of the agent is a loop: send the conversation to the model, check if it wants to call a tool, execute the tool, and feed the result back. Repeat until the model produces a final answer or the test passes.

// src/agent.ts
import OpenAI from 'openai';
import { tools, runTool } from './tools.js';

const client = new OpenAI();

const systemPrompt = `You are an AI assistant that fixes failing tests. You have access to tools. Use them to diagnose the failure and propose a fix. When you are confident, write the fix to the file. Always run the test after writing to verify.`;

export async function runAgent(initialPrompt: string) {
  const messages: any[] = [
    { role: 'system', content: systemPrompt },
    { role: 'user', content: initialPrompt }
  ];

  for (let i = 0; i < 10; i++) {
    const response = await client.chat.completions.create({
      model: 'gpt-4o-mini',
      messages,
      tools: tools as any,
      tool_choice: 'auto'
    });

    const message = response.choices[0].message;
    messages.push(message);

    if (message.tool_calls) {
      for (const call of message.tool_calls) {
        const result = runTool(call.function.name, JSON.parse(call.function.arguments));
        messages.push({
          role: 'tool',
          tool_call_id: call.id,
          content: result
        });
      }
    } else {
      console.log('Final answer:', message.content);
      return;
    }
  }
  console.log('Max iterations reached.');
}

Step 5: Run the agent

Create an entry point that invokes the agent with the failing test description. Run it and watch the agent work through the tools.

// src/index.ts
import { runAgent } from './agent.js';

const prompt = `The test 'rejects missing @' is failing. Diagnose the issue and fix it.`;

runAgent(prompt).catch(console.error);

npx tsx src/index.ts

• The agent will first run the test to see the failure output.
• It will then read the source file and identify the missing @ check.
• It will write a corrected version of the function.
• Finally, it will run the test again to confirm the fix.

Verify it worked

After the agent finishes, run the test suite yourself to confirm both tests pass. If the agent did not fix it correctly, you may need to adjust the system prompt or add more tools like a search tool.

npm test

Recommended setup for real projects

For a production-ready agent, you need more than a single loop. You should add a sandboxed execution environment, better error handling, and a way to review changes before they are applied. Here is a starter configuration that includes a permission prompt and a dry-run mode.

# .env
OPENAI_API_KEY=sk-...
AGENT_MAX_ITERATIONS=10
AGENT_DRY_RUN=true

// src/safeWrite.ts
import { writeFileSync } from 'node:fs';

export function safeWrite(path: string, content: string) {
  if (process.env.AGENT_DRY_RUN === 'true') {
    console.log(`[DRY RUN] Would write to ${path}`);
    console.log(content);
    return;
  }
  writeFileSync(path, content);
}

• Run the agent in a Docker container or CI environment to isolate it from your local filesystem.
• Use a stricter model like gpt-4o for complex codebases, or a faster model like gpt-4o-mini for simple fixes.
• Add a tool to search for symbols or grep for patterns to help the agent navigate large codebases.
• Always review the diff before merging the agent's changes.

Troubleshooting

If the agent fails to fix the test, check the following:

The model may not have enough context. Provide the full test file and source file in the initial prompt.

The tool output might be truncated. Increase the max_tokens or limit the output size.

The agent might loop. Add a maximum iteration count and a timeout.

If the test command has side effects, run it in a sandbox.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can this agent fix any test failure?
• A: No, it works best for deterministic failures with clear error messages. Flaky tests or tests that require external services will confuse it.
• Q: How much does it cost to run?
• A: For a simple fix, you might use 10-20K tokens. At gpt-4o-mini pricing, that is under a cent. For complex fixes with gpt-4o, expect a few cents.
• Q: Is it safe to let the agent write files automatically?
• A: Not without review. Always use a dry-run mode or a human-in-the-loop approval step for production use.
• Q: Can I use other LLMs?
• A: Yes, any model with function calling support works. You can swap the OpenAI client for Anthropic or a local model.

Next step

Run the agent on your own failing test. Start with a small, isolated failure and observe the tool calls. Then extend the agent with more tools and a review step.