Building an AI Agent That Writes Tests Before Writing Code

The problem: tests always come after the code

You know the drill. A ticket lands in your board: 'Add a retry with exponential backoff to the payment client.' You open your editor, write the implementation, push it, and then maybe, if you have time, you write a test. The test often comes after the code, and it usually passes because it was written to match the implementation.

That order is backwards. Test-driven development says write the test first, watch it fail, then write the minimum code to make it pass. But TDD is hard to stick to because it requires discipline. The moment you are under pressure, you skip the red phase and go straight to green.

What if you could automate the discipline? What if an AI agent could generate the failing tests before any implementation exists? This article walks through building a practical AI agent that does exactly that: given a function signature and a description of expected behavior, it writes a test file that fails, then you or another agent can implement the code to make it pass.

This is not a theoretical exercise. We will build a working agent using TypeScript, the OpenAI API, and a simple agent loop. You will be able to run it on your machine and see it produce real test files.

Before you start

You need Node.js 18 or later, an OpenAI API key (or any LLM API that supports chat completions), and a basic understanding of TypeScript and Jest. We will use Jest for testing, but the pattern applies to any test framework.

Create a new project directory and initialize it:

mkdir test-first-agent
cd test-first-agent
npm init -y
npm install typescript ts-node @types/node jest ts-jest @types/jest openai dotenv
npx tsc --init

• Make sure your tsconfig.json has 'target' set to ES2020 and 'module' set to commonjs.
• Create a .env file with your OPENAI_API_KEY.
• Add a test script to package.json: 'test': 'jest'.

Step 1: Define the agent's goal and scope

The agent's job is simple: given a TypeScript function signature and a plain-language description of what the function should do, generate a Jest test file that verifies the behavior. The test file should be complete and runnable, but it will fail because the implementation does not exist yet.

We will constrain the agent to work on pure functions only. No I/O, no side effects. This keeps the generated tests deterministic and easy to reason about. We will also provide a strict output format: the agent must return only the test code, wrapped in a markdown code block, so we can parse it reliably.

• Keep the scope narrow: pure functions only.
• Require the agent to output only code, no explanations.
• Use a JSON schema for the function signature to reduce ambiguity.

Step 2: Set up the agent loop

The core of the agent is a loop that calls the LLM, gets a response, and validates it. If the response contains valid test code, we write it to a file. If not, we feed the error back to the model and ask it to fix the output. This loop is simple but powerful: it gives the model a chance to correct itself.

Here is the main agent loop in TypeScript. It uses the OpenAI SDK and dotenv for configuration.

import OpenAI from 'openai';
import * as fs from 'fs';
import * as path from 'path';
import * as dotenv from 'dotenv';

dotenv.config();

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

interface FunctionSpec {
  name: string;
  signature: string;
  description: string;
  examples: Array<{ input: string; output: string }>;
}

async function generateTests(spec: FunctionSpec, maxRetries = 3): Promise<string> {
  const systemPrompt = `You are a senior test engineer. Write a Jest test file for the given function. The function does not exist yet, so the tests should fail initially. Output only the test code in a single code block, with no extra text.`;

  const userPrompt = `Function name: ${spec.name}\nSignature: ${spec.signature}\nDescription: ${spec.description}\nExamples:\n${spec.examples.map(e => `- Input: ${e.input} -> Output: ${e.output}`).join('\n')}`;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      temperature: 0.2,
      messages: [
        { role: 'system', content: systemPrompt },
        { role: 'user', content: userPrompt }
      ]
    });

    const content = response.choices[0]?.message?.content || '';
    const codeBlockMatch = content.match(/```(?:typescript|ts)?\n([\s\S]*?)```/);
    if (codeBlockMatch) {
      return codeBlockMatch[1];
    }

    // If no code block, ask the model to fix it.
    userPrompt += `\n\nYour previous response did not contain a valid code block. Please output only the test code inside a single code block.`;
  }

  throw new Error('Failed to generate valid test code after multiple attempts.');
}

// Example usage
const spec: FunctionSpec = {
  name: 'add',
  signature: '(a: number, b: number) => number',
  description: 'Adds two numbers and returns the sum.',
  examples: [
    { input: '1, 2', output: '3' },
    { input: '-1, 1', output: '0' }
  ]
};

generateTests(spec).then(testCode => {
  fs.writeFileSync(path.join(__dirname, '__tests__', 'add.test.ts'), testCode);
  console.log('Test file written.');
}).catch(err => console.error(err));

• The loop retries up to three times if the model does not output a code block.
• We use a low temperature (0.2) to keep outputs deterministic.
• The generated tests are saved to a __tests__ directory.

Step 3: Make the agent actually run tests

Generating a test file is only half the job. We want the agent to verify that the test fails for the right reason. That means we need to run the test suite after generating the file. If the test passes, something is wrong (maybe the model wrote a trivial test that always passes). If it fails with a 'module not found' error, that is expected because the implementation is missing.

We will extend the agent to execute the tests using Jest programmatically. We will capture the output and feed it back to the model if something unexpected happens.

import { execSync } from 'child_process';

function runJest(testFile: string): { stdout: string; stderr: string; code: number } {
  try {
    const stdout = execSync(`npx jest ${testFile} --no-coverage`, { encoding: 'utf-8' });
    return { stdout, stderr: '', code: 0 };
  } catch (error: any) {
    return {
      stdout: error.stdout?.toString() || '',
      stderr: error.stderr?.toString() || '',
      code: error.status ?? 1
    };
  }
}

// Inside the loop, after writing the test file:
const result = runJest('__tests__/add.test.ts');
console.log('Jest stdout:', result.stdout);
console.log('Jest stderr:', result.stderr);

// Expected: the test fails because 'add' is not defined.
// If it passes, ask the model to make the test more strict.

• Use execSync for simplicity; in production you might use a non-blocking spawn.
• Check that the test fails with a 'Cannot find name' or similar error, not a syntax error.
• If the test passes, the agent should regenerate with a stronger assertion.

Step 4: Integrate with a code generation agent

The test-first agent is most useful when paired with an implementation agent. You can chain them: first generate the tests, then feed the tests to another agent that writes the implementation. This creates a TDD pipeline where the tests are the contract.

We will build a simple orchestrator that calls the test generator, then calls the implementation generator with the test file as context. The implementation agent must produce code that makes the tests pass.

async function generateImplementation(testCode: string, spec: FunctionSpec): Promise<string> {
  const systemPrompt = `You are an expert TypeScript developer. Write the implementation for the function described below so that it passes the provided Jest tests. Output only the function code in a single code block.`;
  const userPrompt = `Function signature: ${spec.signature}\nDescription: ${spec.description}\nTest file:\n${testCode}`;

  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    temperature: 0.2,
    messages: [
      { role: 'system', content: systemPrompt },
      { role: 'user', content: userPrompt }
    ]
  });

  const content = response.choices[0]?.message?.content || '';
  const codeBlockMatch = content.match(/```(?:typescript|ts)?\n([\s\S]*?)```/);
  if (!codeBlockMatch) throw new Error('No code block in implementation response.');
  return codeBlockMatch[1];
}

// Orchestrate
const testCode = await generateTests(spec);
fs.writeFileSync('__tests__/add.test.ts', testCode);
const implCode = await generateImplementation(testCode, spec);
fs.writeFileSync('add.ts', implCode);
console.log('Implementation written. Run tests to verify.');

• The implementation agent receives the test file as part of the prompt, so it knows the expected behavior.
• After writing the implementation, run `npm test` to confirm all tests pass.
• If tests fail, you can loop back to the implementation agent with the error output.

Step 5: Add safety checks and validation

AI-generated tests can be wrong, too. They might assert incorrect behavior, or they might be too lenient. We need a validation step that checks the tests are meaningful: they should fail for the right reason before implementation, and pass after a correct implementation.

We will implement a validation function that runs the tests twice: once before implementation (expect fail) and once after (expect pass). If the before-run passes, we reject the test as too weak. If the after-run fails, we reject the implementation as incorrect.

function validateTests(testFile: string, implFile: string): boolean {
  // Run tests before implementation (should fail)
  const before = runJest(testFile);
  if (before.code === 0) {
    console.error('Tests passed before implementation. They are too weak.');
    return false;
  }

  // Write a dummy implementation that always throws to ensure failure is due to missing function
  fs.writeFileSync(implFile, 'export function add(a: number, b: number): number { throw new Error("Not implemented"); }');
  const afterDummy = runJest(testFile);
  if (afterDummy.code === 0) {
    console.error('Tests passed with dummy implementation. They are not testing the function.');
    return false;
  }

  return true;
}

• The dummy implementation throws to guarantee failure if the test actually calls the function.
• If the test passes with the dummy, it means the test is not calling the function at all.
• This validation can be extended to check for edge cases in the examples.

What I would do: a recommended setup

For a production setup, I would not run this as a standalone script. I would integrate it into a CI pipeline or a pre-commit hook. The agent can be triggered when a new function signature is added to a file.

Here is a recommended package.json script and a minimal agent configuration that you can copy and adapt.

{
  "scripts": {
    "generate-tests": "ts-node src/generateTests.ts",
    "test": "jest"
  },
  "jest": {
    "testMatch": ["**/__tests__/**/*.test.ts"],
    "transform": {
      "^.+\\.ts$": "ts-jest"
    }
  }
}

• Store function specs in a JSON file that the agent reads.
• Use a model with a larger context window if your functions are complex.
• Always run the validation step before committing generated tests.

Troubleshooting

If the agent produces invalid TypeScript, the Jest run will fail with a syntax error. The retry loop may not catch this because the model might keep outputting the same broken code. Add a check for syntax errors by running `tsc --noEmit` on the generated file.

If the tests pass before implementation, it usually means the model wrote a test that does not actually call the function. For example, it might test a constant. The validation step catches this, but you can also instruct the model to use the function name explicitly in the test.

If the implementation agent cannot make the tests pass, it might be because the tests are too strict or incorrect. In that case, you can have a human review the generated tests and adjust them, or feed the error output back to the implementation agent for another attempt.

• Check the Jest output for 'Cannot find name' errors, which indicate the function is missing.
• Use `npx tsc --noEmit` to catch type errors before running tests.
• If the model repeatedly fails to produce a code block, try a different model or increase the temperature.

FAQ

Q: Can this work with any LLM API?

A: Yes, the loop only depends on chat completions. You can swap the OpenAI client for Anthropic or a local model via Ollama.

Q: What if the function has side effects?

A: The current design is for pure functions only. For side effects, you would need to mock dependencies, which adds complexity.

Q: How do I handle multiple test cases?

A: Provide more examples in the spec. The model will generate tests that cover those examples, but you should also ask for edge cases in the prompt.

Q: Is this ready for production?

A: It is a solid foundation, but you should add more validation, error handling, and human review before using it in a critical codebase.

Next step: run it on your machine

Create a file named spec.json with the function spec for a simple utility, then run the agent script. The agent will generate a failing test file and an implementation that makes it pass. Verify by running `npm test`.

This is the first step toward automating TDD. Once you have the loop working, you can extend it to handle more complex functions, integrate it with your version control, and even let it open pull requests.

echo '{"name":"add","signature":"(a: number, b: number) => number","description":"Adds two numbers","examples":[{"input":"1,2","output":"3"}]}' > spec.json
npx ts-node src/generateTests.ts
npm test