How to Make an AI Agent Stop Breaking Existing Features

Before you start

You added an AI coding agent to your repo. It writes code fast. Then it breaks a feature you shipped months ago, and you only find out in production. This article is a practical guide to keeping that from happening.

You will set up a sandboxed environment for the agent, add guardrails to its tool calls, and wire automated regression tests into its workflow. By the end, you will have a repeatable setup that catches breakage before it merges.

You need a Linux or macOS machine with Docker and Node.js installed. The examples use TypeScript and shell commands. If you use Python or another stack, the principles still apply.

• Docker 20.10 or newer
• Node.js 18 or newer
• A Git repository you can experiment on (use a fork if you are cautious)
• Access to a terminal and a code editor

Step 1: Sandbox the agent with Docker

The first line of defense is to run the agent in a container that has no access to your production systems. This way, even if the agent makes a destructive call, it cannot reach your live database or secrets.

Create a Dockerfile that gives the agent a minimal environment with the tools it needs. The agent will run inside this container, and you will mount only the repository directory.

FROM node:20-slim

# Install git and other tools the agent may need
RUN apt-get update && apt-get install -y git curl jq \
    && rm -rf /var/lib/apt/lists/*

# Create a non-root user
RUN useradd -m agent
USER agent

WORKDIR /workspace

# The repo will be mounted here
COPY --chown=agent:agent . /workspace

# Install dependencies (adjust to your project)
RUN npm ci

CMD ["bash"]

• Do not mount your home directory or SSH keys into the container.
• Use a read-only root filesystem if possible: add --read-only to the run command.
• Keep the container image minimal to reduce attack surface.

Step 2: Restrict tool permissions

Most AI agent frameworks let you define which tools the agent can call. You should only enable the tools that are necessary for coding tasks: file read/write, git, and running tests. Disable network access or shell commands that can hit external services.

Here is an example tool schema for an agent using the OpenAI function-calling style. It only permits file operations and running npm scripts.

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "read_file",
        "description": "Read a file from the repository",
        "parameters": {
          "type": "object",
          "properties": {
            "path": { "type": "string" }
          },
          "required": ["path"]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "write_file",
        "description": "Write a file to the repository",
        "parameters": {
          "type": "object",
          "properties": {
            "path": { "type": "string" },
            "content": { "type": "string" }
          },
          "required": ["path", "content"]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "run_npm_script",
        "description": "Run an npm script defined in package.json",
        "parameters": {
          "type": "object",
          "properties": {
            "script": { "type": "string" }
          },
          "required": ["script"]
        }
      }
    }
  ]
}

• Do not give the agent a generic shell tool unless you absolutely need it.
• If you must allow network calls, proxy them through a firewall that only allows access to your package registry.
• Review the tool list after every agent framework upgrade.

Step 3: Add regression tests to the loop

The most effective way to catch breakage is to run your existing test suite after every agent change. You can automate this with a pre-commit hook that runs tests in the sandbox.

Create a script that the agent must run before it can create a commit. The script runs the tests and exits with a non-zero code if any test fails.

#!/bin/bash
# scripts/agent-test.sh
# Run this script before committing changes.

set -e

# Run the test suite (adjust to your project)
npm test

echo "All tests passed. You can commit."

#!/bin/bash
# .git/hooks/pre-commit
# Make this executable with: chmod +x .git/hooks/pre-commit

# Run the agent test script in the sandbox
docker run --rm \
  -v "$(pwd):/workspace" \
  -w /workspace \
  agent-sandbox \
  ./scripts/agent-test.sh

• Make the hook executable with chmod +x .git/hooks/pre-commit.
• If your tests take too long, run a focused subset first, but always run the full suite in CI.
• Use coverage tools to ensure the tests exercise the code paths the agent is changing.

Step 4: Use CI to enforce the gate

Pre-commit hooks only work if the developer runs them. For a stronger guarantee, set up a CI pipeline that runs the full test suite and a set of integration tests on every pull request. This way, even if the agent bypasses the hook, the merge is blocked.

Here is a GitHub Actions workflow that runs tests in a container similar to your sandbox.

name: CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: node:20-slim
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: npm ci
      - name: Run tests
        run: npm test
      - name: Run type checks
        run: npm run typecheck

• Add a step that runs linting as well, to catch style regressions.
• Require the CI check to pass before merging in your branch protection rules.
• Store CI logs so you can trace which change broke a test.

Step 5: Add guardrails for dangerous operations

Sometimes the agent needs to run commands that can affect the database or external services. For those, you need a guardrail: a wrapper that asks for confirmation or checks a flag before executing.

Create a script that wraps dangerous commands and requires a manual approval token. The agent cannot generate the token on its own.

#!/bin/bash
# scripts/guardrail.sh
# Usage: ./guardrail.sh <command>
# Requires APPROVAL_TOKEN environment variable to match the expected value.

EXPECTED_TOKEN="change-this-to-a-secret"

if [ "$APPROVAL_TOKEN" != "$EXPECTED_TOKEN" ]; then
  echo "Error: This command requires manual approval."
  exit 1
fi

# Execute the command
"$@"

• Set the APPROVAL_TOKEN only in your CI environment, never in the agent sandbox.
• For database migrations, require a human to run them in production, not the agent.
• Log every guarded command to an audit trail.

Step 6: Monitor and observe agent behavior

Even with all these gates, you need visibility into what the agent is doing. Enable logging and tracing for the agent's tool calls. This helps you identify patterns that lead to breakage.

Here is a simple logger you can add to your agent loop to record every tool call and its result.

// agent-logger.ts
import { appendFileSync } from 'fs';

export function logToolCall(toolName: string, input: unknown, output: unknown) {
  const entry = {
    timestamp: new Date().toISOString(),
    toolName,
    input,
    output,
  };
  appendFileSync('agent-audit.log', JSON.stringify(entry) + '\n');
}

// Usage in your agent loop:
// logToolCall('write_file', { path: 'src/index.ts' }, { success: true });

• Store the audit log outside the container, so the agent cannot tamper with it.
• Use a structured format like JSON so you can query it later.
• Set up alerts for unusual patterns, like the agent repeatedly modifying the same file.

Recommended setup: a complete agent guardrail stack

Here is a copy-paste starter that combines everything: a sandboxed Docker container, a restrictive tool schema, a pre-commit test gate, and a CI pipeline. You can adapt it to your stack.

# 1. Build the sandbox image
docker build -t agent-sandbox .

# 2. Run the agent inside the sandbox (example with a simple CLI agent)
docker run --rm \
  -v "$(pwd):/workspace" \
  -w /workspace \
  -e OPENAI_API_KEY="$OPENAI_API_KEY" \
  agent-sandbox \
  npx your-agent-cli --tools schema.json --log audit.log

# 3. Install the pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
docker run --rm -v "$(pwd):/workspace" -w /workspace agent-sandbox ./scripts/agent-test.sh
EOF
chmod +x .git/hooks/pre-commit

# 4. Push to trigger CI (use the workflow from Step 4)

• Replace your-agent-cli with the actual command for your agent framework.
• Make sure the schema.json file is in your repo and references only the allowed tools.
• Test the whole flow by intentionally making a change that breaks a test and verifying the agent cannot commit.

Troubleshooting

If the pre-commit hook does not run, check that the file is executable and that it is in the .git/hooks directory. If the Docker command fails, ensure the image is built and that the volume mount path is correct.

If the agent still finds a way to break things, review the audit log to see what tool calls it made. Look for patterns like writing to unexpected directories or running scripts that are not in your allowlist.

• Pre-commit hook not running: verify the file has exec permission and is named correctly.
• Tests fail in CI but pass locally: ensure your CI uses the same Node version and dependencies.
• Agent creates untracked files: add a .gitignore rule or a cleanup step in the sandbox.
• Agent changes files outside the repo: use a volume mount that only exposes the repo directory.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can I use this setup with a Python-based agent? A: Yes, replace the Node commands with pip and pytest, and adjust the Dockerfile accordingly.
• Q: How do I handle agents that need to make network calls? A: Allow only specific domains through a proxy, and never give the agent your production API keys.
• Q: What if my tests take more than 10 minutes? A: Run a fast subset in the pre-commit hook and the full suite in CI. You can also use test splitting.
• Q: Is it enough to rely on CI alone? A: No, you still want the pre-commit hook for fast feedback. CI is the backstop.
• Q: How do I deal with false positives from tests? A: Investigate before disabling a test. If a test is flaky, fix the flakiness rather than letting the agent bypass it.

Next action

Create the pre-commit hook and the test script in your repo today. Then make a trivial change that breaks a test and watch the hook block the commit. That is the moment you know your agent is under control.

Once that works, add the CI workflow and the audit log. You will have a solid guardrail stack that keeps your features intact.