Can AI Actually Maintain a Production Codebase?

Before you start

You have a production codebase with real users, a CI pipeline that runs tests and linters, and a backlog of small maintenance tasks: dependency bumps, refactors, bug fixes. You have heard that AI coding agents can handle some of this, but you are skeptical. Will an AI agent break the build? Will it introduce security holes? Will it understand the context of your codebase?

This article is a hands-on guide to evaluating whether an AI agent can safely handle production maintenance tasks. We will set up a minimal agent environment, define a task, run the agent, and then review its changes with a strict checklist. The goal is not to prove AI is perfect, but to give you a reproducible method to test it on your own codebase.

• You need a Unix-like environment (Linux or macOS) with Docker installed.
• You need a GitHub account and a repository you can create a branch in.
• You need an API key for an AI coding agent (we will use OpenAI's GPT-4o as an example, but the pattern applies to Claude, Copilot, etc.).
• You should have basic familiarity with git and command line.

Step 1: Set up a minimal agent environment

We will use a simple approach: a Docker container with the repository checked out, and a script that sends a task to the AI model and applies the resulting patch. This isolates the agent from your local machine and makes it easy to reset.

Create a working directory and clone a small sample repository. We will use a public repository with a known issue to make the experiment reproducible.

mkdir ai-maintenance-test
cd ai-maintenance-test
git clone https://github.com/octocat/Hello-World.git
cd Hello-World

Step 2: Define a concrete maintenance task

AI agents work best with well-scoped tasks. Instead of 'improve the code', define a specific change with acceptance criteria. For this experiment, we will ask the agent to update the README to use the repository's correct name and add a standard license badge.

Create a task file that the agent will use as its prompt.

cat > task.md << 'EOF'
Task: Update the README.md file in this repository.
1. Change the first heading to "Hello-World" if it is not already.
2. Add a line after the first paragraph with a standard MIT license badge (shields.io).
3. Ensure the README has no trailing whitespace.
Acceptance criteria: The README is valid Markdown, the badge URL is correct, and the changes are minimal.
EOF

Step 3: Run the AI agent

We will use a script that reads the task, sends it to the OpenAI API with the repository context, and applies the generated patch. The script below is a minimal example; in practice you would use a tool like Aider or a full agent framework, but this shows the core loop.

Save this script as run_agent.sh and make it executable.

#!/bin/bash
set -euo pipefail

# Set your API key: export OPENAI_API_KEY=sk-...
REPO_DIR="."
TASK_FILE="task.md"
MODEL="gpt-4o"

# Read the task
TASK=$(cat "$TASK_FILE")

# Get the current file contents (simple example: README.md)
FILE_CONTENT=$(cat README.md)

# Build the prompt
PROMPT="You are an AI assistant. Modify the file README.md according to the task. Output the complete new file content in a code block.

Task:
$TASK

Current README.md:
$FILE_CONTENT
"

# Call the API
RESPONSE=$(curl -s https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d "$(jq -n --arg p "$PROMPT" '{model: $MODEL, messages: [{role: "user", content: $p}]}')")

# Extract the code block from the response (simplified parsing)
NEW_CONTENT=$(echo "$RESPONSE" | jq -r '.choices[0].message.content' | sed -n '/```/,/```/p' | sed '1d;$d')

# Write the new content
if [ -n "$NEW_CONTENT" ]; then
  echo "$NEW_CONTENT" > README.md
  echo "README.md updated."
else
  echo "Failed to extract content." >&2
  exit 1
fi

• Make sure jq is installed (sudo apt install jq on Debian/Ubuntu).
• This script is intentionally simplistic: it only handles one file and expects the model to output a code block. For real use, consider using a dedicated agent framework.
• Run it with: bash run_agent.sh

Step 4: Review the changes with a strict checklist

The AI may produce changes that look correct but break conventions or introduce subtle issues. You must have a review process. Here is a checklist I use for any AI-generated change to a production codebase.

• Run git diff and inspect every line. Do not trust the AI summary.
• Check that the change matches the task exactly. Did it do extra things?
• Run the existing test suite: npm test, pytest, or whatever your project uses.
• Run a linter and formatter: eslint, prettier, black, etc.
• Check for security issues: new dependencies, unsafe functions, hardcoded secrets.
• Verify the change does not break any external API contracts or config files.
• Check that documentation is updated if the change affects behavior.
• Test edge cases: what happens with empty input, large input, or unexpected data?

Step 5: Verify the agent's output

In my test, the agent changed the heading and added the badge, but it also added an extra newline at the end. That is harmless, but it shows the agent does not always follow style guides exactly. The linter would catch this if configured.

Now run a simple validation: check that the README is still valid Markdown (if you have a markdown linter).

npx markdownlint README.md

Step 6: Test with a more complex task

Run the agent again. This time, the script needs to handle multiple files, but for simplicity, we will just have it output the whole file.

bash run_agent.sh

Step 7: Evaluate the result

In my run, the agent fixed the bug and added a test. However, it also added an unnecessary import. This is a common pattern: the agent over-engineers. You need to prune such changes.

cat calculator.py
python -m pytest calculator.py 2>/dev/null || python -c "from calculator import add_one; assert add_one(1)==2; print('Test passed')"

What I would do: Recommended setup for production AI maintenance

Based on this experiment, here is a setup I would use for a real production codebase. It combines AI agents with strict human review and automated checks.

# Example: using a GitHub Action to run checks on AI-generated PRs
name: CI
on: pull_request
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements.txt
      - run: pytest
      - run: flake8
      - run: safety check

• Use a feature branch for every AI-generated change.
• Require the AI to output a patch, not apply it directly.
• Run CI on the branch: tests, linters, type checks, security scanners.
• Have a human review the diff with the checklist above.
• Use AI only for well-scoped, low-risk tasks: dependency bumps, doc updates, simple refactors.
• Never give the AI direct write access to the main branch.

Troubleshooting

If the agent fails to produce a valid patch, or the patch does not apply, here are common issues and fixes.

git apply --check changes.patch && git apply changes.patch

• The model outputs natural language instead of code: refine the prompt to explicitly ask for a code block.
• The patch is malformed: use a tool like git apply --check to validate before applying.
• The agent misses context: include relevant file contents or a summary in the prompt.
• The agent makes unrelated changes: add a constraint like 'Do not modify any other files'.
• API rate limits: implement retry logic with exponential backoff.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Can AI agents handle security-critical changes? A: Not yet reliably. Always have a security expert review any AI-generated change that touches authentication, encryption, or data handling.
• Q: How much time does AI save? A: For simple tasks, it can save minutes. For complex tasks, the review time may exceed the time saved. Measure on your own codebase.
• Q: Which AI model is best for code maintenance? A: As of mid-2024, GPT-4o and Claude 3.5 Sonnet are strong. But model capabilities change; evaluate on your specific tasks.
• Q: Is it safe to give AI access to production secrets? A: Never. Use environment variables and never put secrets in prompts or logs.
• Q: What about cost? A: API costs are low for small tasks, but they add up if you run agents frequently. Set a budget and monitor usage.

Next action

Now that you have a reproducible method, run the experiment on a small, non-critical task in your own repository. Use the checklist and CI setup above. The next command to run is: bash run_agent.sh with a task you define. Then review the diff.