I Let an AI Agent Build a Feature From a GitHub Issue

The Problem: An Issue Sits in the Backlog for Weeks

Every team has that issue: 'Add pagination to the user list endpoint.' It is clear, well-scoped, and exactly the kind of task that interrupts your flow. You could spend an hour implementing it, but you are in the middle of a refactor. The issue sits. The backlog grows.

AI coding agents promise to pick up these issues and implement them. But can they really? I decided to test one end-to-end. I gave a coding agent a GitHub issue from a real repository and let it build the feature, open a pull request, and run the tests. Here is exactly what happened, including the commands I ran and the configuration I used.

This is not a review of a specific tool. It is a reproducible walkthrough. You can adapt it to your own stack and agent of choice. The goal is to show you the practical steps, the gotchas, and the verification process.

Before You Start: What You Need

To follow along, you need a GitHub repository with a well-defined issue. The agent will need access to that repository. You also need a local environment where the agent can run commands and edit files.

I used the open-source AI agent called OpenHands (formerly OpenDevin) because it is free and runs locally. You can use any agent that can edit files and run shell commands, such as Codex CLI, Cursor, or GitHub Copilot Workspace. The principles are the same.

Here is what I had installed:

Docker (for running OpenHands), Git, Node.js and npm (for the test project), and a GitHub personal access token with repo scope.

git clone https://github.com/your-org/your-repo.git
cd your-repo
git checkout -b feature/pagination

• A GitHub repository with a clear, well-scoped issue
• A local clone of that repository
• Docker installed and running
• A GitHub personal access token with repo scope
• A terminal and basic familiarity with Git

Step 1: Set Up the AI Agent

After the container starts, OpenHands provides a web interface at http://localhost:3000. From there you can create a new session and point it to your workspace.

docker run -it --rm \
  -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.24-nikolaik \
  -e LOG_ALL_EVENTS=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ~/.openhands:/.openhands \
  -v $(pwd):/workspace \
  -e LLM_API_KEY="sk-ant-..." \
  -e LLM_MODEL="anthropic/claude-3-5-sonnet-20241022" \
  -p 3000:3000 \
  --add-host host.docker.internal:host-gateway \
  ghcr.io/all-hands-ai/openhands:0.24

Step 2: Give the Agent the Issue

In the OpenHands interface, I created a new conversation. I pasted the GitHub issue text directly into the prompt. The issue was clear: 'Add pagination to the GET /users endpoint. Use query parameters page and limit, default to page=1 and limit=20. Return a JSON object with users, page, limit, and total.'

I also told the agent to work in the /workspace directory and to run the existing test suite after making changes. This is important because the agent needs to know the context and the expected outcome.

Here is the exact prompt I used:

Implement the feature described in this GitHub issue:

Title: Add pagination to GET /users endpoint

Description: Add pagination support to the GET /users endpoint. Accept query parameters page and limit, default to page=1 and limit=20. Return a JSON object with the following structure: { users: [...], page: 1, limit: 20, total: 100 }.

Work in the /workspace directory. After making changes, run the test suite (npm test) and ensure all tests pass. Do not modify unrelated files.

Step 3: Watch the Agent Work

After editing, the agent ran the test suite. In the interface, I saw the command it executed and the output. The tests passed, but the agent also added new tests for pagination. That was a pleasant surprise.

// src/routes/users.js (before)
router.get('/', (req, res) => {
  const users = db.getUsers();
  res.json(users);
});

// src/routes/users.js (after)
router.get('/', (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const limit = parseInt(req.query.limit) || 20;
  const startIndex = (page - 1) * limit;
  const endIndex = page * limit;
  const allUsers = db.getUsers();
  const users = allUsers.slice(startIndex, endIndex);
  res.json({
    users,
    page,
    limit,
    total: allUsers.length
  });
});

Step 4: Review the Changes and Commit

Once the agent finished, I reviewed the changes in the workspace. I used git diff to see exactly what was modified. The changes were minimal and focused, which is what you want from a coding agent.

After reviewing, I committed the changes and pushed the branch to GitHub. Then I opened a pull request from the command line using the GitHub CLI. This is a step you should always do manually, because the agent might not have the right credentials or context.

Here are the commands I ran:

git diff
git add src/routes/users.js test/users.test.js
git commit -m "Add pagination to GET /users endpoint"
git push origin feature/pagination
gh pr create --title "Add pagination to GET /users" --body "Closes #123"

Step 5: Verify It Worked

The response was exactly as expected:

{
  "users": [
    { "id": 6, "name": "Alice" },
    { "id": 7, "name": "Bob" },
    { "id": 8, "name": "Charlie" },
    { "id": 9, "name": "Diana" },
    { "id": 10, "name": "Eve" }
  ],
  "page": 2,
  "limit": 5,
  "total": 20
}

Troubleshooting: Common Issues

The first run did not go perfectly. The agent tried to use a package that was not installed, and it ran into a missing dependency. It did not install it automatically because I had not given it permission to run arbitrary commands without asking.

In OpenHands, you can configure the agent to ask before running any command. That is a good safety measure. I allowed it to run commands after a prompt, and it installed the missing package with npm install, then continued.

Another issue was that the agent initially modified the test file in a way that broke an existing test. It caught this when it ran the full suite and reverted the change. This shows the importance of having a test suite for the agent to run.

• Always have a test suite the agent can run to verify its changes
• Configure the agent to ask before running destructive commands
• Review the diff before committing; do not blindly trust the agent
• Set a time limit for the agent to prevent it from going down a rabbit hole
• Provide a clear issue with acceptance criteria to get better results

What I Would Do Differently: Recommended Setup

The CONFIRMATION_MODE=require environment variable forces the agent to ask for confirmation before running commands, which adds a safety layer.

docker run -it --rm \
  -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.24-nikolaik \
  -e LOG_ALL_EVENTS=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ~/.openhands:/.openhands \
  -v $(pwd):/workspace \
  -e LLM_API_KEY="your-api-key" \
  -e LLM_MODEL="anthropic/claude-3-5-sonnet-20241022" \
  -e CONFIRMATION_MODE=require \
  -p 3000:3000 \
  --add-host host.docker.internal:host-gateway \
  ghcr.io/all-hands-ai/openhands:0.24

FAQ

Answers to the questions that come up most often on this topic.

• Can AI agents handle complex issues? AI agents work best on well-scoped issues with clear acceptance criteria. Complex architectural changes still require human guidance.
• Do I need to give the agent write access to my repository? No. Let the agent work in a local branch and then review the changes before pushing.
• What if the agent introduces bugs? Always run the test suite and manually test the changes. The agent is a tool, not a replacement for code review.
• How much does it cost? If you use an API-based model, you pay for tokens. A small feature might cost a few cents. Running a local model is free but slower.
• Is this production-ready? It can be, but you should treat the agent's output as a first draft. Review, test, and refine before merging.

Try It on code.live

If you want to explore AI-generated code or compare outputs, try the JSON Diff tool to see changes side by side, or use the Commit Message Generator to create a clear commit message for your AI-generated changes.