How I Use AI to Review Every Pull Request

Before you start

You have probably felt the tension: you want to merge quickly, but you also know a second pair of eyes would catch that off-by-one error or the missing null check. I tried several approaches to automate code review, and what works is a pragmatic combination of a local CLI tool and a CI bot that comments on every pull request.

This article walks through my exact setup. You will need a GitHub repository (or GitLab, with minor tweaks), Node.js 18 or later, and an OpenAI API key or an Anthropic API key. I will show you how to run AI review locally, then how to wire it into GitHub Actions so every pull request gets an automated review comment.

I assume you are comfortable with the command line and basic YAML. No prior AI experience is needed.

node --version
npm --version

• A GitHub repository with a main branch and pull requests enabled.
• Node.js 18+ installed locally (for the CLI) and available in CI.
• An API key from OpenAI or Anthropic. Store it as a secret in your repository.
• A codebase you want to review. I use a small Node.js API as an example, but the setup works for any language.

Step 1: Install the AI review CLI

I use a tool called pr-review-agent, which is an open-source CLI that wraps a language model and posts review comments to your pull request. It is not the only option, but it is simple and scriptable.

Install it globally with npm, or use npx to run it without installing. I prefer a local install in your project so you can pin the version.

npm install -g pr-review-agent
# or, for a project-local install:
npm install --save-dev pr-review-agent

Step 2: Configure the review rules

The CLI reads a configuration file named .pr-review.yml in your repository root. This file defines which checks to run, what language model to use, and how strict the review should be.

Here is a configuration that works for a typical JavaScript or TypeScript project. It focuses on security, correctness, and style, and it ignores generated files.

review:
  model: "gpt-4o-mini"          # or "claude-3-5-sonnet-20241022"
  temperature: 0.2
  max_tokens: 2000
  rules:
    - "Check for security vulnerabilities like SQL injection, XSS, or insecure deserialization."
    - "Identify off-by-one errors, null pointer dereferences, and resource leaks."
    - "Flag code style issues that violate the project's ESLint or Prettier config."
  ignore_files:
    - "package-lock.json"
    - "dist/**"
    - "build/**"
  comment_on_unchanged: false
  max_comments: 10

• Set temperature low (around 0.2) to keep the model focused and deterministic.
• Limit max_comments to avoid overwhelming the author with nitpicks.
• Use ignore_files to skip generated or vendored code, which reduces noise.

Step 3: Run the review locally

Before wiring into CI, test the CLI on an open pull request. The command takes the pull request number and your API key as an environment variable.

The first run will analyze the diff, send it to the model, and post comments directly on the pull request. If you want a dry run without posting, use the --dry-run flag.

export OPENAI_API_KEY=sk-your-key
npx pr-review-agent --pr 42 --dry-run
# Review the output, then post for real:
npx pr-review-agent --pr 42

• Use --dry-run first to see what the AI would say without spamming the PR.
• If you use Anthropic, set ANTHROPIC_API_KEY instead and change the model in the config.

Step 4: Wire it into GitHub Actions

To review every pull request automatically, add a GitHub Actions workflow file. This workflow triggers on pull_request events, installs the CLI, and runs it with the API key stored as a repository secret.

Create the file .github/workflows/ai-review.yml in your repository.

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]
permissions:
  pull-requests: write
  contents: read
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx pr-review-agent --pr ${{ github.event.pull_request.number }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

• The permissions block is required so the bot can post comments.
• The workflow triggers on opened and synchronize, which covers new PRs and updates.
• Store your API key in GitHub Secrets as OPENAI_API_KEY.

Step 5: Verify it worked

Open a new pull request or push a commit to an existing one. Within a minute, you should see a comment from your GitHub App or bot account with the AI review.

Here is a sample output you might see on a simple JavaScript change.

AI Review (pr-review-agent)

Potential issue: In `src/index.js:15`, you are using `==` instead of `===`. This can cause unexpected type coercion. Consider using strict equality.

Security: The `userInput` is directly concatenated into a SQL query. Use parameterized queries to prevent SQL injection.

Style: The function name `getdata` should be `getData` to follow camelCase convention.

• If you do not see a comment, check the Actions tab for errors.
• The bot will only comment on new changes, not on the whole file, unless you set comment_on_unchanged to true.

What I would do (recommended setup)

Based on my experience, here is the setup I recommend for a team that wants to adopt AI review without losing the human touch.

Start with a conservative config, run it in dry-run mode for a week, and then enable posting. You can also add a label or a status check so that the AI review is required before merge, but I advise keeping it informational at first.

mkdir -p .github/workflows
cat > .pr-review.yml <<'EOF'
review:
  model: "gpt-4o-mini"
  temperature: 0.2
  max_tokens: 2000
  rules:
    - "Check for security vulnerabilities like SQL injection, XSS, or insecure deserialization."
    - "Identify off-by-one errors, null pointer dereferences, and resource leaks."
    - "Flag code style issues that violate the project's ESLint or Prettier config."
  ignore_files:
    - "package-lock.json"
    - "dist/**"
    - "build/**"
  comment_on_unchanged: false
  max_comments: 10
EOF
cat > .github/workflows/ai-review.yml <<'EOF'
name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]
permissions:
  pull-requests: write
  contents: read
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx pr-review-agent --pr ${{ github.event.pull_request.number }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
EOF

• Run the setup script in your repo root.
• Add your OPENAI_API_KEY secret in GitHub settings.
• Create a test PR and watch the review appear.

Troubleshooting

Here are the common issues I hit and how to fix them.

If the CLI fails with a 401 error, your API key is wrong or not set. If it times out, the diff might be too large for the model's context window; split the review into smaller chunks or increase max_tokens.

• 401 Unauthorized: Check that OPENAI_API_KEY is set and valid.
• Rate limit errors: Add a sleep between reviews or use a model with a higher rate limit.
• Comments not posting: Verify the GitHub token permissions in the workflow.
• Large diffs: The CLI truncates the diff by default; adjust the max_diff_lines setting in the config.
• Model not found: Ensure you are using a valid model ID for your provider.

FAQ

Answers to the questions that come up most often on this topic.

• Q: Will AI replace human code review? A: No, it augments it. AI catches common issues, but humans still catch design problems and nuanced trade-offs.
• Q: How much does it cost? A: For a typical PR, the cost is a fraction of a cent with gpt-4o-mini. You can monitor usage in your OpenAI dashboard.
• Q: Can I use it with GitLab? A: Yes, pr-review-agent supports GitLab with a different configuration. Check the project README for details.
• Q: How do I prevent the AI from commenting on every line? A: Set max_comments to a low number and comment_on_unchanged to false.
• Q: Can I customize the rules? A: Yes, edit the rules list in .pr-review.yml to match your project's needs.