AI Code Review: What Should You Automate and What Should You Never Automate?

The review backlog is real

Your pull request has been sitting for two days. The author pings you, the build is green, but you have not found time to read through the diff. Meanwhile, the same typo in a variable name has slipped into three different files this month, and a security lint rule is failing on a branch that no one has touched.

AI code review tools promise to clear that backlog. Some teams adopt them and cut review time in half. Others burn hours configuring tools that scream about style while missing real bugs. The difference is not the model. It is knowing what to automate and what to keep human.

This article walks through a concrete split: which review tasks AI handles reliably, which it does not, and how to set up a practical pipeline using open-source tools. You will leave with a working configuration you can adapt to your own repository.

What AI review is actually good at

AI review tools excel at pattern matching across large diffs. They can check for common mistakes, enforce style rules, and spot potential security issues faster than a human scanning line by line. They are also tireless: they run on every commit, not just when a reviewer is available.

The key is to use them for tasks that are deterministic or have clear right answers. Formatting, unused variables, missing error handling, and obvious security anti-patterns all fall into this category. For these, AI can act as a fast, consistent first pass.

• Catching syntax errors and typos that break the build
• Enforcing style guide rules like import order and indentation
• Detecting common security issues like SQL injection or hardcoded secrets
• Flagging unused imports, variables, and dead code
• Checking for missing error handling in obvious places

What AI should never do alone

AI cannot understand the intent behind a change. It does not know the business context, the tradeoffs the team discussed, or the reason a workaround exists. Approving a pull request based solely on AI feedback is dangerous.

Design decisions, architectural changes, and anything that affects the public API need human judgment. AI might suggest a refactor that looks clean but breaks a subtle dependency. It might also miss a bug because the logic is correct but the requirement changed.

The worst use of AI review is as a replacement for human review. Use it as a tool to make the human review faster and more focused, not to skip it.

• Never let AI approve a pull request without human sign-off
• Do not rely on AI for design or architectural decisions
• Do not use AI to review changes that touch security-critical code without a human expert
• Do not assume AI catches all bugs; it is a filter, not a guarantee

Step 1: Set up a baseline with linters and formatters

Before adding AI to the mix, make sure your repository has a solid baseline. Linters and formatters catch the most common issues deterministically. AI review should build on top of these, not replace them.

For JavaScript or TypeScript, ESLint and Prettier are the standard choices. For Python, Ruff and Black are fast and reliable. The goal is to have a consistent style enforced automatically so that AI review can focus on higher-level issues.

npm install --save-dev eslint prettier
npx eslint --init
npx prettier --write .

pip install ruff black
ruff check .
black .

Step 2: Add a lightweight AI review bot

Once your linters are in place, you can add an AI review bot that runs on every pull request. A popular open-source option is CodeRabbit, which integrates with GitHub and provides inline comments. Another is the open-source project 'ai-review' that you can run yourself.

The bot should be configured to focus on issues that linters miss: logic errors, potential bugs, and security concerns. It should not duplicate what ESLint or Ruff already catch.

npx ai-review --provider openai --model gpt-4o --token $GITHUB_TOKEN

• Use a separate GitHub account for the bot so comments are clearly attributed
• Start with a review-only mode to avoid blocking merges
• Set a max diff size to avoid overwhelming the model with huge PRs

Step 3: Configure the bot to focus on high-value checks

A generic AI review will produce a lot of noise. You need to configure it to focus on what matters. Most tools let you write custom instructions for the model.

Here is an example configuration for CodeRabbit that instructs it to ignore style and focus on logic and security.

reviews:
  auto_review:
    enabled: true
    drafts: false
    base_branches:
      - main
    path_instructions: |
      Focus on logic errors, race conditions, and security vulnerabilities.
      Do not comment on style or formatting.
      If you find a potential bug, explain why it is a bug and suggest a fix.
      If the code is correct, do not comment.

Step 4: Run a manual review with AI assistance

Even with a bot, you should do a manual pass. Use an AI assistant to help you understand the diff faster, but make the final call yourself.

A practical workflow: open the pull request, ask an AI chatbot to summarize the changes, then review the summary and the diff. This is faster than reading every line from scratch, but you still have the final say.

git diff main...HEAD | ai-review --summarize

Verify it worked

After setting up the AI review bot, create a test pull request with a known bug. For example, a missing null check or an SQL injection vulnerability. The bot should flag it.

If the bot does not catch it, adjust the configuration or the model. If it produces too many false positives, tighten the instructions.

• Create a PR with a deliberate bug to test the bot
• Check that the bot comments on the right lines
• Ensure the bot does not block the merge if it is in review-only mode

What I would do: a recommended setup

For a typical team, I recommend a layered approach. First, enforce linters and formatters in CI. Second, add an AI review bot that runs on every PR and comments on potential issues. Third, require a human review before merging, but use the AI summary to speed up the human review.

Here is a complete GitHub Actions workflow that runs ESLint, Prettier, and an AI review bot on every pull request.

name: Code Review
on: pull_request
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx eslint .
  ai-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: AI Review
        run: npx ai-review --provider openai --model gpt-4o
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Troubleshooting

If the AI review bot is not commenting, check the permissions. The bot needs read access to pull requests and write access to post comments. Also check that the model API key is valid and has quota.

If the bot is too noisy, increase the max tokens or add more specific instructions. If it misses issues, try a different model or provide more context in the prompt.

• Verify the GitHub token has the 'pull_requests: write' permission
• Check the model API logs for errors
• Start with a small test PR before enabling on all PRs

FAQ

Answers to the questions that come up most often on this topic.

• Can AI review replace human code review? No, AI cannot understand business context and design tradeoffs. Use it to assist, not replace.
• What is the best model for code review? GPT-4o and Claude 3.5 Sonnet are strong choices, but the best model depends on your codebase. Experiment with a few.
• How do I reduce false positives? Give the model clear instructions on what to focus on and ignore style. Also, you can set a confidence threshold.
• Is AI review secure? Your code is sent to the model provider. If you have strict data policies, consider self-hosting a model like CodeLlama.

Next action

Start by setting up ESLint and Prettier in your repository if you have not already. Then add the AI review bot to a single pull request and see how it performs. Adjust the configuration until it adds value without noise.

The goal is to make your code review faster and more reliable, not to add another tool that demands attention. When the bot catches a real bug that a human missed, you will know the setup is working.