AM2RICAN5.DEV · FIELD NOTES

am2rican5
Back to blog
AICode ReviewDeveloper ToolsSoftware Engineering

How to Review AI-Generated Pull Requests: A Risk-First Checklist

Agents now start over 40% of GitHub pull requests, and review is the bottleneck. A risk-tiered process, an evidence-first checklist, and a PR template to adopt.

The Short Answer

You can't review AI-generated pull requests by reading faster. Agents produce diffs faster than any team can read them, and the diffs look clean enough that skimming feels safe. What works is changing what a human reviews and in what order:

  1. Tier the PR by risk, set by the paths it touches, not by who (or what) wrote it.
  2. Read the evidence before the code: what ran, what it returned, what the checks would miss.
  3. Review the decisions the agent made, not only the lines it changed.
  4. Spend your reading time where bugs hide in AI code: tests, contracts, boundaries, dependencies.
  5. Let AI review filter, never approve. A named human owns every merge.

There's a copyable checklist at the end.

Why "Just Read the Diff" Stopped Working

Two data points from the last few weeks frame the problem.

GitHub's Octoverse 2026 report says that by September 2026, more than 40% of pull requests on GitHub started with an agent's first draft, up from 17% in March: about 110 million in September alone. Agent-started PRs more often touch multiple files (about 87% versus 79%), and by September 90% of them merged within two weeks, against about 86% for PRs without detected agent involvement. (GitHub sells Copilot, so read these as a vendor's numbers.)

On the other side, Fiona Chen and James Stratton's Harvard working paper, Artificial Intelligence in the Firm: Bottlenecks in Software Production, uses engineering-analytics data from hundreds of firms. In the November 2026 version, adopting AI agents raises code output substantially, but the time from review submission to approval rises 56.1%, code waits in the backlog longer, and comments per change go up 38.6%. Their conclusion is blunt: "Code review becomes the bottleneck." Most firms in the sample had already adopted AI code review tools, and humans still wrote the overwhelming majority of reviews. (Earlier coverage of the paper, such as Ars Technica's, reported a 49% figure from a previous version.)

Drafting got cheap. The queue moved to the reviewer.

That's the backdrop for the argument that ran across developer X last week. Kent C. Dodds posted, in full, "Don't read the code". Matt Pocock argued that readability now matters mostly as verifiability: agents "read" code by running and probing it, so a module with strong tests is easy for them regardless of how it looks. Jack Ellis pushed back that humans and AI should both review, because "the human is responsible for the work that hits production whilst the AI cannot be held responsible."

And the post that set the tone for the week, Addy Osmani's Critical thinking when agents do the work, describes a failure mode most reviewers of agent PRs will recognize:

Say an agent gives you a 600-line PR. You roughly look at the structure. You don't read every line. The variable names look reasonable and the tests pass, so you approve it. But somewhere in the middle there's a subtle ordering change in a transaction boundary, or a flipped default on an edge case you never thought about.

These positions are less opposed than they look. Nobody serious wants line-by-line reading of every agent diff, or merges nobody is accountable for. The open question is where a limited amount of human attention goes, and how you make sure it goes there on purpose rather than by habit.

The Failure Mode: Review Surrender

Osmani describes a spectrum: cognitive offloading (you delegate the task but keep ownership of the result), deferral (the tool's output becomes your new starting point; it stopped being a hypothesis), and cognitive surrender (the output quietly becomes yours and you feel no need to check it). He borrows the last term from Wharton researchers Steven Shaw and Gideon Nave.

Applied to code review, surrender rarely looks like negligence. It looks like a reasonable day:

  • You read the agent's PR summary first, and it frames everything you read after it.
  • "Tests pass" becomes the review, even though the agent wrote the tests in the same session as the code.
  • A 600-line PR gets an approval with no comments, because nothing looked wrong.
  • Someone asks why a line exists, and the approver can't say, even with the agent's help.

AI-generated code has a particular shape of risk. It's rarely syntactically broken. The bugs that get through are plausible: a default that changed, an error path that now swallows exceptions, a requirement the agent quietly reinterpreted when two of them conflicted, a test that encodes the same misunderstanding as the implementation and therefore passes. Osmani's point is that polish no longer signals quality. Everything comes back well-formatted and confident.

So the process below is built to make surrender hard to do by accident.

Step 1: Tier the PR by Risk, Not by Author

Start by deciding how much human review a change needs before anyone opens the diff. The Claude Code team's write-up, How the Claude Code team does code review, uses three tiers, and the important design choice is in one sentence: "The change sets the tier, not the author."

A starting table you can adapt:

TierTypical changesHuman review
LowDocs, copy, tests-only additions, internal tools, additive changes behind a flag that's offAutomated checks + AI review green; author self-approves or one quick human pass
MediumOrdinary product changes, refactors with behavior preserved, new endpoints behind authOne reviewer who knows the area, using the reading order below
HighAuth, permissions, billing, data migrations, CI/CD and release workflows, infra, cryptography, anything touching secrets or public API contractsTwo humans, at least one a code owner, plus a rollback plan in the PR

Three rules make the tiers hold up:

  • Paths, not judgment calls, decide the high tier. Put the always-human paths in CODEOWNERS and require code-owner review in branch protection, so the policy doesn't depend on someone remembering it.
  • A bot's approval never fills a human slot. AI review can clear a low-tier change if your policy allows it, but it never counts as one of the required human approvals.
  • Approvals expire on new commits. Turn on "dismiss stale pull request approvals when new commits are pushed." An agent's follow-up push after approval is exactly the change nobody looked at.
# .github/CODEOWNERS: paths that always need a named human
/.github/workflows/      @your-org/platform
/infra/                  @your-org/platform
/db/migrations/          @your-org/data-owners
/src/auth/               @your-org/security
/src/billing/            @your-org/payments @your-org/security
/src/api/public/         @your-org/api-owners

Start strict and loosen the low tier with evidence (your revert rate, below), not with optimism.

Step 2: Read the Evidence Before the Code

Before reading a single line, read what the agent actually ran and what came back. Osmani's advice for agent work generally applies directly to PRs: "before you look at the summary, first review the code snippet of the command executed and its output." A green check proves one property. Ask which one.

Make this cheap by putting it in the PR template. A version tuned for agent-authored PRs:

## Intent (written by the human who asked for this change)
What problem does this solve, for whom, and what is explicitly out of scope?

## Decisions the agent made
- Defaults chosen:
- Requirements interpreted or traded off:
- Things deliberately not handled:

## Verification evidence
- Commands run and their output (paste or link the CI step):
- Screenshot / trace / log for user-visible or runtime behavior:
- Which test fails if the main change is reverted?

## What the checks would NOT catch
(Concurrency, data volume, permissions, migrations on real data, third-party behavior, ...)

## Risk tier: low / medium / high
## How to revert

Two fields do most of the work.

"Intent," written by a human. The person who prompted the agent owns the change. If they can't state the intent and the out-of-scope list in their own words, the PR isn't ready for anyone else's time. Osmani calls the gap intent debt: goals and constraints that never got written down, which an agent will fill in for you with something that looks finished.

"What the checks would not catch." This comes straight from the Claude Code team's rule of thumb: "If you can't say what the checks would miss, ask for a real review rather than a quick approval." An author who writes "nothing" in that field on a medium or high-tier change has told you where to look.

Step 3: Review the Decisions, Not Only the Diff

Over a long session an agent makes dozens of small decisions that never show up as an obvious diff hunk: which default to use, what counts as out of scope, which of two conflicting requirements wins, whether to retry or fail. A summary that says "implemented X, all tests pass" hides all of them.

So ask for them explicitly. If your team uses Claude Code, Codex, or Cursor, add a closing step to the task prompt or a project command such as:

Before opening the PR, list every decision you made that a reviewer might
disagree with: defaults, error handling, scope you excluded, requirements you
reinterpreted, and anything you were unsure about. For each, give the file and
line. Then list what your tests do not cover.

Then review that list like a design review. The questions that catch the most:

  • Does the change solve the stated problem, or a nearby easier one?
  • Which default changed, and who depends on the old one?
  • What happens on the error path: retried, surfaced, logged, or swallowed?
  • Where did the agent resolve a conflict between requirements, and would the product owner agree?
  • Is anything now done twice: a helper re-implemented, a validation duplicated in a second place?

If you can't answer "would I defend this in front of the person who has to maintain it?" (Osmani's bar for answerability), you haven't finished the review, whatever the checks say.

Step 4: Read the Diff Where Bugs Hide (a 15-Minute Pass)

Now open the diff, in this order. The order matters because the first thing you read anchors everything after it, and in agent PRs the riskiest changes are rarely in the file the summary leads with. Google's long-standing guide to what to look for in a code review (design, functionality, complexity, tests) still applies; what changes with agent PRs is where you start.

  1. Tests first. Were tests deleted, skipped, loosened, or snapshot-updated? Does any assertion now accept more than before? Did the agent change the test and the code to agree with each other?
  2. Contracts and data. Schemas, migrations, API request/response shapes, config defaults, feature flags, environment variables.
  3. Boundaries. Transactions and their ordering, locking and concurrency, retries and idempotency, auth and permission checks, input validation, timeouts.
  4. Dependencies. New packages, major-version bumps, lockfile churn that doesn't match the manifest change.
  5. Duplication and dead code. New helpers that already exist elsewhere, unused exports, commented-out blocks.
  6. Everything else, at skim speed.

A few commands make steps 1 and 4 fast:

BASE=origin/main

# Shape of the change: where are the lines?
git diff --stat "$BASE"...HEAD

# Tests added, modified, or deleted
git diff --name-status "$BASE"...HEAD -- '*test*' '*spec*' '__snapshots__'

# Added lines that silence checks or skip tests
git diff "$BASE"...HEAD | grep -nE '^\+.*(\.skip\(|\.only\(|xit\(|@ts-ignore|@ts-expect-error|eslint-disable|# type: ignore|# noqa)'

# Dependency changes
git diff "$BASE"...HEAD -- package.json requirements*.txt pyproject.toml go.mod Cargo.toml

And one check that catches the "test written with the same mistake" problem better than reading does: revert the implementation and run the new tests. If they still pass, they don't test the change.

# Put the old implementation back, keep the new tests, run them
git checkout "$BASE" -- src/orders/refund.ts
pnpm test src/orders/refund.test.ts   # should FAIL
git checkout HEAD -- src/orders/refund.ts

You can ask the agent to do this and paste the failing output into the PR. That's a better use of its speed than another summary.

Step 5: Use AI Review as a Filter, Not an Approver

AI review earns its place by catching routine mistakes before a human spends attention on them. The Claude Code team reports that once they ran AI review on every PR, the share of internal PRs getting substantive review comments went from 16% to 54%. Jack Ellis's position from the X debate is the right default policy: run human and AI review, and don't make either one the only gate.

What separates a useful AI reviewer from a noisy one is mostly configuration:

  • Give it a short REVIEW.md, written by code owners, that says what failure looks like in this repo. The Claude Code team's advice: keep it short, cap the nits, list paths to skip, and ask for file-and-line evidence instead of guesses.
  • Require every finding to be resolved: fixed, or closed with a one-line reason. That's how false positives get cleared without being ignored.
  • Cap the rounds. Agent authors and agent reviewers can keep each other busy indefinitely. The same write-up describes one PR that went through more than fifty rounds overnight, and found that about 59% of what the reviewer kept flagging was code that had been there since its first pass. Their fix: after three rounds, step back and reconsider the approach; after the first few passes, re-review only the new diff.

A starting REVIEW.md:

# REVIEW.md

## What failure looks like here
- Money: any change to refund, invoice, or currency rounding logic.
- Data: migrations that are not reversible or that lock large tables.
- Access: endpoints missing the `requireRole` check.

## Always flag
- Deleted, skipped, or loosened tests.
- New dependencies, and changes to CI workflow files.
- Error handling that catches and drops exceptions.

## Do not flag
- Formatting and import order (Biome enforces these).
- Anything under `src/generated/`.

## Format
Start with a count of findings. Each finding: severity, file:line, evidence.
If there is nothing to fix, say so in one line.

When the same finding keeps coming back, move it out of review and into the harness: a lint rule, a CI check, a hook, or a line in your agent's instruction file. (If you're deciding what belongs in that file, the evidence is in Does AGENTS.md Help Coding Agents?.)

Step 6: Make Review Capacity a Team Policy

Osmani names the gap between what your agents generate and what you'd put your name on the orchestration tax, and puts it this way: "When you're producing more than you can stand behind, you're no longer reviewing. You're hoping."

Individual discipline won't fix a queue. Team policy can:

  • Cap open agent PRs per person. Pick a number, start low, and raise it only if review time and reverts stay flat. Whatever the number, generation stops when review can't keep up.
  • Set a size budget, excluding generated files and lockfiles, and ask agents to split work that exceeds it. GitHub's stacked pull requests (in preview since July) and plain branch-per-step workflows both make this easier.
  • Prefer reversible changes. Feature flags, staged rollouts, and one-step reverts are part of review capacity: the Claude Code team's stated approach is to move most reading to machines and "invest heavily in making it fast and easy to undo mistakes."
  • Schedule review as real work. Checking the machine's output isn't free; put it in the plan, or it quietly won't happen.

Step 7: Keep Your Reviewers Able to Review

There's a slower risk underneath all of this. Anthropic's own research, cited by Osmani, calls it the paradox of supervision: "supervising Claude requires the very coding skills that may atrophy from AI overuse."

You don't need anyone to hold the whole system in their head. The Claude Code team says nobody there does; they ask Claude for an explainer of the slice they're about to change. But you do need reviewers who can rebuild understanding when it matters. Practical habits:

  • Write a hypothesis before reading. Before opening a medium or high-tier diff, write one line: how would you have done it? Differences are where to look.
  • Quiz before merge on high-tier changes: have the agent write a short report on the change with questions at the end, and merge only after you can answer them.
  • Pair juniors with seniors on agent-PR review. Review used to be how people learned a codebase. With agents writing first drafts, review is where most of that learning now happens, but only if someone teaches what to look for.
  • Ask for explainers, not just approvals. "Explain this module to someone about to change it" is a cheap request and a good test of whether the change made sense.

Measure Whether It's Working

The Claude Code team's framing is the useful one: optimize for the minimum human review time that doesn't raise reverts or incidents. Track both sides:

MetricWhy
Time to first human review, and submission-to-merge timeThe bottleneck the Chen and Stratton paper measured
Human approvals and comments per PR, by tierWhere attention actually goes
Review rounds per PRNon-converging AI review shows up here first
Changes-requested shareRising share means drafts are getting worse or intent is unclear
Revert rate and incidents per merged PRThe guardrail: if these rise, you didn't save time, you moved it

Most of this is available from your Git host's API. For example, a rough monthly pull with the GitHub CLI:

gh pr list --state merged --search "merged:>=2026-10-01" --limit 500 \
  --json number,createdAt,mergedAt,additions,deletions,reviews,labels \
  > merged-prs.json

Label agent-authored PRs (many tools do this automatically) so you can compare them with the rest. Look at the trend after you change a policy, not at a single month.

The Checklist

Copy this into your PR template or your team's review guide. On GitHub, write each item as - [ ] item to get task-list checkboxes.

Before reading code

  • Risk tier set by paths touched; high-tier paths are in CODEOWNERS
  • Intent and out-of-scope written by the human who requested the change
  • Verification evidence attached: commands, output, screenshot or trace
  • "What the checks would not catch" is filled in, and not with "nothing"
  • AI review findings resolved (fixed, or closed with a reason)

Reading

  • Agent's decisions and assumptions listed and reviewed
  • Tests read first: none deleted, skipped, or loosened without a reason
  • New tests fail when the implementation is reverted
  • Contracts, migrations, config defaults, and flags checked
  • Transactions, concurrency, retries, auth checks, and validation checked
  • New dependencies justified; lockfile churn matches the manifest

Approving

  • I can explain every line I'm approving, with the agent's help if needed
  • A named human owns the merge; bot approvals didn't fill a human slot
  • Rollback is one step (revert, flag, or staged release)

As a team

  • Open agent PRs per person and PR size are capped
  • Repeated review findings become lint rules, checks, or instruction-file lines
  • Review time, rounds, and revert rate are tracked monthly

The practical answer for a team right now: read less code, but read the right code, read the evidence first, and make sure the person who approves can defend what shipped. Agents can take most of the reading. They can't take the accountability.

Sources

Share this post

PostLinkedIn

Harness Engineering for Coding Agents: A Practical Team Setup

Same model, different results: the harness decides. What a team controls around Claude Code or Codex, with hook configs, fast checks, and a setup checklist.

14 min readAICoding AgentsDeveloper ToolsSoftware Engineering

Does AGENTS.md Help Coding Agents? What 2026 Research Actually Shows

Three 2026 studies disagree on whether AGENTS.md helps coding agents. What the evidence supports, what to cut, and a one-afternoon eval to test your file.

11 min readAIDeveloper ToolsSoftware Engineering