How to measure AI code review quality: signal, noise, and cost

Most AI code review pages use one word for quality: accuracy. That is not enough to choose a reviewer.

A tool can find many real issues and bury the team in guesses. It can post almost nothing and look precise because nobody has enough feedback to measure what it missed. It can also find a useful bug while taking twenty minutes and costing more than the human review it was meant to support.

Measure the review as a workflow. Count useful findings, missed issues, noise, time and cost together.

Decide what a good review means

Write the review contract before you compare tools. A useful finding should:

• point to a real line or small set of lines
• describe a concrete failure
• explain why the failure matters
• use evidence from the change or repository
• give the author a reasonable next step

Decide what the reviewer should ignore. Formatting, naming preferences and broad redesign suggestions may be valid discussion topics, but they should not compete with authorization bugs, broken invariants or data loss in the same comment stream.

The contract determines what you measure. If the reviewer is meant to find correctness and security issues, do not reward it for producing summaries and style suggestions.

Build a review set from real changes

Use pull requests that reflect the work your team actually does. A hand-written set of toy examples is useful for checking a single rule. It is a poor measure of a production review workflow.

Include changes with different shapes:

• a small bug fix
• a change that crosses module boundaries
• an authorization or data-access change
• a migration or schema change
• a dependency or configuration update
• a larger feature change

For each change, record the revision, repository rules, available tests and the findings a careful human review should have raised. Keep the reference set separate from the tool's output. The reviewer must not get the answer key in its prompt.

Public projects such as SWE-PRBench use human-annotated pull requests to evaluate review quality. That is a useful model for test design. Your own repository still matters more because its conventions, risk and review standards are different.

Do not use only changes that already had a dramatic bug. Include ordinary pull requests where the correct result is silence. A reviewer that comments on every harmless change is not precise.

Use more than one quality metric

Precision and recall need a reference set. A team cannot calculate recall from the comments it received because it does not know which bugs the reviewer missed. That is why a quiet tool is not automatically a good tool.

Acceptance is a practical signal, not proof of correctness. An author may accept a comment because it is easy to fix, or dismiss a correct comment because the team chose a different trade-off. Record the reason rather than treating every button click as a verdict.

Separate discovery from publication

Ask the reviewer to find candidate issues, then make it prove each one before publishing it. This gives you two measurements:

1. How many plausible candidates did the first pass produce?
2. How many survived evidence checks and deserved a comment?

The second number is closer to the experience your team pays for. A system that generates fifty candidates and publishes four may be doing useful work. A system that publishes all fifty has moved the filtering cost to the human reviewer.

For every published finding, save the evidence used to support it:

• the relevant changed lines
• the surrounding function or type
• the caller, schema or rule that creates the constraint
• the test or documentation that confirms the behavior
• the failure scenario

If nobody can explain why a finding survived, it is hard to improve the reviewer when it is wrong.

Grade findings with a small rubric

Use the same grading questions for every tool. A simple rubric is enough:

### Finding review

- Correctness: correct, partly correct, incorrect, or unclear
- Impact: blocking, warning, or informational
- Evidence: sufficient, incomplete, or unsupported
- Location: useful line, nearby line, or wrong location
- Action: changed, discussed and kept, or dismissed
- Reason for dismissal: wrong, duplicate, out of scope, or accepted trade-off

Keep "unclear" separate from "incorrect". Sometimes a reviewer identifies a risk that needs a human decision but cannot prove a bug from the available evidence. That is a context problem, not the same as a hallucinated issue.

If different people grade the same finding, discuss disagreements before changing the scorecard. Otherwise the metric will mostly measure who happened to read the comment.

Compare tools under the same conditions

When comparing reviewers, keep the input stable:

• use the same pull-request revision
• provide the same repository rules when the tool supports them
• record model, provider and configuration
• record files, tokens and time limits
• run deterministic checks separately
• do not let one tool inspect a test or answer key hidden from the others

If a tool needs a different integration, document that as part of the comparison. A review that requires an engineer to prepare context for every change has a different operating cost from one that gathers context automatically.

Do not turn the comparison into a race to produce the largest number of comments. A reviewer that posts a useful finding in five minutes may be better than one that posts twelve guesses in two minutes.

Measure the costs people forget

Record more than the model invoice.

Time to useful feedback

Measure queue time, retrieval time, model time and publication time separately when possible. A fast model cannot help an author if the review waits behind a queue for an hour.

Investigation time

Ask how long an engineer spent deciding whether each finding was real. Ten comments that take two minutes to dismiss each can cost more than two comments that lead to a code change.

Re-review work

Count what happens after the author pushes a fix. Does the reviewer inspect the new revision, repeat old comments or leave feedback on code that no longer exists? Stale comments are a quality problem and a workflow cost.

Infrastructure and provider cost

Include model calls, retrieval calls, retries, storage and the service that runs the review. The AI code review cost guide shows how to compare those costs with saved review time.

Interpret the common result patterns

High recall with low precision means the reviewer finds real problems but creates too much noise. Improve repository context, custom rules, candidate verification and publication limits before asking for more findings.

High precision with low recall means the reviewer is quiet and selective. That may be right for a safety-first workflow, or it may be missing important classes of bugs. Add known examples and check the missed cases.

High acceptance with low evidence can be misleading. Easy fixes get accepted even when the comment did not explain the problem. Keep evidence quality in the rubric.

Low comment volume with no reference set tells you almost nothing. Silence can mean a clean change, a good filter or a missed issue.

High recall on synthetic examples can also mislead. Real repositories contain undocumented contracts, generated files, stale tests and conventions that do not appear in a prompt. Test both simple known cases and real changes.

Run the evaluation more than once

One run is a demonstration. A useful evaluation has a baseline and a repeat.

Start by running the current process on the reference set. Then run the candidate reviewer with its intended rules and workflow. Grade the output without changing the rubric halfway through.

After you adjust context, rules or verification, run the same set again and add new changes. Improvements that work only on the original examples are not improvements you can trust.

Keep a dated record of:

• tool version and configuration
• model and provider
• reference changes and known findings
• published findings and grades
• cost and elapsed time
• changes made after the evaluation

AI review systems change when the model, prompt, retrieval logic or filters change. A score from last month is not a permanent product property.

What Scopy should be measured on

I build Scopy, so this is not an independent benchmark. The product is designed around repository context, review rules and verification before publication. That makes the useful questions fairly specific:

• Does the reviewer find issues that depend on callers, types or tests outside the diff?
• Do repository rules improve recall without flooding the pull request?
• How many candidate findings disappear during verification?
• Can a person trace each published finding to evidence?
• Does the workflow stay useful as pull requests get larger?

The false positives guide explains why comment count is a bad proxy for quality. The diff-only review guide explains why context changes both what a reviewer catches and what it incorrectly suspects.

A scorecard you can use

Copy this into a spreadsheet or issue before running a comparison:

## AI code review evaluation

Tool:
Version:
Model and provider:
Review rules:
Reference set:

Published findings:
Correct findings:
Incorrect findings:
Unclear findings:
Known findings in reference set:
Known findings caught:
Dismissed findings:
Duplicate findings:
Accepted findings:

Median time to first useful finding:
Total review cost:
Cost per accepted finding:

Top missed category:
Top noise category:
Next change to test:

The scorecard is deliberately plain. If a metric cannot change what you do next, it probably does not belong in the first version.

Frequently asked questions

What is the most important AI code review metric?

For most teams, start with precision and recall on real changes, then add investigation time and cost per useful finding. No single number captures review quality.

Is a high acceptance rate proof that a reviewer is accurate?

No. Acceptance reflects human behavior and can include easy fixes, incomplete comments and accepted trade-offs. Grade correctness and evidence separately.

How do I measure recall when nobody knows every bug?

Use a reference set with findings agreed on by experienced reviewers, supplemented by seeded test cases and later-discovered defects. State what the set contains and what it cannot measure.

Should I use benchmark pull requests or our own code?

Use both when possible. Public benchmarks help compare methods. Your own changes reveal repository rules, data flows and risk categories that generic examples miss.

How many comments should an AI reviewer publish?

There is no useful universal number. Publish findings that meet your review contract and suppress unsupported or low-impact commentary. A clean change should be allowed to produce no comments.