Machine code review is no longer a pilot project. You’ve read the vendor pages and you want a winner.
The same pair of tools scores 82% versus 44% on one vendor’s benchmark and 24% versus 46% on another’s. A ranking that flips this far reflects who ran the benchmark. This article is about machine review: a model reading a diff with repository context and leaving comments, distinct from linting, tests and summarisation. By the end you can read any leaderboard sceptically and place each tool on a detection-versus-noise axis. That is the question of what actually replaces the review gate.
Greptile vs CodeRabbit vs Macroscope vs Cursor Bugbot: how do they compare on real bug detection and noise, and where do I find benchmarks that aren’t vendor-published?
Read provenance before any number. The only independent benchmark, Martian’s Code Review Bench, puts Greptile just ahead of CodeRabbit, while Macroscope’s self-published run reverses the ordering. Until you know who paid for the benchmark, the ranking means nothing.
Start with how the benchmark was built. A credible benchmark uses real repositories with real defect labels, holds data out from the tools being scored, reports precision as well as recall, and discloses who paid. Martian’s Code Review Bench is the place to begin: a lab that doesn’t train models or sell coding tools, which open-sources its data and judge prompts, and runs a gold set plus a live benchmark of what developers act on. No tool in its data found more than 63% of known issues, and Martian concedes the gold set understates precision and overstates recall.
On that benchmark Greptile sits first at 60.8% F1, from 76.2% precision and 50.6% recall, ahead of CodeRabbit’s 57.5% (64.9% precision, 51.6% recall). The two tools find a similar share of bugs; Greptile’s comments are simply more often worth reading. Precision asks how many comments are worth reading; recall asks how many real bugs were found. F1, the harmonic mean, hides the trade-off you feel: false positives cost your reviewers’ attention, and noise becomes alert fatigue.
Greptile indexes the whole codebase graph and stays quiet; one field study saw it post 120 findings with a 0% false-positive rate, though its check-run appeared on some PRs and not others. CodeRabbit is the loudest: the same study measured a 2.3% false-positive rate and five or six review cycles per non-trivial PR. Macroscope’s own benchmark reverses the ordering, putting itself first at 48.31% and Greptile (23.61%) behind CodeRabbit (45.76%). Greptile’s own run scores it 82% against CodeRabbit’s 44%. Cursor Bugbot posted a 4.8% false-positive rate and was cut by one team over the Cursor IDE seat; in-editor review may just move the queue.
Scale proves workflow integration, not detection quality. Microsoft’s internal assistant covers 90%+ of pull requests across 5,000 repositories, more than 600,000 a month, and reports no precision or recall figure at all.
Vendor documentation is fine for capability claims, but read it as marketing, and leave SWE-bench out: it measures issue resolution, a different task from review detection. Two risks rarely appear in comparison tables. Context hallucination is the tool asserting repository context it doesn’t have. Prompt injection is untrusted text reaching an agent with write access.
GitHub Copilot Code Review is the incumbent default. Already in the platform matters because it’s bundled and GitHub-native; its advantage is distribution, not measured accuracy. Graphite Diamond sits at the reviewer-assist end, the quietest at 0.62 comments a PR with the highest precision and lowest recall.
AI review versus human review: what does each actually catch that the other misses?
Human review catches intent mismatch, the change that does more or less than claimed, plus design drift and local conventions. Machine review catches pattern-level defects at volume and with consistency. Each misses what the other’s method cannot see.
The detection-versus-noise axis above is really a division of labour between two readers that see different things. Human reviewers carry the context the model was never given: the organisational memory behind why your code is shaped the way it is. Machines catch the recurring bug classes, null handling, resource leaks, injection surfaces, inconsistent error paths, and they do it without fatigue, applying one standard to every diff.
Humans miss scale and consistency, and degrade under fatigue. Machines miss intent and context they were never given, and produce confident noise. Snap’s reviewer would sometimes hallucinate intent, guessing wrong what the code was for; that’s the context hallucination named above, reported first-hand by a single vendor rather than measured across tools.
The structural argument is that these are correlated blind spots. AI writes the code and AI reviews it, reasoning over the same distribution, so the failures they share are structural rather than coincidence. That’s why a second model is not a second opinion. In the same four-tool field study, 93.4% of flagged coordinates were caught by exactly one reviewer and none by all four, which argues for pairing a human with a machine. It’s the capability question across the cluster, including what is lost when no human reads the code.
Agent review vs deterministic static analysis and CI checks: where is the boundary?
Draw the boundary by whether the check is decidable. Type errors, known-vulnerable dependencies, style rules and test failures always produce the same answer, carry near-zero false-positive cost, and belong in CI as hard gates. Agent review is probabilistic: it reasons about intent, cross-file consequence and unfamiliar patterns, and returns a judgement with an error rate. The boundary sits wherever a check can be made deterministic.
The machine side itself splits into two kinds of check. A check reducible to a rule should never go to a model. GitHub Copilot’s security coverage runs through CodeQL as a separate scan, outside the model itself; scanning and judgement are different layers. The failure mode is replacing CI gates with agent commentary. An agent comment functions differently from a check-run, and a neutral status does not mean the change has passed.
The boundary connects to governance. An auto-approve threshold is only defensible on the deterministic side; setting one is a governance decision rather than a configuration setting. With Microsoft’s assistant and GitHub Copilot Code Review already in near-universal use, the question is how machine review is governed. The Agent Review Guide is the artefact for how agent review is configured and evaluated.
Comparison resolves when you read a benchmark’s provenance before any number, place each tool on a detection-versus-noise axis, and govern the boundary between deterministic checks and probabilistic review.
A number without a disclosed paymaster is noise. Martian’s 60.8% F1 for Greptile is the strongest independent signal, and still short of reliable, because F1 hides the cost of false positives, which is what your reviewers actually feel. Shrinking what has to be reviewed moves more of the pipeline onto the deterministic side, part of the overview of the replacement stack. Read any leaderboard like a reviewer.
Frequently Asked Questions
What is a good F1 score for an AI code review tool?
There is no universal threshold, but 60.8% is currently the best independent score available, and it is not high. F1 is the harmonic mean of precision and recall, so it punishes a tool that finds bugs by commenting on everything. Judge a score against the cost you attach to a missed bug versus a wasted comment, and prefer benchmarks that let you weight those differently.
Is a three-point F1 gap enough to choose one tool over another?
Usually not. Greptile’s 60.8% and CodeRabbit’s 57.5% sit on the same independent leaderboard, but the gap between them is smaller than the swing between benchmarks, where the same pair has scored 82% versus 44% and 24% versus 48%. Sample size, gold-set labelling and judge prompts move a score by more than three points, so treat the ordering as indicative rather than settled.
What actually counts as a false positive in AI code review?
A false positive is any comment that costs reviewer attention without earning it. That includes a finding that is factually wrong, one that is correct but irrelevant because the problem predates the diff, a duplicate of a comment already made, and a style preference the repository does not enforce. Teams often count only the first, which is why measured false-positive rates look better than the experience of reading the queue.
Is it true that a reviewer commenting more is catching more?
No. Volume measures how willing a tool is to speak, not how often it is right. A reviewer that comments on every diff will have high recall and poor precision, and the reviewer reading it pays for each irrelevant comment. Graphite Diamond sits at the other end: the quietest volume, the highest precision and the lowest recall. Neither end is automatically better; the trade-off has to match the cost of a miss in your codebase.
Does an AI code reviewer get quieter as it learns your codebase?
Not by exposure alone. Most tools do not retrain on your repository, so the improvement people notice comes from configuration: per-path rules, ignore patterns, severity thresholds and feedback on dismissed comments. Snap’s engineering team reported recall climbing from 30% to 80% over a quarter, and that generic reviews started generating noise in its largest repositories until that configuration was added. Expect tuning, not osmosis.
Do AI code review tools cope with monorepos and legacy code?
Less well than vendor pages suggest. Detection depends on repository context, and in a monorepo the tool has to work out which of thousands of packages a diff actually touches. Legacy code adds unfamiliar patterns and inconsistent conventions, so reviewers either flag what is merely old or stay quiet about what is genuinely wrong. A benchmark run on tidy public repositories flatters a tool that will struggle in your oldest directory.
How real is the prompt injection risk in a pull request?
Real enough to design around. An agent reviewer reads pull request titles, descriptions, comments and code, and any of those can carry instructions written by an untrusted contributor. The risk becomes serious when the agent holds write access, can approve, or can call tools that reach secrets. Treat pull request text as data, keep tokens read-only, and never let a probabilistic reviewer auto-approve.
Does the Martian benchmark reflect a typical private codebase?
Only partly. It measures public repositories against fixes developers later merged, so it rewards tools that recognise common open-source patterns. It cannot see your internal frameworks, naming conventions or the wiring that makes a monorepo behave. Martian concedes its gold set understates precision and overstates recall, and no tool in its data found more than 63% of known issues. Use it as a prior, not a verdict.
What happens to detection quality on very large pull requests?
It falls, and noise rises. Bigger diffs dilute the context a reviewer can hold, so the tool leans harder on pattern matching and comments on more of what it sees. Benchmarks rarely report the diff sizes behind their scores, which means a headline number tells you little about a 2,000-line change. Splitting work into smaller, stacked pull requests improves both detection and precision before any tool setting changes.
Is AI code review good at finding security vulnerabilities?
It is good at pattern-level ones and unreliable beyond that. Injection surfaces, unsafe deserialisation, missing authorisation checks in a handler and known-vulnerable dependencies are the kind of defect a model spots consistently. Business-logic authorisation, threat-model-specific flaws and anything that depends on deployment context still need a human. Deterministic scanning stays the gate for dependency and configuration issues, where the answer is decidable.
How much should review speed weigh against detection rate?
More than leaderboards admit, because latency compounds. CodeRabbit’s field study put it at five or six review cycles per non-trivial pull request, which adds days to every change, and Greptile’s intermittent check-run adds uncertainty on top. But speed without precision just moves the queue from authors to reviewers. Weigh noise-adjusted throughput: how many changes reach a defensible merge per day, not how fast the first comment appears.
