Teach an LLM to review your pull requests — and to know when to shut up

Why build one when you can buy five

You can buy an AI code reviewer before lunch. CodeRabbit, Qodo’s open-source PR-Agent, Copilot code review, Anthropic’s Claude Code action — the space is crowded and the demos are good. So this piece needs a reason to exist, and it has two.

The first is that the hard part of an AI reviewer is not the reviewing; it is the noise policy. Every one of those tools makes silent decisions about what is worth saying, how sure it must be before saying it, and when a human should be pulled in — and buying a tool means inheriting those decisions, which is a business call wearing a technical costume whether you noticed making it or not. The second reason is that building a small one — about two hundred unglamorous lines — teaches you exactly what the vendors are selling, and what to demand from them.

The design stance for the whole build: treat the model as a sharp junior reviewer you do not fully trust yet. Everything it says gets checked for shape, scored for confidence, gated by a threshold, and audited by a test suite it cannot see. That stance produces four steps.

Step 1 — force a shape

A reviewer that answers in prose cannot be gated, filtered, or measured, so the first guardrail is structural: the model is only allowed to speak through a schema. With tool use, you define the shape of a finding and force the model to fill it in.

import anthropic

REVIEW_TOOL = {
    "name": "report_findings",
    "description": "Report code review findings for this pull request.",
    "input_schema": {
        "type": "object",
        "properties": {
            "findings": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "file": {"type": "string"},
                        "line": {"type": "integer"},
                        "severity": {"type": "string",
                                     "enum": ["blocking", "major", "minor"]},
                        "issue": {"type": "string"},
                        "why_it_matters": {"type": "string"},
                        "confidence": {"type": "number",
                                       "minimum": 0, "maximum": 1},
                    },
                    "required": ["file", "line", "severity", "issue",
                                 "why_it_matters", "confidence"],
                },
            }
        },
        "required": ["findings"],
    },
}

client = anthropic.Anthropic()

def review(diff: str, description: str) -> list[dict]:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=4096,
        tools=[REVIEW_TOOL],
        tool_choice={"type": "tool", "name": "report_findings"},
        messages=[{"role": "user",
                   "content": PROMPT.format(diff=diff, description=description)}],
    )
    block = next(b for b in response.content if b.type == "tool_use")
    return block.input["findings"]

Two fields do most of the work. why_it_matters forces the model to argue consequences instead of asserting taste, and reading it is the fastest way to spot a hallucinated finding. confidence we will deal with in a moment, because it lies.

The prompt matters less than people hope and more than they budget for. Three lines of it are load-bearing:

Report only issues you would block or seriously question a merge over.
Do not report anything a linter or type checker would catch.
Set confidence to how likely the issue is real, not how strongly you
feel about it. An empty findings list is a valid and common answer.

That last sentence is the cheapest noise reduction available. Without explicit permission to find nothing, the model treats every diff as a test it must not fail, and invents feedback to fill the silence.

One warning before moving on: a schema guarantees shape, not truth. Structured output eliminates the parsing problem and none of the judgment problem. A finding can be perfectly formed, typed, bounded — and wrong.

Step 2 — don’t believe the confidence

So the model now reports a confidence number with every finding. The obvious move is to trust it, and the research is direct about why you should not. Asking a model to verbalize its confidence produces a more usable signal than its raw token probabilities, but that signal skews systematically overconfident — the model saying 0.9 is not right nine times out of ten. The number is a ranking, not a probability. It reliably tells you this finding is shakier than that one; it does not tell you how shaky either is in absolute terms.

The practical consequence: you never hardcode a meaning for 0.8. You choose the threshold empirically, against labeled data, in step four — the same way you would calibrate any other instrument you did not build yourself.

While we are distrusting the model, one more receipt belongs here. LLM judges carry measurable biases — they prefer the first option shown, they reward verbosity, and they rate their own output about ten percent too kindly. That last one sets a rule for this whole build: the reviewer never grades its own homework — not in production, and not in the eval suite that judges it.

Step 3 — the shut-up gate

Now the part that makes the tool trustworthy, which is not the model call. It is fifteen lines of routing.

POST, FLAG, DROP = "post", "flag_for_human", "drop"

def route(finding: dict, threshold: float = 0.8) -> str:
    confident = finding["confidence"] >= threshold
    serious = finding["severity"] in ("blocking", "major")
    if confident:
        return POST
    if serious:
        return FLAG   # never silently drop a possible blocker
    return DROP

Read the three outcomes as policy, because that is what they are. A confident finding gets posted as a review comment. An unconfident but serious finding does not get posted as if it were fact — and does not get discarded either. It becomes a flag: the reviewer suspects a real problem here and is not sure — human input needed. Only the unconfident trivia gets dropped, and dropping it is the point. Every false positive a reviewer posts is a tax on the whole team, collected one eye-roll at a time, and a reviewer that costs more attention than it saves gets muted within a month — the fate of most AI reviewers, and deservedly.

This gate has a proper name in the literature: a reject option, the classifier’s right to say “not my call.” The threshold is a dial trading coverage against precision, and where you set it is a team decision, not a constant. The flag lane matters most: it is the executable form of a rule I have argued belongs in every AI policy — a human owns every merge. The reviewer’s job is never to approve or block. It is to make the human’s judgment cheaper and better aimed, and to be honest about which findings deserve their attention rather than their trust.

Step 4 — eval the reviewer

At this point you have a reviewer with opinions and a gate with a made-up threshold. Here is why you cannot skip what comes next. Greptile publishes a planted-bug benchmark scoring itself at 82 percent; when a competitor re-ran the same repos, that number came out near 45. I am not accusing anyone of cheating — the two runs measured subtly different things, which is exactly the point. Vendor numbers do not transfer, to each other or to your codebase. If you want to know whether your reviewer works, you need a golden set of your own.

Building one is the same discipline I argued for with evals generally, applied to this tool. Take twenty or thirty pull requests from your own history. For each real bug you can trace to a fix commit, reconstruct the PR that introduced it — you now have a diff with a documented, planted defect and a known location. Add a handful of genuinely clean PRs, because a reviewer must also pass the test of staying quiet. Then the eval suite is ordinary pytest:

import json, pathlib, pytest

GOLDEN = [json.loads(p.read_text())
          for p in pathlib.Path("golden").glob("*.json")]

@pytest.mark.parametrize("case", GOLDEN, ids=lambda c: c["name"])
def test_reviewer_on_golden_pr(case):
    findings = review(case["diff"], case["description"])
    posted = [f for f in findings if route(f) == POST]

    if case["bug"] is not None:
        hits = [f for f in posted
                if f["file"] == case["bug"]["file"]
                and abs(f["line"] - case["bug"]["line"]) <= 3]
        assert hits, f"missed the planted bug in {case['name']}"
        assert len(posted) - len(hits) <= 2, "too noisy to trust"
    else:
        assert len(posted) <= 1, "invented problems in a clean PR"

Score precision before recall — a reviewer that catches sixty percent of real bugs and posts almost no junk earns trust; one that catches ninety percent inside a haystack of noise gets muted, and its true positives get muted with it. And this suite is where the threshold from step three stops being made up: run the golden set at 0.6, 0.7, 0.8, and pick the value where the noise budget holds. From then on, one habit keeps the whole thing honest — every bug that reaches production through a reviewed PR becomes a new golden case. The suite stops being what you imagined going wrong and becomes a record of what did.

Step 5 — wire it in

The plumbing is deliberately boring.

name: llm-review
on: pull_request

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: pip install anthropic
      - run: python reviewer.py --pr ${{ github.event.number }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Inside reviewer.py, the three routing lanes become GitHub furniture: posted findings arrive as review comments, flags add a needs-human-review label with a short comment naming the file and the doubt, drops go to a log you can audit later. If you would rather buy this layer, the claude-code-action exists — but keep the gate and the golden set yours, because that is where the policy lives.

What you actually built

Strip away the API calls and look at what the two hundred lines amount to: a schema that constrains what the machine may say, a confidence signal you deliberately refused to take at face value, a gate that routes doubt to people instead of hiding it, and a test suite that audits the auditor. None of that is machine learning — it is the same governance you would wrap around any fast, confident, occasionally wrong junior colleague, written down as code.

Which is the real lesson of the build, and it echoes everything I have written about AI systems lately. Teaching the model to find problems took an afternoon and someone else’s billions. Teaching it when to shut up — what was worth saying, how sure is sure enough, when to ask for help — took every interesting decision in the project, and not one of those decisions was made by the model.