Reviewbot: A Pull Request Reviewer That Shows Its Work

What it does

Reviewbot watches a set of GitHub repositories, picks up the open pull requests that need a look, runs a pipeline of Claude agents over each one, and turns the result into a review: a verdict, a short summary, and inline comments written in one consistent voice. It runs as a long-lived daemon, from cron, or on demand for a single pull request from a terminal. Around the core there is a web dashboard, a terminal UI, an API server, and an MCP endpoint so other agents can ask it questions.

Three decisions shaped it more than the choice of model did. Every stage that can be plain code is plain code, and models are used only where judgment is needed. The pipeline is declared as data, so changing which agents exist or how they connect is an edit to a file. And every run leaves a complete record, so what an agent actually did can be inspected afterwards instead of guessed at.

Decisions worth explaining

Follow the events feed, not webhooks

A laptop that sleeps cannot receive webhooks, and polling the whole pull request list every few minutes rereads everything to learn about one comment. The sync daemon follows each repository's events feed instead and refreshes only the pull requests an event touched, into a local cache. For a machine that is not always awake there is an optional small recorder service that captures the feed continuously and serves it back faithfully, so nothing that happened during a nap is lost. On macOS a tiny scheduled job rides each wake: it nudges the daemon and holds the wake open until the poll finishes.

Decide the spend before spending it

Triage is a small agent that reads the pull request and decides which specialists are worth running and how many turns each one gets. It is allowed to run none. A change to a comment does not need five agents. A specialist that runs out of turns has its session resumed for one more pass whose only job is to produce the answer, so a long investigation still ends with output instead of a truncated transcript.

One voice at the end

Specialists return structured findings, each with its own confidence. Anything under a threshold is dropped. What is left goes to a single synthesizer, which merges duplicates, resolves contradictions between agents, discards false positives, rewrites the comments to match a configurable style guide, and proposes a verdict. It is the only stage whose words are posted. Having specialists that disagree is fine; having the author of the pull request read five different voices is not.

Second thoughts on a second look

Review a pull request twice and the second review will find something the first did not, even when nothing relevant changed. Some of those are real misses. Many are the model changing its mind about code it already read and passed. When I measured this on re-reviews, roughly four in five of the late findings that sat outside the new changes, on lines an earlier round had read, had never been raised before and turned out to be noise.

So on a re-review each finding carries provenance: whether it sits inside the new hunks, what the previous round did with that line (read it, opened the file but not the line, never opened the file), and whether the same point was raised before. A deterministic gate then holds back a finding that contradicts an earlier read unless it argues its case and is serious. It is held, not deleted: it stays visible in the dashboard and is never posted. Findings on lines nobody read are the opposite case. Those are real gaps in coverage, and the gate never touches them. The demo's third pull request shows both.

Verdicts are policy

The model proposes approve, request changes or comment. The repository's configuration decides which of those the bot may say. Approval is opt-in per repository; a request for changes can be downgraded to a comment. In the dashboard the bot recommends and a person submits. From the command line it posts directly. Both paths pass through the same policy.

Measure what the agents actually read

Every review writes an append-only log of every prompt and every tool call. Because the log is the full conversation, metrics can be computed from it afterwards, including for reviews that ran before the metric existed: tool calls per agent, which line ranges each agent opened, and how much of the changed code was read in full, seen only inside the diff, or never looked at. That is the coverage gutter in the demo. A finding is much easier to trust when you can see the agent opened the file it is talking about.

A broken environment is not a bad review

A review that found nothing and a review that could not run git look similar from outside. A health command reports configuration, authentication, the CLI, the daemon and recent probe failures, and it separates tooling failures from real defects. A common one is a tool that resolves in a terminal but not under a launch agent's thinner PATH, which the command calls out by name.

A smaller runtime for agents that only read

I counted the tool calls across a few hundred reviews. Five tools covered more than 99 percent of them, and writes were close to nonexistent. Review agents read. That made it practical to write an alternative runtime for a single agent role, off by default, that drives a cheaper reasoning model from another provider with just those five tools served in-process. It is deliberately stricter than the main path: shell commands are checked against an allowlist of read-only programs, with substitution, redirection and backgrounding refused, and every path is resolved and refused if it leaves the worktree. The synthesizer is excluded unless named, because it writes what gets posted.

Plugins instead of forks

New behavior lives in plugins that are discovered automatically and implement only the extension points they need: reacting to sync events, injecting context into the briefing, guarding the posted verdict, adding command-line tools agents can call, and contributing tabs, buttons and routes to the dashboard. The MCP endpoint is the same idea pointed outward, so another agent can find a review, read its findings, or start one.

What the filters cost

Nearly every stage after the specialists exists to remove something: the confidence threshold, the synthesizer's discards, the reversal gate, the cap on rounds per pull request, the verdict policy. Each one trades some recall for a quieter bot, and the trade is intentional. A reviewer that comments on everything is a reviewer people stop reading, and a finding nobody reads is worth nothing. The cost is that a real problem can be filtered with the noise, which is why the dashboard keeps what was dropped and says why.