A half-assed assessment of open source AI code review tools

Source

Introduction

Hi there, blog readers! For the last week or so I've been poking into AI code review tools. Yes, this is partly because of the Red Hat "you must do AI things!" policy. But also, to be honest, because they seem to be...actually good now. I set up AI reviews for pull requests to our openQA test repo as an experiment. But especially over the last couple of months, they've got to the point where well over half of the review notes are actually useful, and the writing style isn't so awful I want to stab myself in the eyeballs. So I'd quite like to keep doing them, but in a more open source-y way. So far I've simply been cloning the pull requests to a GitHub mirror of the repo that exists solely to get AI reviews done. That repo has Gemini Code Assist enabled so the PRs are reviewed by Gemini automatically, e.g. here. It's very simple, but entirely closed source, there's no control over it, and Google could take it away at any time.

We're in the middle of migrating Fedora projects from Pagure to our new Forgejo instance, so I decided to try and get some sort of AI review system integrated with Forgejo. And I kinda succeeded! I wrote a Forgejo integration for ai-code-review, a tool I found that was written by another Red Hatter, and managed to set up a proof-of-concept Forgejo Actions workflow using it on a repo I own that's hosted at Codeberg (since Codeberg has public Forgejo Actions runners available; we don't have Actions entirely set up in the Fedora instance yet). Right now it's using Gemini as the model provider just because that was the easiest thing to set up for a PoC, but ai-code-review's design makes the LLM provider easily pluggable, so it's trivial to swap it out. Long term I hope we'll get a Fedora LLM provider set up, serving open source models, and we can make it use that. There's an Ollama backend, and adding an OpenAI API backend should be pretty easy.

Before going any further with that, though, I decided to look around and see if there are other tools out there, and if so, which might be the best one. I poked around a bit and found a few, and wrote up a very half-assed comparative assessment. I figured this might interest others, so I've prettied it up a tiny bit and put it below. I make no claims that this is comprehensive, accurate or fair, please send all complaints to the happyassassin.net HR department! The takeaway is that I'll probably keep working on the ai-code-review approach and also experiment with forking Qodo's archived open-source pr-agent project and see if I can add Forgejo support to it, to compare it against ai-code-review.

If anyone knows of any I missed, please let me know! I briefly looked at RhodeCode but discounted it because it's a whole-ass forge, not just a review tool. ReviewBoard doesn't seem to have any LLM integration as best as I could tell.

The Contenders

ai-code-review

  • Repo: https://gitlab.com/redhat/edge/ci-cd/ai-code-review
  • Author: Juanje Ojeda (Red Hat)
  • Language: Python (typed)
  • Architecture: Modular
  • Tests: Yes, LLM-generated, fairly comprehensive unit tests, very limited integration tests
  • Begun: August 2025
  • Status: Active
  • Forges: GitLab, GitHub, local changes (Forgejo supported submitted)
  • Model providers: Gemini, Anthropic, Ollama
  • Output: Console or PR/MR comment
  • Deployment: Local execution, GitLab CI, GitHub Actions (one-shot deployment via container image in CI job)
  • Prompts: Here

ai-codereview

  • Repo: Red Hat internal
  • Author: Tuvya Korol (Red Hat)
  • Language: Python (untyped)
  • Architecture: Monolithic
  • Tests: No
  • Begun: June 2025
  • Status: Active
  • Forges: GitLab, local changes
  • Model providers: RH-internal Claude, Gemini, Granite
  • Output: Console or MR comment
  • Deployment: Local execution, GitLab CI (ad hoc deployment via curl/pip in CI job)
  • Prompts: Red Hat internal

kodus-ai

  • Repo: https://github.com/kodustech/kodus-ai
  • Author: Kodus
  • Language: Typescript
  • Architecture: Modular
  • Tests: Yes, handwritten, unit and integration, not sure of coverage
  • Begun: April 2025
  • Status: Active
  • Forges: GitHub, GitLab, BitBucket
  • Model providers: OpenAI, Gemini, Anthropic, Novita, OpenRouter, any OpenAI-compatible
  • Output: MR/PR comment and/or review (seems to depend on configuration)
  • Deployment: Local via yarn (indicated as for development only), as containerized webapp (for prod) with own installer - looks complex
  • Prompts: Here

pr-agent

  • Repo: https://github.com/qodo-ai/pr-agent
  • Author: Qodo (formerly Codium)
  • Language: Python (untyped)
  • Architecture: Modular
  • Tests: Yes, handwritten, unit and integration, somewhat primitive, many commented out, 24% coverage (per codecov)
  • Begun: July 2023
  • Status: Archived (Nov 2025)
  • Forges: GitHub, GitLab, Gitea, Gerrit, BitBucket, AWS CodeCommit, Azure DevOps, local changes
  • Model providers: Any OpenAI-compatible (looks like some special handling for Azure), LiteLLM
  • Output: MR/PR comment and/or review, has interactive features
  • Deployment: Local execution or Forge CI. There's a custom GitHub action but it may be abandoned. Installable via pip, should be trivial to containerize for simple one-shot CI job deployment
  • Prompts: Here

ai-pr-reviewer

  • Repo: https://github.com/coderabbitai/ai-pr-reviewer
  • Author: CodeRabbit
  • Language: Typescript
  • Architecture: Modular
  • Tests: Barely any
  • Begun: Feb 2023
  • Status: Archived (Nov 2023)
  • Forges: GitHub
  • Model providers: OpenAI
  • Output: PR review/comment
  • Deployment: GitHub Action (no longer maintained). No generic or local deployment documented
  • Prompts: Here

Conclusions

ai-code-review (Juanje) and pr-agent (Qodo/Codium) seem the best options.

Of the RH-developed, greenfield projects, ai-code-review is more featureful and better architected than ai-codereview, and not tied to an RH-internal model provider.

Of the existing public projects, ai-pr-reviewer (CodeRabbit) was very tied to GitHub, has no documented standalone deployment ability, and was archived fairly early in development. Plus it's in TypeScript. Kodus is actively developed, but similarly is in TypeScript, deployment looks complex, and from what I've seen I don't love its review style. Hard to say why but the project overall gives me a sloppy vibe. pr-agent (Qodo) had the longest development history and seems the most mature and capable at the point where it was abandoned (well, they actually seem to have done a heel turn and gone closed source / SaaS). It has a documented standalone deployment process which looks relatively simple and subject to integration into generic CI workflows.

Comments

No comments.