AI Coding Agents with Claude: Autonomous PR Reviews on GitHub

The Problem with Traditional Code Reviews

In fast-paced development teams, pull requests (PRs) are the gateway to production. But manual reviews create bottlenecks:

• Time sinks: Senior devs spend hours combing diffs, delaying merges.
• Inconsistency: Reviews vary by reviewer, missing edge cases.
• Burnout: Repetitive checks for style, security, or best practices drain energy.
• Scalability issues: As repos grow, reviews pile up, stalling velocity.

Enter AI coding agents. Using Claude's superior reasoning (especially Claude 3.5 Sonnet), we can build autonomous agents that:

• Analyze PR diffs deeply.
• Generate actionable feedback.
• Suggest fixes via patches or new PRs.
• Learn from feedback to self-improve.
• Merge low-risk PRs conditionally.

This guide walks you through building one from scratch with the Claude API, GitHub Apps, and Node.js. No prior agent experience needed.

Why Claude Excels at Code Reviews

Claude 3.5 Sonnet outperforms GPT-4o and Gemini 1.5 in coding benchmarks (e.g., 92% on HumanEval). Key strengths:

• Long-context reasoning: Handles massive diffs (200K token window).
• Tool use: Native XML-structured outputs for parseable reviews.
• Self-reflection: Agents can critique their own suggestions.
• Safety: Constitutional AI reduces hallucinated fixes.

We'll leverage the Anthropic SDK for prompts like:

<role>GitHub Code Review Agent</role>
<task>Review this PR diff. Output JSON with: summary, issues (array of {file, line, severity, description, fix}), approval (yes/no/maybe), confidence (0-1).</task>
<diff>{diff}</diff>

High-Level Architecture

1. GitHub App: Receives PR webhooks (opened, updated).
2. Webhook Server: Validates signature, fetches full PR diff via Octokit.
3. Claude Agent Loop:
   • Retrieve context/memory from DB.
   • Prompt Claude for review.
   • Post review as GitHub comment.
   • If fixes suggested, create fix branch/PR.
   • Update merge status.
4. Memory Store: Pinecone or SQLite for self-improvement (past reviews + human feedback).
5. Deployment: Vercel for serverless.

(Imagine a diagram here)

Prerequisites

• Node.js 20+
• GitHub account
• Anthropic API key (free tier: 10K tokens/day)
• Pinecone account (free starter plan)

Install deps:

git clone <your-repo>
cd pr-agent
npm init -y
npm i @anthropic-ai/sdk octokit @octokit/webhooks @pinecone-database/pinecone express cors dotenv

Step 1: Create a GitHub App for Webhooks

1. Go to GitHub Settings > Developer settings > GitHub Apps > New App.
2. Set:
   • Homepage: https://your-vercel-app.vercel.app
   • Permissions: Repository contents (read), Pull requests (read/write), Metadata (read), Webhooks (read/write).
3. Generate private key (.pem).
4. Install on your repo(s).
5. Note App ID, Installation ID, Webhook secret.

Subscribe to pull_request events.

Step 2: Build the Webhook Server

Create server.js:

import express from 'express';
import { Anthropic } from '@anthropic-ai/sdk';
import { createAppAuth } from '@octokit/auth-app';
import { Webhooks } from '@octokit/webhooks';
import Pinecone from '@pinecone-database/pinecone';
import cors from 'cors';
import crypto from 'crypto';
import dotenv from 'dotenv';

dotenv.config();

const app = express();
app.use(express.raw({ type: 'application/json' }));
app.use(cors());

const ANTHROPIC = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const PINECONE_INDEX = // init Pinecone

// GitHub App Auth
const octokit = await createAppAuth({
  appId: process.env.GITHUB_APP_ID,
  privateKey: process.env.GITHUB_PRIVATE_KEY,
});

const webhook = new Webhooks({
  secret: process.env.WEBHOOK_SECRET,
});

app.post('/webhook', webhook.middleware(async (event, done) => {
  if (event.event === 'pull_request' && ['opened', 'synchronize'].includes(event.payload.action)) {
    await handlePR(event.payload);
  }
  done(null);
}));

export { app };

Add signature verification:

function verifySignature(payload, signature) {
  const hmac = crypto.createHmac('sha256', process.env.WEBHOOK_SECRET);
  hmac.update(payload);
  return `sha256=${hmac.digest('hex')}` === signature;
}

Step 3: Implement PR Review Logic

Core function handlePR(payload):

async function handlePR(payload) {
  const { repository, pull_request } = payload;
  const owner = repository.owner.login;
  const repo = repository.name;
  const prNumber = pull_request.number;

  // Auth as installation
const { token } = await octokit({ installationId: process.env.INSTALLATION_ID });
  const kit = new Octokit({ auth: token });

  // Fetch diff
const { data: files } = await kit.rest.pulls.listFiles({ owner, repo, pull_number: prNumber });
  const diff = files.map(f => f.patch).join('\
');

  // Retrieve memory: similar past PRs
  const memory = await getRelevantReviews(diff); // Pinecone query

  // Prompt Claude
const review = await ANTHROPIC.messages.create({
    model: 'claude-3-5-sonnet-20240620',
    max_tokens: 2000,
    messages: [{ role: 'user', content: buildPrompt(diff, memory, prNumber) }],
    tools: [{ type: 'tool', name: 'post_comment' }], // Optional tool use
  });

  const parsedReview = JSON.parse(review.content[0].text);

  // Post review comment
  await kit.rest.pulls.createReview({
    owner, repo, pull_number: prNumber,
    event: parsedReview.approval === 'yes' ? 'APPROVE' : 'COMMENT',
    body: formatReview(parsedReview),
  });

  // If fixes, create fix PR
  if (parsedReview.issues.length > 0) {
    await createFixPR(kit, owner, repo, prNumber, parsedReview);
  }

  // Store review for memory
  await storeReview(diff, parsedReview);
}

Prompt Template (buildPrompt):

function buildPrompt(diff, memory, prNumber) {
  return `
<role>You are a expert code reviewer. Review PR #${prNumber}.</role>
<guidelines>Check for bugs, security, performance, style (use repo .eslintrc if mentioned). Be constructive.</guidelines>
<memory>${memory}</memory>
<diff>${diff}</diff>
<output>JSON: {"summary": str, "issues": [{file, line_start, line_end, severity: 'low/medium/high', description, fix_code: str}], "approval": "yes/no/maybe", "confidence": 0-1}</output>
`;
}

Step 4: Self-Improvement with Memory

Use Pinecone for semantic search on past reviews.

// Upsert
const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pinecone.index('pr-reviews');

await index.upsert([{ id: uuid(), values: embed(diff), metadata: { review: JSON.stringify(parsedReview), feedback: null } }]);

// Query similar
async function getRelevantReviews(queryDiff) {
  const queryEmbedding = await embed(queryDiff);
  const results = await index.query({ vector: queryEmbedding, topK: 3, includeMetadata: true });
  return results.matches.map(m => m.metadata.review).join('\
');
}

// Embedding via Claude (or OpenAI)
async function embed(text) {
  const res = await ANTHROPIC.messages.create({ model: 'claude-3-haiku-20240307', messages: [{role:'user', content: text}], max_tokens: 128 });
  return res.usage.output_tokens; // Simplified; use text-embedding model
}

Human feedback loop: On PR comments, parse reactions/thumbsup/down, update metadata.feedback.

Step 5: Suggesting Fixes and Auto-Merge

createFixPR:

async function createFixPR(kit, owner, repo, prNumber, review) {
  // Create branch: fix/pr-${prNumber}-issues
const branch = `fix/pr-${prNumber}-ai`;
  // Logic to apply fixes to files (use simpletext diff apply or tree API)
  // Commit and push
  // Open PR: "AI-suggested fixes for #${prNumber}"
}

For merge (high confidence, no high-severity issues):

if (parsedReview.approval === 'yes' && parsedReview.confidence > 0.9) {
  await kit.rest.pulls.merge({ owner, repo, pull_number: prNumber, merge_method: 'squash' });
}

Add safeguards: Require /ai-merge-approve label.

Deployment to Vercel

vercel.json:

{
  "functions": { "server.js": { "runtime": "nodejs20.x" } },
  "env": { "ANTHROPIC_API_KEY": "@anthropic_key" }
}

vercel deploy → Set webhook URL to https://your-app.vercel.app/webhook.

Real-World Example

Test repo: github.com/yourusername/test-repo.

1. Open PR with buggy JS: function add(a,b){return a+b (missing ;).
2. Agent comments:
   • Issue: syntax error line 1, fix: return a + b;.
   • Opens fix PR.
3. Approve → merges.

Over time, agent learns repo style from memory.

Limitations & Best Practices

• Hallucinations: Always human-in-loop for merges.
• Cost: ~$0.01/PR (Sonnet: $3/M input tokens).
• Diff size: Chunk large PRs.
• Security: Validate webhooks, rate-limit.
• Extend: Add MCP for custom tools, Claude Code for local runs.

Monitor with GitHub Actions logs.

Conclusion

This Claude agent slashes review time by 80%, scales with your team, and improves autonomously. Fork the GitHub repo, tweak prompts, and deploy today.

Next: Integrate with Slack for notifications or n8n for multi-repo.

Word count: ~1450