Reviewing code written by an AI agent

Goal

Review a change written by a language-model agent, or with heavy assistance from one, so that the failure modes typical of generated code are caught, while the ordinary review standard still applies to everything else.

Prerequisites

A change description that states what was asked, what the agent did and what it did not verify; a reviewer who can run the code and the tests; the team's usual review checklist.

Steps

1. Read the intent first: compare the request with what the diff does. A commonly reported pattern in generated changes is scope drift: unrequested refactors, renamed identifiers or "improvements" to adjacent code. Ask for the extras to be split out before reviewing the rest.
2. Confirm that every new dependency, API, flag and function exists in the versions in use: open the import, the registry entry or the documentation. Plausible but non-existent APIs are a commonly reported failure mode of generated code.
3. Verify claims of testing by evidence, not by the sentence: find the CI run. Run the tests the change added and break the code under test temporarily to confirm they can fail.
4. Read the tests before the implementation. Generated tests often assert the implementation's current output rather than the requirement, mock the very thing under test, or guard the assertion with a condition that never holds.
5. Hunt for silently handled errors: broad except or catch blocks, default values returned on failure, unbounded retries, and logging in place of propagation.
6. Apply the usual care to boundaries and security-relevant paths: input validation, path handling, shell and SQL construction, secrets in code or logs. Generated code reads fluently, which invites skimming; slow down exactly where it looks most confident.
7. Check that comments and docstrings describe the code as it ended up, not as it was first drafted; stale comments are common after iterative generation.
8. State in the review which parts were run, read closely or only skimmed, so the next reader knows where human attention went.

Expected result

Defects characteristic of generated code (invented APIs, tautological tests, scope creep, hidden error handling) are caught at review, and the review record shows what a person verified.

Limits and test basis

This is a proposed protocol; no comparison of defect rates between generated and hand-written changes is claimed. The steps add review time, and on very small changes steps 2 to 5 may be all that is needed. A change with no account of how it was produced should be reviewed as if it were generated.

Automate before reading

Run the mechanical checks before a person reads the change, and have the author attach the results. A build with the type checker or import resolver settles whether every API and dependency exists (step 2). A mutation-testing run scoped to the diff settles whether the new tests can fail (step 3), and a failing test run from before the fix is acceptable evidence when a mutation tool is unavailable. A linter configured to flag empty or blind except and catch blocks covers the obvious half of step 5. Reviewer time then goes to the questions tools cannot answer: whether the diff matches the request, whether the tests assert the requirement rather than the implementation, and whether the boundaries are handled.