Can we use analytics cookies? They count visits and form use. We run no ads.

Types of Automation Testing

AI code review checklist: how to review AI-generated code, with a tested example

14 min read

Published . Last updated . Published by Zapple.Tech, founded by the founders of CodeceptJS and Codeception, open-source testing frameworks.

AI code review checklist: how to review AI-generated code, with a tested example
Short answer

An AI code review checklist covers what a green build cannot show. Does the code meet the requirement, including the cases the prompt left out? Can its tests fail when the code is wrong, and only then? Do its packages exist, and are permission checks still in place? Write down what the code must do first, then review the diff.

This post is about reviewing code that an AI assistant or agent wrote. It is not about AI tools that do the review; one section near the end covers those. It gives one AI code review checklist, then tests its hardest item on real assistant output.

The timing matters. A Harvard working paper finds that code review became the bottleneck after firms adopted AI coding agents (Chen and Stratton, 2026). More code arrives, and it reads well. Whatever the answer to whether AI will replace software testers, someone still has to decide whether to merge it.

What should an AI code review checklist cover?

An AI code review checklist covers seven groups, each one a thing the build result does not tell you. Start with the obligations, before you open the diff.

Group What the build result does not tell you
Obligations Before reading the diff, write what the code must do and must never do: constraints, invariants, edge cases, error handling
Requirement The code does what was asked, including the cases the prompt did not mention
Tests Can the tests fail for the wrong reason, and can they fail for the right one? Expected values come from the requirement; no test is deleted or skipped
Existing behavior A regression set shows that nothing that worked has changed
Dependencies Every package and function exists, is used as documented and was chosen on purpose
Boundaries Permission checks on every role, input validation, secrets, error handling that does not hide failures
Consistency The code follows the code base’s structure; no second copy of an existing helper

This post’s AI code review checklist. The groups follow the five review areas published on Zapple.Tech’s page on testing AI-generated code, plus obligations and tests.

Paste it into your pull request template, so the author answers it before the reviewer starts.

The Tests row asks two questions on purpose. A test that cannot fail when the code is wrong protects nothing. A test that fails when the code is right sends someone after a bug that is not there. Repeated, it teaches people to ignore red builds. The run below found the second kind.

Why does AI-generated code need a different review?

Because it reads well and can still be wrong, and people distrust it and still do not check it.

A survey by Sonar, which sells code quality tools, asked over 1,100 developers about AI-written code (Sonar, January 2026). 96% said they do not fully trust that AI-generated code is functionally correct. Only 48% said they always check it before committing. 38% said reviewing it takes more effort than reviewing a colleague’s code.

Distrust without checking is one gap. Overconfidence is the other. In a controlled experiment, participants without explicit obligations overestimated the code’s correctness by about 23 percentage points (Onagh and Nayebi, November 2026).

So the review needs a written standard to check the code against, not only a careful read.

How do you check that AI-generated code meets the requirement?

Write the obligations before you read the code. Then check the code against your list, not against its own comments.

Onagh and Nayebi tested what explicit obligations in the specification change. Their 72 participants implemented and revised Python tasks with an LLM. Lightweight structure in the specification cut hidden-test failures by only 8.7%. Requiring constraints, invariants, negative requirements, edge cases, tie-breaking rules and exception handling cut them by 47.2%. Invariant violations fell by 66.1%, and unsupported assumptions by 59.5%.

In that study the obligations went into the specification used with the LLM, so it measures better specifications, not better review. Its link to review is calibration: with explicit obligations, participants judged generated code more cautiously. The paper was posted on arXiv in November 2026; its arXiv entry says it was accepted to ICSE 2026.

Six kinds of obligation, one question each:

  1. Constraints. What limits must every result respect?
  2. Invariants. What must stay true for every input, such as “the parts add up to the whole”?
  3. Negative requirements. What must the code never do?
  4. Edge cases. What happens at zero, at one, at the largest value and just past it?
  5. Tie-breaking rules. When two answers are valid, which one wins, and does anyone care?
  6. Exception handling. Which inputs must be refused, and how?

The task in the next section leaves its main invariant unstated: the installments must add up to the total. In all five runs, the assistant kept that rule anyway. The list still paid off on run 2, in a narrower way. Tests written from it passed on that correct code and got a mutation score. The assistant’s own suite got none.

How can you tell whether tests for AI-generated code can fail?

Break the code on purpose and see which tests notice. That is mutation testing: a tool makes small changes to the code, called mutants, and reruns the suite for each one. A mutant that no test kills marks a gap in the tests. For the idea in more depth, see this blog’s talk recording on mutation testing.

Below is one run of that check on code from a coding assistant, with all five drafts reported.

The task the assistant received

Each run was a fresh, headless session of Claude Code in an empty folder. It could write files but not run commands, so it never ran its own tests. It received only this text:

Write a TypeScript function splitIntoInstalments(total: number, count: number): number[] in instalments.ts that splits an order total in euros into count monthly instalments, each rounded to cents. Also write Vitest unit tests for it in instalments.test.ts.

The obligations were written before any draft was read, and the assistant never saw them. The installments sum exactly to the total, compared in integer cents. Each has at most two decimals. They differ by at most one cent. A count that is not a positive integer throws, a negative total throws, and a total of 0 returns zeros.

What the assistant wrote

0 of 5 drafts broke an obligation. The 16 obligation tests passed against the code of every run. Run 2 is shown because it is the only run with a failing test, and the test is wrong, not the code. Its code, saved as the assistant wrote it:

/**
 * Splits an order total in euros into `count` monthly instalments, each
 * rounded to cents. The instalments always sum exactly to the total: any
 * leftover cents are spread one each over the first instalments, so earlier
 * instalments may be 1 cent higher than later ones.
 *
 * Works in integer cents internally to avoid floating-point drift.
 */
export function splitIntoInstalments(total: number, count: number): number[] {
  if (!Number.isFinite(total) || total < 0) {
    throw new RangeError("total must be a finite, non-negative number");
  }
  if (!Number.isInteger(count) || count < 1) {
    throw new RangeError("count must be a positive integer");
  }

  const totalCents = Math.round(total * 100);
  const base = Math.floor(totalCents / count);
  const remainder = totalCents - base * count;

  return Array.from({ length: count }, (_, i) =>
    (base + (i < remainder ? 1 : 0)) / 100,
  );
}

In this run, the assistant got the business rule right. It works in integer cents, spreads the leftover cents and refuses bad input.

The test that failed on correct code

Run 2’s own suite has 17 tests. One of them failed, on code that meets every obligation:

  it("keeps instalments within one cent of each other", () => {
    const result = splitIntoInstalments(1000.01, 7);
    expect(Math.max(...result) - Math.min(...result)).toBeLessThanOrEqual(0.01);
  });

An excerpt of the Vitest output:

 ❯ runs/run-2/instalments.test.ts (17 tests | 1 failed) 6ms
   ❯ splitIntoInstalments (17)
     × keeps instalments within one cent of each other 2ms

 Test Files  1 failed (1)
      Tests  1 failed | 16 passed (17)

 FAIL  runs/run-2/instalments.test.ts > splitIntoInstalments > keeps instalments within one cent of each other
AssertionError: expected 0.010000000000019327 to be less than or equal to 0.01

The installments are 142.86 and 142.85, one cent apart, as the rule asks. The test subtracts them as floating-point euros, gets 0.010000000000019327 and compares it with 0.01. This test fails for the wrong reason. The same suite already compares sums in integer cents in another test; this one does not.

The fix is to compare integer cents:

  it("keeps instalments within one cent of each other", () => {
    const cents = splitIntoInstalments(1000.01, 7).map((x) => Math.round(x * 100));
    expect(Math.max(...cents) - Math.min(...cents)).toBeLessThanOrEqual(1);
  });

The fix was run on 2026-11-18 on a copy of run 2; the saved files were not edited. The fixed suite passed 17 of 17 tests. StrykerJS then scored it at 29 mutants killed and 2 survived (93.55%), as for the obligation tests.

The obligation tests

These tests were written from the obligations list, with literal expected values. They do not assert which installment takes the leftover cent or the error type, because the list does not say.

// Obligation tests, written by a reviewer from the obligations list before reading any draft.
// The implementation under test is chosen by the RUN variable (see vitest.config.ts).
// Amounts are compared in integer cents (Math.round(x * 100)), so floating-point noise never fails correct code.
// The list does not say which instalment takes a leftover cent, so cent lists are compared sorted.
import { describe, expect, it } from "vitest";
import { splitIntoInstalments } from "impl-under-test";

const cents = (x: number) => Math.round(x * 100);
const sortedCents = (xs: number[]) => xs.map(cents).sort((a, b) => b - a);

describe("obligations for splitIntoInstalments", () => {
  it("splits 100.00 into 3 as 33.34 + 33.33 + 33.33", () => {
    const parts = splitIntoInstalments(100, 3);
    expect(sortedCents(parts)).toEqual([3334, 3333, 3333]);
  });

  it("splits 10.00 into 6 as four of 1.67 and two of 1.66", () => {
    expect(sortedCents(splitIntoInstalments(10, 6))).toEqual([167, 167, 167, 167, 166, 166]);
  });

  it("splits 0.02 into 5 as two of 0.01 and three of 0.00", () => {
    expect(sortedCents(splitIntoInstalments(0.02, 5))).toEqual([1, 1, 0, 0, 0]);
  });

  it("splits 100.00 into 4 as 25.00 each", () => {
    expect(sortedCents(splitIntoInstalments(100, 4))).toEqual([2500, 2500, 2500, 2500]);
  });

  it("returns 49.99 for a single instalment", () => {
    expect(sortedCents(splitIntoInstalments(49.99, 1))).toEqual([4999]);
  });

  it("sums exactly to the total, in integer cents, for every total and count", () => {
    const totals = [0.01, 0.05, 0.1, 0.3, 1, 9.99, 10, 19.99, 33.33, 99.99, 100, 123.45, 1000.01, 1234.56, 99999.99];
    for (const total of totals) {
      for (let count = 1; count <= 36; count++) {
        const parts = splitIntoInstalments(total, count);
        expect(parts, `${total} in ${count}`).toHaveLength(count);
        const sum = parts.reduce((s, x) => s + cents(x), 0);
        expect(sum, `${total} in ${count}`).toBe(cents(total));
      }
    }
  });

  it("gives each instalment at most two decimals", () => {
    for (const x of splitIntoInstalments(1000.01, 7)) {
      expect(Math.abs(x * 100 - cents(x))).toBeLessThan(1e-6);
    }
    for (const x of splitIntoInstalments(19.99, 3)) {
      expect(Math.abs(x * 100 - cents(x))).toBeLessThan(1e-6);
    }
  });

  it("keeps instalments within one cent of each other", () => {
    for (const [total, count] of [[1000.01, 7], [0.05, 3], [99.99, 12]] as const) {
      const c = splitIntoInstalments(total, count).map(cents);
      expect(Math.max(...c) - Math.min(...c), `${total} in ${count}`).toBeLessThanOrEqual(1);
    }
  });

  it.each([0, -1, 2.5, NaN, Infinity])("throws when count is %s (not a positive integer)", (count) => {
    expect(() => splitIntoInstalments(100, count)).toThrow();
  });

  it.each([-1, -0.01])("throws when the total is negative (%s)", (total) => {
    expect(() => splitIntoInstalments(total, 3)).toThrow();
  });

  it("returns zeros for a total of 0", () => {
    expect(splitIntoInstalments(0, 3).map(cents)).toEqual([0, 0, 0]);
  });
});
Mutation check: the code from an AI assistant is changed on purpose in small ways, then two test sets run, the tests the assistant wrote and tests written from the obligations; a change that no test notices marks a gap in the tests

What the mutation run found

StrykerJS mutated each draft and scored both suites against the same mutants.

Run Assistant’s tests Obligation tests (16) Assistant’s tests: killed / survived Obligation tests: killed / survived
1 pass, 9 of 9 pass 28 / 2 (93.33%) 28 / 2 (93.33%)
2 fail, 16 of 17 pass not scored 29 / 2 (93.55%)
3 pass, 18 of 18 pass 28 / 2 (93.33%) 28 / 2 (93.33%)
4 pass, 17 of 17 pass 29 / 2 (93.55%) 29 / 2 (93.55%)
5 pass, 11 of 11 pass 29 / 2 (93.55%) 29 / 2 (93.55%)

In the four scored runs, the assistant’s tests and the obligation tests killed the same mutants. Both left the same two alive: the two error-message strings, replaced by an empty string. Both suites assert that the function throws, and neither checks the message. Whether that matters is a reviewer’s call.

Run 2’s own suite got no score at all, because StrykerJS stops when a test fails before mutation starts:

19:33:07 (8823) ERROR DryRunExecutor One or more tests failed in the initial test run:
	splitIntoInstalments > keeps instalments within one cent of each other
		expected 0.010000000000019327 to be less than or equal to 0.01
19:33:07 (8823) ERROR Stryker There were failed tests in the initial test run.

The obligation tests on run 2, Stryker’s summary lines:

----------------|------------------|----------|-----------|------------|----------|----------|
                | % Mutation score |          |           |            |          |          |
File            |  total | covered | # killed | # timeout | # survived | # no cov | # errors |
----------------|--------|---------|----------|-----------|------------|----------|----------|
All files       |  93.55 |   93.55 |       29 |         0 |          2 |        0 |        0 |
 instalments.ts |  93.55 |   93.55 |       29 |         0 |          2 |        0 |        0 |

So a test that fails for the wrong reason does double harm. It sends the developer after a bug that is not there. It also stops the mutation run from scoring the suite.

Tested with Node.js 26.13.1, Vitest 5.1.0 and StrykerJS 10.1.1 on 2026-11-18; code from Claude Code 2.1.327, model claude-sonnet-5-5, on 2026-11-18.

What this shows, and what it does not. It shows that a generated test can fail on correct code because of how it compares, not what it checks. It also shows that such a failure stops a mutation run from scoring the suite. In the scored runs, the assistant’s own tests did as well against mutants as tests written from the obligations. The assistant could not run commands. So the runs do not show what an assistant that runs its tests would do with a red one. They do not show how often assistants get a rule right or wrong. Five runs of one task, with one assistant and one model, are not a benchmark. The example reproduces no figure from the studies cited here. And mutation testing measures the tests, not the code; a high score against the wrong obligations is still wrong.

Zapple.Tech’s testing of AI-generated code covers review of what coding assistants wrote, tests that can fail, and regression checks in CI. It starts with a one-week code audit.

How do you secure AI-generated code?

Check the safeguards your code base already has, because the assistant may not have seen them. Then check the boundaries of the new code itself.

A.S.E 2.0, a preprint on arXiv, measured this with six LLMs (Yu et al., November 2026). Its benchmark has 145 tasks from real CVEs in 51 projects. Coding agents observed the relevant safeguard in only 27 to 36% of their trajectories. One insecure example added to the repository context raised correct-but-exploitable code by 6.3 percentage points.

What to check in the diff:

  • Authorization on every role, not only authentication. An endpoint that works for an admin may also work for everyone else.
  • Input validation at every boundary the new code opens.
  • No secrets in code, configuration or logs.
  • Packages that exist and are maintained. An invented package name can be registered later by someone else.
  • Error handling that fails loudly. Returning empty data on failure hides the failure.

What this does not solve: a code review is not a security assessment. Code that handles money, accounts or personal data also needs security testing.

Can an AI tool review AI-generated code?

It can add comments; people still decide. In Chen and Stratton’s data, 79.1% of firms had adopted AI code review tools (Chen and Stratton, 2026). Yet humans still wrote the overwhelming majority of reviews and comments.

An AI reviewer given the same context as the AI author can share its blind spots. So the obligations list and the mutation check stay with a person who did not prompt the assistant.

How do you keep review of AI-written code from becoming the bottleneck?

Decide review depth by risk, and let CI take every check a machine can run. A person then reads only what CI cannot judge.

Chen and Stratton put numbers on the bottleneck. After adoption of AI agents, the time from review submission to approval rose by 56.1%, and comments per change by 38.6%. For AI assistants, as opposed to agents, they found no significant effect on these measures. This is a Harvard working paper (version of 2026-11-14), not peer reviewed. It uses a proprietary dataset from an engineering analytics vendor covering 655 firms and 720,542 workers (Chen and Stratton, 2026).

Three rules keep the checklist from making the queue longer:

  1. A pull request says which parts an assistant wrote. The reviewer then knows where to apply the obligations check.
  2. Risk decides depth. Payments, permissions, data deletion and migrations get a second reviewer and a mutation check. A copy change does not.
  3. Machine checks go into CI first. Linting, type checks, the test suite and automated regression testing run before a person looks.

What can’t an AI code review checklist catch?

A checklist makes review consistent. It does not make code correct.

  • It cannot prove the code correct. It only lowers the odds of merging a known kind of mistake.
  • It cannot restore a requirement nobody wrote. If the obligations are wrong, every check passes against the wrong target.
  • Mutation testing costs CI time. Every surviving mutant still needs a person to judge it, like the two error-message survivors above.
  • A long checklist stops being read. Past one screen, people tick boxes instead of checking.
  • It does not move ownership. The developer who merges the code still answers for it, whoever wrote it.

Use the AI code review checklist on the next pull request an assistant wrote. Then run one mutation check on one risky module. To have Zapple.Tech review AI-written code in your repository, tell us about it in the form below.

Questions about this article

Is AI-generated code safe?

Not by default, and not unsafe by default either: it is as safe as the checks it passes. Review permission checks for every role, input validation, secrets and the packages it pulls in, then test the boundaries of what it accepts. Keep in mind that a code review is not a security assessment, so code that handles money, accounts or personal data needs both.

How do you debug AI-generated code?

Start from the requirement, not the code. Write down the behavior the code must keep, reproduce the failure as a test, then read the code path that the test runs. Ask the assistant to explain its choice only after a test pins the expected result. Then the explanation cannot move the target.

Should the same AI write the code and its tests?

It can draft both, but a person has to check that the tests come from the requirement. Tests read off the code confirm what it does, not what it should do. Check two things. Each test must fail when the code is wrong, which a mutation run shows. And it must not fail for a wrong reason, such as a floating-point comparison.

What is mutation testing?

Mutation testing is a way to test the tests. A tool makes small changes to the code, such as flipping a comparison or removing a check, and reruns the suite for each change. A change that no test notices marks a gap in the tests. It costs CI time, so teams usually run it on risky modules rather than on every file.

How long does reviewing AI-generated code take?

Longer than many teams plan for, and it depends on risk more than on size. Put the checks a machine can run into CI, so the reviewer reads only what CI cannot judge. Give payments, permissions, data deletion and migrations a second reviewer, and let a copy change or a small internal script go through with one.

Request a QA audit quote

Tell us what you ship and what worries you.

What happens after you send this

  1. You send the form: your situation and what you need tested, then your name and work email.
  2. Within one business day, we reply with a suggested first step, its scope, timeline and a quote.
  • Fixed scope
  • You decide after each step
  • The tests live in your repository
  • An NDA signed before access

“They helped us to close the positions and consult our team how to proceed further. High-quality quick feedbacks with the deep checking of the required tasks, professional approach, great understanding of the company's technologies and product, detailed reports.”

Kristina Lavrentieva, Manager at keepgo.com, Israel
  • Rated 4.9 out of 5 on Clutch, from 7 client reviews
  • 42 completed test automation and infrastructure projects
  • Founded by the founders of CodeceptJS and Codeception, open-source testing frameworks

Request a QA audit quote

    What is closest to your situation?
    Optional. The product, the platforms, how often you release, what broke last.

    You decide after our reply whether to go ahead.

    Prefer to talk first? Book a call

    One-week QA audit · written report

    Request a QA audit quote