AI-generated tests often miss bugs because they inherit the implementation’s assumptions. The same agent writes a function, decides what that function should mean and then writes tests proving its own interpretation. The suite can look thorough while never challenging the decision that was wrong.
Review generated tests by asking what failure each test would catch. Break the relevant behavior on purpose and confirm that the test fails. Then add cases derived from requirements, security boundaries and production failure modes rather than from the code’s current shape.
This is the testing companion to the broader AI-generated code review checklist.
Why generated tests look stronger than they are
Coding agents are good at producing the visible features of a test suite: descriptive names, setup helpers, mocks, fixtures and coverage across public methods. Those features make a test easy to read. They do not tell you whether it protects a useful contract.
The problem usually begins with shared assumptions. Suppose an agent implements a workspace invitation endpoint and forgets to verify that the invitation belongs to the current workspace. When asked to add tests, it reads the implementation and creates fixtures for one workspace. Every test passes. The missing authorization rule never enters the suite because it never entered the agent’s understanding of the feature.
An independent test design starts somewhere else: “A workspace administrator can revoke invitations from their workspace and cannot revoke invitations from another workspace.” That requirement produces the case the implementation omitted.
1. Happy-path coverage crowds out failure behavior
Generated tests commonly prove that valid input returns the expected result. That is useful, but it is usually the easiest part of the feature.
For a create-order endpoint, the generated suite might cover:
- valid order creates a record
- response contains the new ID
- repository method receives the expected values
The expensive failures may be elsewhere:
- product price changes between quote and checkout
- payment succeeds but the database write fails
- the request is delivered twice
- inventory reaches zero concurrently
- the user tries to purchase for another workspace
- the payment provider times out after processing the charge
Write at least one inconvenient scenario before reading the generated tests. It keeps the implementation from setting the boundaries of your imagination.
2. Tests assert implementation details instead of behavior
An implementation-shaped test checks which private function ran, which dependency was called first or how many times a helper executed. It can pass while the user-visible result is wrong, then fail during a harmless refactor.
This test says little about the contract:
it("revokes an invitation", async () => {
await revokeInvitation("invitation-1", "workspace-1")
expect(invitationRepository.findById).toHaveBeenCalledWith("invitation-1")
expect(invitationRepository.delete).toHaveBeenCalledTimes(1)
})It proves that two mocked methods were used. It does not prove that another workspace is blocked or that the invitation is gone.
A more useful pair checks outcomes and permissions:
it("removes an invitation owned by the workspace", async () => {
await createInvitation({ id: "invitation-1", workspaceId: "workspace-1" })
await revokeInvitation("invitation-1", "workspace-1")
await expect(findInvitation("invitation-1")).resolves.toBeNull()
})
it("does not remove an invitation from another workspace", async () => {
await createInvitation({ id: "invitation-1", workspaceId: "workspace-2" })
await expect(
revokeInvitation("invitation-1", "workspace-1")
).rejects.toThrow("Invitation not found")
await expect(findInvitation("invitation-1")).resolves.not.toBeNull()
})The exact helpers will differ in your repository. The distinction matters: assert the promise the system makes, not the route the current code takes to keep it.
3. Mocks remove the part that can fail
Generated tests often mock every dependency because it is fast and straightforward. A mock can also erase serialization bugs, database constraints, transaction behavior and framework configuration.
Watch for tests that:
- mock the database while claiming to test a query
- mock the HTTP client while claiming to test request serialization
- mock authentication middleware while claiming to test authorization
- mock the queue while claiming to test retry behavior
- mock the parser while claiming to test input validation
Unit tests still have a place. Use them for local branching and calculations. Add an integration test at the boundary where the risk lives. If the bug would happen because two real components disagree, a test where both components are replaced by agreeable mocks cannot catch it.
4. Assertions can pass without proving anything
Some generated assertions are so broad that almost any result satisfies them:
expect(result).toBeDefined()
expect(response.status).toBeLessThan(500)
expect(items.length).toBeGreaterThanOrEqual(0)Others only repeat fixture values or snapshot a large object nobody will inspect. A snapshot update can turn an accidental behavior change into an approved one with a single keypress.
Ask what wrong result would still pass the assertion. If the answer includes the bug you care about, tighten it.
Prefer assertions such as:
- unauthorized request returns the expected status and performs no write
- duplicate delivery creates one side effect
- migration preserves a specific pre-existing value
- validation rejects a boundary value before calling the dependency
- response excludes a sensitive field
Specific assertions make failures useful. They also make suspicious snapshot changes easier to reject.
5. Generated fixtures are too clean
AI-created fixtures tend to be small, valid and internally consistent. Production data contains missing optional fields, old enum values, duplicate relationships, partially migrated rows and text nobody expected to be that long.
Build cases from the states your system has accumulated:
- records created before the latest migration
- inactive users who still own resources
- deleted parents with retained audit records
- two tenants using the same external identifier
- timestamps on daylight-saving boundaries
- empty collections and maximum-size collections
- Unicode, newlines and delimiters in names
Do not turn every unit test into a data museum. Put realistic dirty states where they challenge an invariant or a migration assumption.
6. Coverage can reward low-value tests
Line coverage tells you which lines ran. It does not tell you whether assertions would catch a regression. An agent can raise coverage quickly by calling every branch with weak checks.
Use coverage to find code nobody exercised, then inspect the tests protecting high-risk behavior. For critical conditions, mutation testing or a manual mutation provides stronger evidence: change the condition, remove the write or alter the returned field and see whether a test fails.
You do not need a full mutation-testing system to begin. Pick the condition you would least like to break in production and invert it locally.
7. The test and implementation share the same misunderstanding
Different files do not create independent verification when they come from the same prompt and evidence.
Common shared misunderstandings include:
- interpreting “admin” as any authenticated user
- applying a limit per user instead of per workspace
- treating a timeout as proof that an operation failed
- assuming a third-party API is idempotent
- using local time where the product rule uses UTC
- accepting a library method that exists only in a newer version
Derive tests from a source outside the implementation:
- product acceptance criteria
- API and protocol documentation
- database constraints and invariants
- security policies
- prior incidents and bug reports
- behavior of the previous version
This is also why asking a second model to review the same diff may not be enough. The article on whether AI should review its own code explains how to create a more independent review process.
8. Tests omit time, retries and concurrency
Agents usually generate synchronous examples unless the prompt calls out distributed behavior. Real systems retry, overlap and fail halfway through.
For code with side effects, ask:
- What happens if this request runs twice?
- Can two requests observe the same starting state?
- Does the job retry after the side effect but before acknowledgement?
- Can clock changes or expiry boundaries alter the result?
- What remains when the second step fails?
Use a controllable clock for expiry logic. Use barriers or explicit transaction steps for concurrency tests rather than hoping two promises overlap at the right moment. Make duplicate delivery a normal fixture for webhook and queue handlers.
A process for reviewing AI-generated tests
1. Restate the contract
Write one sentence describing the behavior the test suite must protect. Do this without copying the function name or implementation.
2. Name the expensive failures
List the two or three wrong outcomes that would matter most: unauthorized access, duplicate charge, lost data, broken migration or incompatible API response.
3. Map each failure to a test
Find the exact test that would fail. If there is no clear answer, the suite has a gap.
4. Break the implementation
Remove or invert the important condition. Confirm that the mapped test fails for the expected reason.
5. Reduce over-mocking
Run at least one test across the boundary where components can disagree.
6. Add independent cases
Use requirements, incidents, security boundaries and old production states. Do not ask the implementation to invent its own opposition.
7. Delete decorative tests
Remove tests that only restate mocks, assert that values exist or duplicate a stronger test. A shorter suite with meaningful failures is easier to trust.
A review checklist for generated tests
### AI-generated test review
- [ ] Each important test protects a named behavior or invariant.
- [ ] Failure and permission cases are derived independently from the code.
- [ ] Assertions reject the wrong outcomes we care about.
- [ ] Mocks do not remove the boundary under test.
- [ ] Fixtures include relevant old, empty, duplicate and cross-tenant states.
- [ ] Retries, partial failures and concurrency are covered where applicable.
- [ ] Critical tests fail when their protected condition is removed or reversed.
- [ ] Coverage is not being used as proof of correctness.Frequently asked questions
Are AI-generated tests reliable? They can be useful starting points. Reliability comes from whether the tests protect real requirements and fail when behavior breaks, not from who wrote them.
Can AI write good unit tests? Yes, especially for well-specified local behavior. Give it the contract, boundary cases and existing test conventions. Review the assertions and add cases from outside the implementation’s assumptions.
Why do AI-generated tests use so many mocks? Mocks make examples easy to construct from a single file. They also let the agent avoid setup it does not understand. Keep mocks for local behavior and use real boundaries where serialization, permissions, transactions or configuration create the risk.
How can I tell whether a test is useful? Name the bug it should catch, introduce that bug and run the test. If it stays green or fails for an unrelated reason, improve or remove it.
For the broader pull request pass, use the AI-generated code review checklist. Make the test-specific checks above part of that review whenever a coding agent produced both implementation and tests.