AI-Generated Tests That Actually Help, and Where They Quietly Fail

Most teams discover the same thing the hard way: AI can produce a lot of tests very quickly, but that is not the same as producing tests you should trust. A model will happily generate assertions that mirror the implementation, lock in a bug, or pass because the fixture is too forgiving.

The useful version of AI-generated testing is narrower and more practical. Let the model draft cases, edge conditions, and fixture scaffolding, then force those tests through stronger review gates like mutation checks, invariant assertions, and flaky-test detection.

This is the workflow I would actually use in a production repo, where the goal is not more tests. The goal is faster coverage gains without filling CI with pretty nonsense.

Why this matters

AI-generated tests are attractive because they compress the slowest part of test writing: enumerating cases, setting up data, and sketching repetitive assertions. That helps most when a codebase already has a clear architecture and a human reviewer who knows what correct behavior looks like.

They become risky when teams treat passing tests as evidence of correctness. In practice, the weak points are predictable: copied implementation logic, happy-path bias, over-mocking, and fixtures that encode the very bug the test should catch.

Architecture or workflow overview

behavior brief
    ↓
LLM drafts tests + fixtures
    ↓
human strengthens assertions
    ↓
pytest or integration suite
    ↓
mutation or invariant checks
    ↓
merge only if bad changes are killed

flowchart LR
    A[Engineer writes behavior brief] --> B[LLM drafts tests and fixtures]
    B --> C[Human narrows assertions]
    C --> D[Run unit or integration suite]
    D --> E[Run mutation or invariant checks]
    E --> F{Tests kill bad changes?}
    F -- yes --> G[Merge]
    F -- no --> H[Strengthen assertions or fixtures]

Implementation details

A good prompt for AI-generated tests is contract-first and annoyingly specific. Do not ask for tests for a file. Ask for behavior.

Write pytest tests for `build_invoice_summary`.
Focus on public behavior, not helper internals.
Include:
- one golden path case
- one duplicate line-item case
- one currency mismatch case
- one empty invoice case
Use factories instead of inline dict walls.
Avoid snapshots.
Every assertion must check a business invariant or externally visible field.

Here is the kind of test shape I want after cleanup:

import pytest
from billing.summary import build_invoice_summary
from tests.factories import invoice_factory, line_item_factory


def test_build_invoice_summary_rejects_mixed_currencies():
    invoice = invoice_factory(
        items=[
            line_item_factory(amount_cents=2500, currency='USD'),
            line_item_factory(amount_cents=1900, currency='EUR'),
        ]
    )

    with pytest.raises(ValueError, match='currency'):
        build_invoice_summary(invoice)


def test_build_invoice_summary_computes_totals_from_visible_items_only():
    invoice = invoice_factory(
        items=[
            line_item_factory(amount_cents=2500, hidden=False),
            line_item_factory(amount_cents=1800, hidden=True),
        ]
    )

    summary = build_invoice_summary(invoice)

    assert summary.total_cents == 2500
    assert summary.item_count == 1
    assert summary.currency == 'USD'

The important part is not that AI drafted it. The important part is that the final assertions target outcomes, branch conditions, and business rules instead of internal steps.

name: test-quality

on:
  pull_request:
    paths:
      - "billing/**"
      - "tests/**"

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-dev.txt
      - run: pytest tests/billing -q
      - run: mutmut run --paths-to-mutate billing/summary.py
      - run: mutmut results

You can also keep one lightweight integration seam so the model cannot hide behind mocks forever.

def test_invoice_summary_api_returns_contract_fields(client, seeded_invoice):
    response = client.get(f"/api/invoices/{seeded_invoice.id}/summary")

    assert response.status_code == 200
    payload = response.json()
    assert set(payload) >= {"invoice_id", "currency", "total_cents", "item_count"}
    assert payload["invoice_id"] == seeded_invoice.id
    assert payload["total_cents"] > 0

What went wrong and the tradeoffs

Implementation mirroring

Models often reproduce the same logic that already exists in the function under test. That gives you a passing test that agrees with the bug. It looks professional and catches almost nothing.

Over-mocking

If every dependency is mocked, an LLM can produce a wall of green tests that never exercise serialization, database constraints, queue payloads, or permission checks. That is fast, but fragile.

Cost and latency

The win appears when fixtures already exist and reviewers can reject weak assertions fast. If the contract is unclear, AI mostly amplifies ambiguity and review churn.

Security and reliability

Do not feed private customer payloads or raw incident data into test-generation prompts unless your environment is explicitly set up for that. Synthetic fixtures are the safer default.

Terminal output I actually care about

$ pytest tests/billing -q
18 passed in 2.41s

$ mutmut run --paths-to-mutate billing/summary.py
1 survived, 14 killed, 0 timeout

$ mutmut results
billing.summary.x_total_discount_percent: survived

Practical checklist

• Does the test assert public behavior instead of helper internals?
• Would the test fail if a condition flipped or a field disappeared?
• Does at least one case cover invalid input or a real boundary?
• Is there a genuine integration seam somewhere in the suite?
• Did we avoid giant snapshots and over-mocking?
• Would I keep this test if a human, not an LLM, had written it?

Useful references

• Pytest docs
• Mutmut mutation testing
• Hypothesis property-based testing
• Playwright testing
• Martin Fowler on mocks and stubs

Conclusion

AI-generated tests are worth using, but only if you treat them as drafts that must earn trust. Let the model do the repetitive part. Keep the judgment, invariants, and fault-detection bar firmly human.