AI for QA

AI can write your Pact tests. It cannot know what your consumer relies on

Generated contract tests describe what the provider offers, not what your consumer reads. A tracing rule, a review checklist and an acceptance criterion for AI-written consumer contract tests.

Picture a checkout service that calls an internal orders API. The orders team is about to start a refactor, and the QA lead wants consumer contract tests in place first. An AI assistant offers a shortcut: give it the orders API’s OpenAPI file and let it write the tests. By evening there are 38 interactions, each with a full response body, and provider verification passes first time.

Two weeks later the orders team removes legacyStatus, a field no consumer has read for a year. Verification fails and their release is held for a day. In the same sprint they change how currency is populated, and for one market it is now absent. Verification passes, because no generated interaction asserted currency: the spec marked it optional and its example left it out. The checkout page reads that field for every price it shows. The example is illustrative, not a client case, but both failures share one cause. The generator wrote down what the provider offers. It had no way of knowing what this consumer relies on.

Generating contract tests is now a product feature. As of October 2026, PactFlow’s AI page offers to build them from “existing code, OpenAPI, or traffic data”, and in May 2026 the Pact project released agent skills and agents for Claude Code, GitHub Copilot, Cursor, Windsurf and other assistants. These pages are evidence of the trend, not recommendations. The problem applies to any generator, including a general-purpose assistant.

What a consumer contract is meant to capture

The Pact documentation defines contract testing as “a technique for testing an integration point by checking each application in isolation”. In the consumer-driven form, the consumer’s tests run its real API client against a mock provider, and “the contract is generated during the execution of the automated consumer tests”. The provider team then verifies its real service against that contract.

The value lies in what the contract leaves out. If it records only what consumers depend on, the provider can change everything else without coordinating a release. Pact’s consumer guidance states the rule directly: “Only make assertions about things that would affect your consumer if they changed.” A contract written from the provider’s own description breaks that rule from the first line.

How the problem looks from each role

QA lead or test manager. Sees a contract suite that went from nothing to dozens of green interactions in a day. The fear runs in two directions: false alarms that teach the provider team to ignore verification, and silent gaps that let a breaking change through while the dashboard suggests full coverage. The decision is what a generated interaction must prove before it is merged, and who on the consumer side signs it off.

CTO or Head of Engineering. Sees contract testing presented as a way to release services independently, an alternative, in Pact’s words, to “expensive and brittle integration tests”. The fear is paying for tooling and review time and still coordinating releases by hand, because nobody trusts what verification says. The decision is whether generated contracts may gate deployments, and how much consumer-developer review time to spend on each generated contract before it gates deployments.

Why the usual responses fall short

“Verification passes, so the contract is right.” Verification shows that the provider satisfies the contract, not that the contract captures what the consumer needs. A contract derived from the provider’s spec, recorded traffic or source code describes the provider and is then checked against the provider. It will usually pass, whether or not it protects anything.

“More interactions means more coverage.” Counting interactions measures how much of the provider’s surface has been written down. Every asserted field that no consumer code reads is a way for a harmless change to fail verification. Pact’s guidance asks for the opposite: “Only include data and operations used by the consumer”, because this “keeps the contract minimal and avoids unnecessary coupling to unrelated parts of the API.”

“The AI reviewer will catch it.” The tools’ own reviewers look for exactly these patterns. The pact-reviewer agent described in the PactFlow skill documentation lists “Testing provider business logic rather than the consumer’s API client” and “Exact matchers where type matchers would suffice” among its consumer-test red flags, and the same page says the skill flags “over-specifying response bodies”. Run such checks. But they inspect the form of a test. Whether the checkout page reads currency is a fact about consumer code outside the test file.

“Generate from the consumer’s client code instead.” Better, and the same page lists API client code as an input. Yet a client generated from the provider’s OpenAPI file typically deserialises every field in the response model, so it appears to read legacyStatus too. The real dependency sits further down, in the code that uses the client’s result. Regenerating contracts whenever the provider changes has a similar catch: the contract then follows the provider, the reverse of the intended direction.

A method: trace every interaction to a consumer call site

The rule behind the method: every interaction, and every field it asserts, must trace to a place in consumer code that depends on it. Anything that cannot be traced is removed or loosened. Anything the consumer depends on that is not asserted is a gap.

1. Build a call-site inventory before generating anything

For each provider the consumer calls, record one row per call site.

Column Illustrative entry
Call site checkout/pricing.ts, renderLineItems()
Request GET /orders/{id}, with the headers and parameters actually sent
Fields read items[].unitPrice, items[].currency, status
How each field is used Displayed, branched on, or passed on unchanged
Scenarios handled Order not found, empty item list, currency absent

To find what is read, search for usages of the client’s response type, or remove a field from the response model and see what fails to compile or test. An assistant can draft this inventory from the consumer repository, a better use of it than drafting tests from the provider spec. A consumer developer confirms each row.

2. Generate from the inventory, with the spec as reference

Give the generator the consumer’s API client, the inventory and the provider’s OpenAPI file, in that order of authority. The spec supplies shapes, example values and status codes; the inventory decides scope. Ask for each interaction to carry a comment naming its call site and the fields it covers, so review checks the trace instead of reconstructing it. Pact’s guidance applies unchanged: “Always exercise the real consumer code in your contract tests.” An interaction that builds HTTP requests by hand is rejected.

3. Set matching from how each field is used

How the consumer uses the field Matching (Pact JS names)
Displays, stores or passes it on Type matching, such as like()
Branches on its value, such as a status A regex or exact value covering only the values the code handles
Iterates over a list eachLike() with one representative element
Holds an ID or timestamp Type or format, never the exact value
Never reads it Leave it out of the contract

This mapping is our rule, but it is consistent with Pact’s consumer guidance: “Avoid asserting on exact values unless necessary”, and for responses “loose matching is generally the recommended option”. The same page asks for deterministic contract files: “Avoid dynamic data without fixed examples.”

4. Name provider states after consumer scenarios

A provider state describes the data the provider must hold for an interaction, such as “order 123 exists and has no currency”. Name states after situations the consumer handles, not provider internals. A spec lists what the provider can send; only consumer code shows which cases matter, so the unhappy paths come from the inventory’s last column.

5. Review in two passes

The first pass is automated where a reviewer or linter exists: exact matchers, random data, missing states, over-specified bodies. In the second, a consumer developer checks against the inventory in both directions:

  • every interaction and asserted field has an inventory row, or it goes;
  • every inventory row has an interaction and an assertion, or there is a gap to fill.

A test-file reviewer cannot perform the second direction, because the information it needs is not in the test.

6. Accept against two probes

With Pact, gating usually means the Pact Broker’s can-i-deploy check, which Pact’s consumer guidance recommends for a “definitive answer” on whether the consumer version being deployed is compatible with its providers. Before a generated contract is allowed to feed that check, ask the provider team to run verification against two throwaway branches:

  • Harmless change. Add a new field and remove one this consumer does not read. Verify only this consumer’s pact, selected by consumer name, because other consumers’ pacts may still assert that field. Verification must pass.
  • Breaking change. Remove a field that is in the inventory, or change its type. Verification must fail on the interaction that traces to it.

The acceptance criterion is then short: a generated interaction is merged when it meets every item on the checklist below, and a generated batch gates releases only after both probes give the expected result.

For each generated batch, record how many interactions were kept, edited and deleted, and classify later verification failures as real breaks or false alarms. Many deletions suggest the generator is getting the wrong inputs; many false alarms, that the scope rule is not being applied. Count review minutes as part of the workflow’s cost, as in a pilot with a quality bar.

Where this approach stops working

Unknown consumers. Pact’s page on what it is good for lists “Testing APIs where the consumers cannot be individually identified (eg. public APIs)” as a poor fit. Without a consumer codebase there is no call site to trace.

Pass-through providers and generic consumers. The same page lists “pass through” APIs, “where the provider merely passes on the request contents to a downstream service without validating them”, as a poor fit: the contract that matters is between the consumer and the downstream system. In our view, the same applies to a consumer that forwards a response untouched; the real reader is downstream and the inventory has to follow it there. A consumer that renders whatever fields arrive makes tracing impractical.

Event-driven integrations. As of October 2026, PactFlow’s AI page says it “supports consumer Pact tests for HTTP interactions”, with provider verification and event-based architectures planned. The tracing rule still applies to message contracts, but that generator does not draft them.

Provider behaviour. A consumer contract does not show that the provider calculates totals correctly; Pact’s page lists “Functional testing” of the provider as a poor fit and leaves it to the provider’s own tests.

An inventory that decays. New call sites appear every sprint. Make the inventory part of consumer code review: a pull request that starts reading a provider field adds the row and the assertion with it.

A checklist before a generated contract test is merged

  • It runs through the consumer’s real API client, with no hand-built HTTP requests.
  • It names its call site, and that call site is in the inventory.
  • It asserts every field the call site reads and no field it does not.
  • Exact values appear only where the consumer branches on them; IDs and timestamps are matched by type or format.
  • Example data is fixed, with no random values.
  • Provider states describe consumer scenarios, including the unhappy paths the code handles.
  • No provider business rules are asserted.
  • A consumer developer reviewed it against the inventory in both directions.

Before its batch gates releases:

  • Verification passed the harmless-change probe and failed the breaking-change probe.

Generating a contract test is now the cheap part; deciding its scope still needs someone who knows the consumer. Our AI for QA service starts from one repeated task your team brings, such as drafting consumer contract tests, together with representative examples from your own work. It sets acceptance criteria for the output, prepares the context and review boundaries the generator needs, and leaves an operating guide covering when to use the workflow and when to escalate.

Keep exploring.

All insights