AI for QA

The model under your AI workflow has a retirement date: run the swap as a regression test

Providers retire models on published dates, and the replacement may not be a drop-in. A runbook for re-evaluating an AI-assisted QA workflow before the old model is switched off.

Picture a team whose AI-assisted failure triage has run quietly for eight months. During the pilot it tuned the prompt, scored classifications against a written rubric and agreed an acceptance rate it could live with. Then an email arrives: the model behind the workflow will be switched off in two months. Someone raises a ticket called “update model ID”, and it is picked up in the final week. The first request with the new identifier returns an error, because the workflow set a low temperature to keep classifications repeatable. The engineer removes the setting, the job turns green, a few outputs look sensible and the change is merged.

Nothing has visibly broken. But the acceptance rate the team still reports was measured on a model that no longer answers, and nobody has checked whether it holds for the one that does.

Retirement notices are not unusual. As of October 2026, Anthropic’s deprecations page lists claude-sonnet-4-5-20250929 as deprecated on 30 September 2026 and due to retire on 30 November 2026. It commits to “at least 60 days’ notice” for publicly released models and states that “Requests to models past the retirement date will fail.” OpenAI’s deprecations page sets a minimum of six months’ notice for generally available models, barring safety or compliance concerns, and an entry dated 1 October 2026 schedules gpt-5.3-codex, gpt-5.4-nano and gpt-5.1 for removal on 1 April 2027. Seat-based assistants, licensed per user with a model list the vendor maintains, move as well. GitHub’s changelog announced on 18 September 2026 that six models would be deprecated across all Copilot experiences on 19 October, 31 days later, and a post on 2 October recorded four more as deprecated that day, after 29 days’ notice. Vendors use “deprecated” differently: check whether it marks a warning or the cut-off, and plan against the date the model becomes unavailable.

These pages change often, so read them for current dates and treat the examples as evidence, not as recommendations. The lasting point is that retirement is a scheduled event across vendors, and a scheduled change to something the team depends on is a familiar problem for QA. It calls for a regression test.

How a model retirement looks from each role

QA lead or test manager. Sees a workflow the team has come to rely on and a date on which its evidence expires. The fear is a quiet shift: triage labels that are wrong slightly more often, or generated tests with slightly weaker assertions, noticed only after a defect escapes. The decision is what must be re-measured before the swap and which result would block it.

CTO or Head of Engineering. Sees a dependency with an end-of-life date that appears in no dependency file. Model identifiers sit in CI variables, scripts, tool settings and vendor admin consoles. The fear is a hard failure on the retirement date in a pipeline nobody listed, or a rushed change that nobody can explain afterwards. The decision is who owns the model register, who signs the go/no-go and how much engineering time each swap is allowed.

Product or business owner. Sees nothing, which is the difficulty: outputs keep arriving in the same format. The fear is a release decision resting on triage whose quality changed unannounced. The decision is whether to accept manual fallback or heavier review if the replacement misses the bar.

Why the usual responses fall short

Editing the identifier and glancing at the output. This treats a change in behaviour as a change in configuration. The request may not even be accepted. As of October 2026, the migration guide for the replacement Anthropic recommends for that model says that for sampling parameters “a non-default value returns a 400 error” and that “Prefill returns an error”. A workflow that used a low temperature for repeatability, or a prefilled opening character to force JSON, has to be redesigned at exactly those points. Each redesign changes behaviour and needs its own check. That is one provider’s documented case; the specifics will differ by provider.

Assuming newer means better. A replacement may well be more capable in general. But the prompt, the context and the reviewers’ habits were tuned against the old model’s particular behaviour, and a general claim says nothing about your rubric. Other things move too: the same guide notes that, against the older model, the same text produces “about 30% more tokens, depending on the content”, which affects cost and output limits.

Relying on the provider’s migration checklist. A checklist covers the request: which fields to remove and what to send instead. It cannot know your acceptance criteria, and the providers do not pretend otherwise. OpenAI’s page describes its notice periods as time to evaluate replacement models and test application behaviour. Anthropic’s page advises “thorough testing of your applications with the new models well before the retirement date”. On replacing thinking budgets, the migration guide says: “There’s no fixed mapping from a budget to an effort level, so run your evaluations at two or three levels.” That sentence assumes you have evaluations to run.

A runbook for the swap

The runbook borrows the shape of a regression test: a fixed suite, a known baseline, a comparison and a gate.

1. Keep a model register

List every place where a model is chosen, including those outside the repository.

Column What to record
Workflow For example, failure triage or test generation from requirements
Where the model is set CI variable, script, tool configuration or vendor admin console
Identifier and platform The exact model name and where it is hosted
Settings relied on Temperature, forced output format, tool choice, reasoning budget, output limit
Owner The person who runs the swap
Published dates Deprecation and retirement dates, and the page they came from

The platform matters because, according to Anthropic’s page, partner-operated cloud platforms set their own retirement schedules, so one model can carry different dates depending on where it runs. Provider usage reports help with discovery: the same page describes a usage export broken down by API key and model. For seat-based assistants and testing tools with an embedded model, record who controls the model list and where changes are announced.

2. Put the dates in a calendar

Where an API provider emails notice, check its deprecation page monthly and whenever a notice arrives. Seat-based tools need closer watching: a monthly check could use up most of the 31 days GitHub gave above. Subscribe to the vendor’s changelog or admin notices, or check more often than the shortest notice seen. Either way, start the swap when deprecation is announced, not when retirement approaches. Some dates are known earlier: as of October 2026, Anthropic’s status table gives each active model an earliest retirement date. OpenAI’s page says preview models may be retired with much shorter notice, such as two weeks, so a routine workflow should not depend on one.

3. Freeze the evaluation set and capture the baseline

The evaluation set and rubric should already exist from the pilot: typical inputs, difficult cases and examples held back from tuning. Freeze three things: the inputs, the rubric and the old model’s scored outputs. If the old outputs were not kept, generate and score them now, and assume the baseline cannot be reproduced once the model is retired. Add cases from failures seen since the pilot before the comparison starts, not during it. If the rubric itself needs to change, change it first and rescore both sides.

4. Make the replacement run and log every adaptation

Get the workflow working on the new model and write down each setting that was rejected or removed, and what replaced it. Each entry becomes a test target. If a temperature setting kept output stable, measure run-to-run variation by running each case three times. If a prefill enforced the output format, count parse failures. If a reasoning setting has no direct equivalent, run the set at two or three levels, as the migration guide advises. A script can compare these repeats: agreement between runs and with the baseline’s accepted labels, and parse success. Use the counts to choose one configuration; only that one goes to blind scoring.

5. Compare side by side, scored blind

Run the frozen inputs through the chosen configuration, or reuse its first run from step 4. Give reviewers two outputs per case, old and new, unlabelled and in random order, and score them with the original rubric. Record:

  • acceptance rate, overall and for the difficult cases;
  • critical errors, meaning those the pilot named as stop conditions;
  • review minutes per accepted outcome;
  • format or parse failures and run-to-run disagreement;
  • cost and latency per case.

This runbook only measures the change in cost per case; what it does to the business case belongs on a payback sheet, which should already carry a reserve for this exercise.

6. Decide against thresholds written beforehand

The workflow owner writes the thresholds before the comparison is run, and a named person signs the decision. The values below are illustrative; yours should come from the pilot’s quality bar.

Result on the frozen set Decision
Acceptance no more than 3 percentage points below the baseline, no new critical errors, review effort no higher Go, unless the last row applies.
Acceptance below the threshold, with a specific cause in the prompt or context Revise once, then confirm on the held-back cases.
A new critical error, or acceptance below the threshold with no clear cause No-go. Evaluate another model or configuration, narrow the scope or return to the manual process.
Acceptance lower, but inside a threshold only one or two cases wide Inconclusive. Lengthen the parallel run in step 7; a Go waits for it to confirm the result.

A new critical error is a no-go whatever the acceptance rate. Higher review effort alone is not a Go: treat it like acceptance below the threshold. Each check logged in step 4 gets its own pass condition, written beforehand: for example, no parse failures and an agreed limit on cases where the three runs disagree. So does acceptance on the difficult cases. Cost and latency are gated only against a hard limit such as a job timeout; otherwise they go to the payback sheet.

7. Switch while a rollback still exists

Run the new model beside the old one on live inputs for an agreed period, with the old model’s output still the one in use. Then switch, with heavier review sampling for the first weeks. Rolling back to the old model is possible only while the provider still serves it, so switch early enough to leave a real rollback window before the retirement date. The step 6 thresholds, applied to the post-switch review sample, trigger the rollback, and the person who signed the decision calls it. After that date the fallback is the manual process or a second evaluated model. Finally, update the register, store the new scored outputs as the next baseline and record the decision in the operating guide.

Where this runbook stops working

Small sets cannot see small shifts. With 40 cases, one changed verdict moves the acceptance rate by 2.5 percentage points, so the illustrative three-point threshold amounts to a single case. Step 6 therefore treats a one-case drop as inconclusive; give more weight to critical errors and the parallel run.

Seat-based and embedded models limit what you control. You may be unable to pin a version, set parameters or keep the old model after the date. The workable version is to run a fixed set of representative tasks through both models while both are still offered, then watch acceptance and review effort after the change.

Not every change is announced as a retirement. Behaviour may also shift for reasons no deprecation page records, such as a tool changing its own prompts or defaults. Catching that takes routine sampling of live outputs against the rubric, not a one-off comparison.

The frozen set ages. The product changes and new kinds of failure appear. Refresh the set between swaps, never during one, and rescore the baseline when you do.

A checklist for the next retirement notice

  • Every place a model is chosen is in the register, with its owner, platform and settings.
  • Published deprecation and retirement dates are in a shared calendar; provider pages are checked monthly and seat-based tools followed by changelog or admin notice.
  • The evaluation inputs, the rubric and the old model’s scored outputs were frozen while the old model was still available.
  • Every rejected or replaced setting is logged and has a matching check.
  • Old and new outputs were scored blind with the original rubric.
  • Thresholds, a pass condition for each check and the decision-maker’s name were written down before the comparison.
  • The switch date leaves a rollback window before retirement, with a named trigger and owner, and the fallback after that date is agreed.

A model swap is ordinary maintenance when the workflow was built to be re-evaluated. Our AI for QA service starts with one repeated task and representative examples, and leaves the artefacts this runbook depends on: acceptance criteria, a representative evaluation set and an operating guide covering when to use the workflow, when to escalate and how to track quality, cost and model changes.

Keep exploring.

All insights