Field note
ChatGPT Model Change: Fix Output Regressions
A model update is a production change when people depend on the output. Keep a small set of known-good tasks and compare behaviour before the new default reaches everyone.
A model update is a production change when people depend on the output. Keep a small set of known-good tasks and compare behaviour before the new default reaches everyone.
Symptoms
- A prompt that produced a clean quotation email for months now produces something longer, shorter, wrongly formatted or in a different tone.
- A custom GPT or project that reliably filled a template now skips fields or invents new ones.
- Extracted data that used to arrive as a tidy table now arrives as prose, and a downstream copy-and-paste step breaks.
- Answers are slower or faster, and the level of reasoning has clearly changed.
- The model picker shows a name you have not seen before, or a model you relied on has moved to a legacy menu or disappeared.
None of these mean the new model is worse in general. They mean it is different from the thing your workflow was tuned to.
Why this happens
OpenAI changes the default model in ChatGPT, retires older models and adjusts how the product routes a request between faster and slower models. Each change alters how the same prompt is interpreted. Instructions that a previous model followed implicitly may now need to be explicit. Formatting that emerged by habit may not survive.
At the time of writing, OpenAI publishes model release notes and a deprecation schedule for the API, and ChatGPT plans differ in which models remain selectable and for how long. Check the current pages rather than relying on what a colleague remembers.
For a small business the practical lesson is that a prompt copied into a shared document is not a stable asset. It is configuration that needs a test.
Step-by-step fix when something has already broken
- Confirm what changed. Check the model name in the conversation and compare it with the model used in an older conversation that worked. Look at the release notes for the date the behaviour changed.
- Reproduce with a saved example. Take an input from last month that worked, run it again with the same prompt and keep both outputs side by side. This is your evidence.
- Make the implicit explicit. Most regressions are fixed by adding what the old model assumed: the exact output format, the maximum length, the fields that must appear, the tone and one worked example.
- Move stable instructions out of the prompt. Put them in the project instructions or the custom GPT configuration so every user runs the same version.
- Try the previous model if your plan still offers it. If it fixes the problem, you have bought time, not a solution, because it will be retired.
- Re-run your saved examples after each change. Stop when the outputs match the old ones on the things that matter, not on every word.
- Tell the people who use the workflow what changed and what to check in their own outputs for a week.
Build a small evaluation set before the next update
You do not need a data science team. You need a folder.
- Pick five to ten real inputs for each workflow that matters: a supplier email to summarise, a job description to draft, a set of notes to turn into minutes.
- For each, save the output your team agreed was good, with a note of why. That note is the acceptance rule.
- Write down the checks that are pass or fail: contains the required fields, under the length limit, no invented figures, correct format for the downstream tool.
- Give the set an owner and a re-run date. When OpenAI announces a model change, or when a user reports drift, the owner runs the set and compares.
Keep the set free of real personal data where you can. Use redacted or synthetic examples that still exercise the same structure. If you want to scale this beyond a folder, OpenAI's evaluation tools on the API platform let you run the same idea with stored test cases and graders, but the folder version is enough for most small teams.
What IT or the administrator should check
- Whether your plan lets the workspace administrator control which models staff can select, and whether a default has been set for the workspace.
- Whether any automation depends on ChatGPT output being pasted into another system. Those are the workflows to test first.
- Whether custom GPTs and projects have owners. An orphaned GPT with a regressed prompt is the kind of thing nobody notices until a customer does.
- Whether anything business-critical should move from the consumer product to the API, where you can pin a specific model snapshot and receive a deprecation date. Consumer products change under you; the API tells you when.
When to escalate to OpenAI
Raise a case when a documented feature no longer works as described, when a model that the release notes say is available is missing from your workspace, or when an update produces clearly unsafe or broken output on a benign, reproducible input. Include the model name, the prompt, the input, the old and new outputs and the timestamp. A change of tone on its own is a prompt issue, not a support case.
Our note on a Claude Code rollout eval suite covers the same evaluation habit for code. If several workflows now depend on AI output, an AI governance review can set the owners, test sets and review dates in one pass.