A model your product relies on can change in two ways. The provider posts a dated end-of-life notice, or it repoints the same name at new behaviour, no version bump, no changelog. Late 2026 delivered both: OpenAI gave GPT-4o three months’ notice on 14 October, while Anthropic’s Opus 5.5 moved its default reasoning effort from high to medium under the same name. Neither tells you whether your outputs got worse, and answering that question is most of the exit cost of your AI stack, the cost of leaving a model behind.
What is model deprecation, and how is it different from a model silently changing behaviour under the same name?
Model deprecation starts when the provider sets a retirement date, and the model keeps serving normally until that date. OpenAI’s 14 October 2026 notice is the reference point, three months’ notice for GPT-4o and its family. Fine-tuning ends on 6 January, and full retirement follows on 14 January 2027. Silent drift is the other failure: the provider updates the model behind a stable alias, no notice, no version bump. No error appears, and the output just changes under the same name in your config.
What actually happens when a model is deprecated: does it get slower, or just start returning errors?
A deprecated model does not get slower or dumber. It keeps serving at full quality until the retirement date, then every request returns model_not_found, a 404 that never recovers. Retry logic makes it worse, assuming the resource comes back, which it won’t. Some vendors skip the 404 and auto-upgrade callers to a replacement, which Azure’s versionUpgradeOption governs. The provider flags the move, so it is not silent drift, but there is no 404 to alert on. Your fine-tuned models die with their base model, and new fine-tuning jobs are cut off before the retirement date arrives.
Model deprecation vs silent drift: which is more dangerous to a production system?
Silent drift is the more dangerous failure. Deprecation produces an explicit 404 on a known date, and the model_not_found error leaves no ambiguity about what broke and when. Silent drift produces no error: latency stays normal, the status stays 200, and the output is a little worse each time, so monitoring alone cannot see it. Opus 5.5 demonstrates the pattern: narration that used to arrive as text blocks now comes back as empty progress-update thinking blocks, muting agent interfaces between tool calls. A 404 shows the endpoint changed, but it does not show whether your outputs got worse. Both failure modes still require a regression suite to detect output changes.
What is the difference between a dated snapshot model ID and a latest alias, and why do they fail in opposite ways?
A dated snapshot is a pinned ID like gpt-4o-2024-05-13. Its weights never move until the retirement date, then every call returns 404, a loud failure on a calendar. A latest alias is a floating pointer to the newest version, so behaviour changes under an unchanged name, unannounced. They fail in opposite directions: the snapshot retires on a fixed date, the alias tracks whatever the provider shipped. When you pin a snapshot, you accept the retirement date that comes with that model. The next question is how much notice the date gives you.
What actually breaks when you swap one model version for another, beyond the endpoint returning a 404?
The tail is what mostly breaks on a swap: the small set of high-stakes cases that keep returning 200 while producing worse output. The 404 is the rare, easy part. Tokenisation shifts across generations, so token counts change and with them your prompt cost and truncation points. Thinking blocks are bound to the model that produced them, so a failover that cannot read them runs without that reasoning. Fine-tuned models follow their base model’s shutdown schedule, and a backup with a smaller context window rejects requests the primary accepted. Tokenisation, thinking blocks, fine-tunes and context windows all change without notice, and that is where the exit cost accrues.
How do you know whether a model swap regressed your workload, and how much eval coverage do you need to trust that verdict?
A regression suite is a fixed set of real inputs with known-good outputs, run against the replacement model on the same cases the old model handled. Reporting per category matters, because a new model can raise overall accuracy while category-level accuracy falls in the cases that carry the most weight, and an aggregate score will not show the drop. Coverage has to begin with the most important high-variance prompts and the error paths. The discipline already exists in tools like Braintrust, Langfuse and Promptfoo, and a fallback chain reuses the same eval suite.
What is a golden set, and why is it the asset that outlives every model?
The golden set is the asset behind that verdict: a curated collection of real production inputs with known-good outputs or graders, overweighting the tail cases that hurt when they are wrong. A model is something you rent from a provider, and the golden set is the part you own. It is seeded from your traces, versioned and owned by you, with open-ended outputs graded by a pinned LLM-as-a-judge checked against human labels. It outlives every model because it encodes what a correct answer looks like independently of any model. The suite runs the evaluation, and the golden set is the dataset it runs against.
The model_not_found 404 arrives on a known date, so it is easy to schedule and costs little to handle. Drift produces no error that monitoring alone can detect, so it costs more to find. Which failure you get follows from your naming regime, because a snapshot and an alias fail in opposite ways. A swap breaks more than the endpoint, and only a golden set run through a regression suite can expose the regressions it causes. The golden set is the part you own, and it remains the asset you keep through the four-part exit-cost map.
Frequently Asked Questions
Should I use a dated snapshot or a latest alias in production?
Pin a dated snapshot in production. An alias keeps resolving to the newest version, so your outputs can change between deploys with no notification. A snapshot makes the model immutable until its published retirement date, which turns a surprise behaviour change into a scheduled migration you can plan for. Reserve the latest alias for experiments and prototypes, where silent drift is cheap to absorb.
Is pinning a model ID enough to protect me from changes?
No. Pinning protects you from silent drift but relocates the risk to a date, it does not remove it. The pinned snapshot is the model that gets retired, so one day the same ID you pinned will return model_not_found. Pinning buys you a scheduled, testable migration instead of an arbitrary behaviour change, but the swap still has to happen.
Do I need to act when I see a deprecation notice, or can I wait until the retirement date?
Act as soon as the notice lands. A deprecated model keeps serving at normal quality, so the endpoint looks healthy right up to the cutoff, which makes it easy to defer. But the fine-tune cutoff arrives before retirement, and your migration needs lead time for building and validating a replacement against your golden set. Waiting until the date means doing the swap under a hard deadline.
Can I still create new fine-tunes after the base model is deprecated?
Usually not for long. Providers cut off new fine-tuning jobs before the full retirement date, and any existing fine-tuned model follows its base model’s shutdown schedule. In OpenAI’s October 2026 case, fine-tuning was cut off on 6 January 2027 while full retirement followed on 14 January 2027. Treat the fine-tune cutoff as your real deadline, not the retirement date.
Can a model disappear without a deprecation notice?
Yes. The dated notice is a provider’s good practice, not a guarantee. Smaller or less-documented models, and some marketplace or third-party endpoints, can be retired with far less warning, or repointed silently. That is why you pin snapshots, watch the endpoints you depend on, and treat the golden set as your own property: it is the one asset that survives however abruptly a model goes away.
What is the difference between an auto-upgrade and silent drift?
An auto-upgrade is a deliberate, transparent remap of your request to a newer model, usually logged and flagged by the provider, while silent drift changes behaviour behind the same name with no notification at all. Azure OpenAI’s versionUpgradeOption is an example of the former: it moves callers off a retiring model instead of returning a 404. Both change your outputs without your deploy, but only the auto-upgrade tells you it happened.
What does it mean that thinking blocks are bound to the producing model?
It means the reasoning inside a thinking block is produced by a specific model and cannot be transplanted. If you swap in a non-reasoning model, the request does not error, it simply runs without any prior reasoning. The thinking block is silently dropped, and your output loses the step that was doing the heavy lifting, which is exactly the kind of quiet regression a 404 never shows you.
Is a model swap the same thing as a failover?
No. A swap is a deliberate migration from one model to another, usually planned around a deprecation or a version upgrade. A failover is an automatic fallback to a backup model when the primary fails. The danger in a failover is that it can silently change behaviour mid-request, dropping reasoning or narrowing the context window, without the planning and eval coverage a proper swap gets.
How do I detect silent drift in production before my users notice?
You cannot rely on monitoring alone, because drift returns a 200 with no error to alert on. Run your golden set against the model on a schedule and compare per-category results against the known-good baseline. A drop in one category is drift, even if the aggregate score barely moves. In practice you usually catch it first in structured output failing schema validation, or in evals changing without a deploy.
How often should I re-run my golden set against the model in production?
Run it on a schedule and before every swap, not as a one-off. Providers can repoint an alias or change behaviour at any time, so a golden set that only runs quarterly will miss drift for months. At minimum, run the suite whenever you change models, and re-run the high-variance and error-path categories on a short cadence, because those are where silent drift shows up first.
Does silent drift always make outputs worse?
No, and that is part of the trap. Drift is any behaviour change under the same name, so it can just as easily improve one metric while quietly degrading another. A model that gets cheaper and more confident may still break your structured output schema or shift tone in ways a human grader misses. Drift is not about better or worse, it is about unannounced change, which is why you measure it against a fixed golden set.