Guide

Maintaining AI systems after the build: what your team has to run

Yes, if the handover is planned from the start. After launch, model versions retire on the vendor's schedule, quality drifts as inputs change, and prompts and tools need updating. Costs need watching, access needs reviewing and incidents need an owner. Anthropic gives at least 60 days' notice before retiring a publicly released model, and requests to a retired model fail. A team can do this work once it holds the code, the evaluation suite, a runbook and the access to change them.

AI system maintenance. The ongoing work that keeps an AI system correct, affordable and safe after launch, from model migrations and evaluation to cost, access and incidents.

Written by Tayyeb Mahmud, founder and Principal Architect. Claude Certified Architect – Professional · LinkedIn profile of Tayyeb Mahmud

1AYM is an OpenAI Select Partner.

Checked 22 October 2026. Model retirement dates and notice periods were read on Anthropic's, OpenAI's, Google Cloud's and Microsoft's own pages on this date. Vendors change them, so check the pages before you plan a migration.

Model retirements set the calendar

On 30 September 2026, Anthropic told developers that Claude Sonnet 4.5 retires from its API on 30 November 2026. Its recommended replacement is Claude Sonnet 5.5. Anthropic's policy is at least 60 days' notice for publicly released models. After the retirement date, requests to that model fail.

OpenAI notified developers on 14 October 2026 that GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini leave its API on 14 January 2027. Fine-tuned versions of those models follow the same date. On Google Cloud, Gemini 2.5 Pro and Gemini 2.5 Flash reached their listed retirement date on 20 October 2026. Google promises at least 45 days to migrate once it sets a date.

Hosting through a cloud provider changes the dates. Microsoft's Foundry gives its generally available models an 18-month lifecycle, or 12 months for models from Anthropic, DeepSeek, Fireworks and Mistral AI. Once a model retires there, every request to it returns an error. Anthropic notes that Amazon Bedrock and Google Cloud set their own retirement schedules for Claude.

Anthropic's API version of 21 October 2026 removed the temperature, top_p and top_k settings. A request that still sends them gets an error back. Code that worked last week can break on a version change with no model retired at all.

Six kinds of ongoing work

Model changes are one of six kinds of work that come with every production AI system. An agent that takes actions needs each of them done more often.

Model retirements and version changes: Move to the replacement inside the notice window. Rerun the evaluation on the new model first, because a newer model can still answer your cases differently.
Evaluation drift: Quality slips when the inputs change, such as when a supplier changes its invoice layout, a new product line appears or staff start asking different questions. A weekly sample of live output, checked by the process owner, catches it before users do.
Prompt and tool changes: A policy changes, a field is renamed in the CRM, or an API the agent calls gets a new version. Each change goes through the same tests and release path as code.
Cost monitoring: Usage grows, a prompt gets longer or an agent loops. Spend alerts per workflow and a monthly look at cost per case keep the bill tied to the value.
Access reviews: People join, move and leave, and the system's own service identity collects permissions over time. Review both on the same cycle as your other systems.
Incident handling: Someone has to own a wrong answer, a bad write or a data exposure. The NCSC's guidelines expect a team to monitor the system's behaviour and inputs, and to fix issues quickly when they appear.

The maintenance schedule: task, frequency, owner and effort

This is the schedule to agree before the build team leaves. The owner column names a role, so the table still works when people change jobs.

Maintenance tasks for a production AI system: how often each comes up, who owns it and the effort involved
Dimension	How often	Owner	Effort
Check vendor retirement notices	Monthly, and on every vendor email	Engineering or platform lead	Low: read the deprecation pages and update the model inventory
Migrate to a replacement model	Each retirement, inside the notice window	Engineering lead	A small project: rerun the evaluation, adjust prompts, release with a rollback ready
Rerun the evaluation suite	On every change, plus a scheduled weekly run	Engineering, automated in CI	Low once it is automated
Review a sample of live output	Weekly for high-risk work, monthly otherwise	Process owner	Medium: read the sample and add each failure to the test set
Update prompts, tools and instructions	When a process, policy or connected system changes	Engineering, with the process owner's sign-off	Low to medium, through the normal release path
Monitor cost	Alerts as they fire, and a monthly review	Budget holder, with engineering	Low with a dashboard in place
Review access	Quarterly, and when people join, move or leave	System owner, with IT security	Low: compare the access list with the roles that need it
Handle incidents	As they happen, with a short review after each	On-call engineer and system owner	Varies: the runbook keeps it to a known set of steps
Update libraries and API versions	Monthly	Engineering	Low, with the evaluation suite as the safety net

Four tests that show your team can maintain it

Your team can run the system without the supplier if an engineer of yours can do four things alone. Ask them to show each one on a real change before the contract ends.

Most of this work sits in the engineering layer around the model, made of tools, permissions, checks and logs. Our explainer on the layer of tools and checks around a model describes it. A software engineer who knows your systems can learn it, and a machine learning researcher is not needed.

Change a prompt and see the result: Edit a prompt, run the evaluation suite and read which cases passed or failed.
Switch the model version: Point the system at a new model in a test environment and compare the scores with the current one.
Explain a past decision: Find a case from last month in the logs and show what the system saw, what it did and who approved it.
Turn it off: Switch the capability off without a deploy, and turn it back on.

A handover checklist for when the consultancy leaves

A handover written in the last week records what people remember. One built from the first month records what was decided. Our guide to what an AI build should hand over by day 30 shows how to start the pack early.

Copy this list into the contract or the closing statement of work, and have your own engineer tick each line.

AI system handover checklist

AI system handover: [system name]

Date: [date]
System owner after handover: [name, role]
Engineering owner after handover: [name, role]

1. Code, prompts, tool definitions and test sets are in our repositories: [locations].

2. Infrastructure is in our cloud accounts and defined as code: [locations].

3. Model inventory lists each model ID, where it is called and its vendor retirement date: [location].

4. The evaluation suite runs in our CI on every change, with pass thresholds written down: [location].

5. Dashboards and alerts cover quality, latency, errors and cost, and alert our team: [links].

6. The runbook covers incidents, rollback, switch-off and a model migration: [location].

7. The decision log records why each model, prompt and control was chosen, with dates: [location].

8. Access: the system's service identities and who can change them are listed: [location].

9. A cost baseline per case or per month is agreed: [figure, date].

10. Our engineer has shipped one change alone, end to end: [change, date].

11. Supplier access is removed through our identity provider: [date].

12. Copyright in the deliverables is assigned to us where the contract needs it: [clause].

In-house maintenance or retained support: pros and cons

There are three ways to staff the work after launch. The choice turns on how many AI systems you run and how often the vendors change under them.

Pros and cons of three ways to maintain a production AI system
Dimension	Pros	Cons	Fits when
Your own team	The knowledge stays inside. Changes fit your release cycle and priorities.	Model migrations and incidents compete with other work. A small team can lose the one person who knew the system.	You run several AI systems and have engineers who took part in the build.
Retained support from the builder	The people who built it handle migrations and incidents, and know where each control sits.	An ongoing cost, and your team learns less unless enablement is part of the retainer.	You run one or two systems, or your team is busy elsewhere.
Your owner, with retained help for the heavy jobs	Your team does the routine work and calls in help for model migrations, new tools and reviews.	Needs a clear line between routine work and the jobs that go to the supplier.	You want to own the system while the vendors are still changing fast.

Examples: what maintenance looks like in practice

Three of these are worked illustrations, and the access example comes from our own work. Each is routine once the tests, logs and alerts exist. Without them, a user finds the problem first.

A retiring model: A support assistant calls Claude Sonnet 4.5. The notice arrives on 30 September for a 30 November retirement. The team reruns its evaluation on Claude Sonnet 5.5 and adjusts two prompts whose answers changed. It releases behind a feature flag, with the old model kept as a fallback until the date.
Drift in document intake: An invoice extraction workflow starts sending more cases to the review queue. The weekly sample shows a large supplier changed its layout. The process owner adds ten of the new invoices to the test set, and the fix ships once they pass.
Access that follows people: For a PE-backed firm, a finance question that took around two hours to come back is now answered in the meeting. The connector we built carries each person's existing Looker entitlements. No second permission model was created, so there is no separate access list to keep in step.
Cost creep in an agent: An agent that drafts replies starts running more steps per ticket after a tool change. The spend alert fires, the logs show the loop, and a step limit plus a prompt fix brings cost per ticket back to the baseline.

Where 1AYM fits

If you are about to take over an AI system from a supplier, or need someone to keep one running, bring in 1AYM. Our Enterprise AI Platform & Agentic Workflows builds deliver evaluation and regression suites, audit logging and observability, and runbook and handover documentation. With those in hand, your team can run what we build. Copyright in the deliverables is assigned to you where the contract needs it.

For ongoing help, our Fractional AI Implementation Team is retained, typically two to three days a week under one statement of work. It ends with a written handover. If you already have a scoped job, we can resource it on contract from the collective of associates who work with us, held to the same standard. Embedded Engineers on Contract run from three months, priced per day.

Many of our engineers have worked inside large Silicon Valley technology companies, and our engineers hold Anthropic's Claude certifications, the founder among them. Day rates start at £750 to £1,250, depending on seniority and the project. Our pricing page shows how retained and fixed-scope work are priced. Book a 30-minute call to go through your handover or your next model retirement, or email enquiries@1aym.com.

For engineers: the maintenance setup to ask for

The pieces that turn maintenance from guesswork into routine. Each should be in the repository or the monitoring stack before the build team leaves.

Model inventory: One file listing every model ID the system calls, where it is called, the vendor, the hosting route and the retirement date from the vendor's deprecation page. Anthropic's Usage page exports usage by API key and model, which helps find stray calls.
Pinned IDs: Call pinned model IDs so a model change is a release you test. Anthropic says every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on. On Foundry, set each Standard deployment's upgrade option on purpose, and migrate Provisioned deployments yourself, because Foundry does not upgrade them.
API version pinning: Pin the vendor API version as well as the model, and test a new version in CI before switching. Anthropic's API version of 21 October 2026 removed sampling parameters outright.
Evaluation as regression tests: The test set lives in the repository with thresholds, runs on every change and on a schedule, and gains a case for every failure found in production.
Tracing: One structured record per request or agent step: input reference, model and prompt version, tool calls, tokens, latency, check results and approver. Agents need the full step trace to explain a loop or a bad action.
Cost controls: Spend alerts per workflow and per team, a token ceiling per request and a step limit per agent run.
Release path: Prompt and tool changes ship through the same pull request, review and CI as code, behind a feature flag, with a tested rollback and a switch-off that needs no deploy.

Sources

Frequently asked questions

Can our team maintain AI systems after a consultancy leaves?

Yes, if the handover gives them the code, prompts and tests in your repositories, a runbook, a model inventory and the access to change it all. Before the contract ends, ask one of your engineers to ship a change alone. They should also switch a model version in test and explain a past decision from the logs.

How often do AI models need updating?

Whenever the vendor retires the version you use, and the vendor sets that date. Anthropic gives at least 60 days' notice for publicly released models, and Google Cloud gives at least 45 days once it sets a date. Microsoft's Foundry runs its generally available models on an 18-month lifecycle, or 12 months for Anthropic's and some other publishers' models. Check the deprecation pages monthly.

What happens when an AI model we use is retired?

Requests to it fail. Anthropic says so for its own API, and Microsoft's Foundry returns an error to every request. Some hosted deployments upgrade to a replacement for you, so check how yours is set and test the replacement before the date.

What does it cost to maintain an AI system?

The running cost is model usage, hosting, evaluation runs and the time of the people who check output and handle changes. Set a baseline cost per case at launch and review it monthly. If you bring in 1AYM for the heavier jobs, day rates start at £750 to £1,250, depending on seniority and the project.

Do we need machine learning specialists to maintain an AI system?

Usually not for a system built on vendor models. The work is software engineering, covering tests, releases, logs, access and cost. A software engineer who knows your systems and has the evaluation suite and runbook can do it.

Who can help us maintain an AI system after the build?

1AYM. A model migration or a handover can be a small fixed-scope statement of work. Our Fractional AI Implementation Team is retained, typically two to three days a week. Our production builds ship with runbook and handover documentation. Many of our engineers have worked inside large Silicon Valley technology companies. Book a 30-minute call or email enquiries@1aym.com.