Draft for Tamir's review. Not published.
Find the cause before you change the cadence
Trace each break to its cause: an interface, a report, a data mapping or the upgrade's own change. Build a timeline from approval to failure, and compare the release notes with your procedure to list what neither covered.
Skipping upgrades builds a backlog that makes the next one worse. Read the vendor's release policy and support terms for skipped versions before you slow down.
Set a gate every change must pass
Build a test environment with masked production data and a test that runs before every upgrade, such as a full reconciliation. Require a tested rollback, the vendor's sign-off on the plan and a production window agreed with operations.
Treat AI model versions as dependencies. Pin them, run your evaluation set on each new version, and fix prompts and parsing before switching. Test a second provider on the same set so your fallback has known quality.
Depending on your seat
If you buy a product with an AI model inside, define which changes require notice, evidence and renewed acceptance. Check whether you can suspend a change or return to the previous configuration.
If an upgrade already caused a loss, decide the claim with counsel on the timeline evidence once the cause is clear. Report to your CFO or board on the cause and the test change rather than on how often you upgrade.
What to check before you decide
- Trace each break to an interface, a report, a data mapping or the upgrade's own change.
- Check whether the pre-upgrade test covered those interfaces and reports with real volumes.
- Build a test environment with masked production data and a test that runs before every upgrade.
- Define an upgrade gate: tested rollback, vendor sign-off and an agreed production window.
- Pin model versions explicitly, track deprecation dates and run the full evaluation set before any switch.
- Identify every component a supplier may change remotely, and require version records tied to accepted evaluation evidence.
- Read the support terms for documentation, testing and skipped versions.
Questions people ask
A plant system upgrade caused a shutdown and vendor and site blame each other, how do we find the cause and change how upgrades are approved?
Reconstruct the upgrade step by step against the release notes and the site procedure, because the gap between the two usually shows where it broke. Then fix the approval process so no upgrade runs without a tested rollback. It depends on what the vendor documented and on whether the site had a test environment.
A core system upgrade broke reconciliations for weeks, should we upgrade less often or change how we test?
The breaks usually come from untested interfaces and reports rather than from the upgrade itself, so the fix is in what you test and with which data. Smaller, more frequent upgrades with a full reconciliation test on real data beat rare large ones. It depends on your vendor's release cadence and on whether you can build a test environment with production-like data.
Model provider update made our product worse and the old version is being deprecated, how do we manage provider changes?
Treat provider versions like any dependency: pin them, test every new version against your evaluation set before switching, and keep prompts and parsing adaptable. Build the evaluation so a version change is a measured decision rather than a surprise. It depends on how large your evaluation set is and on whether a second provider can take the load.
What control should a hospital retain over vendor AI model changes?
Define which changes require notice, evidence, and renewed acceptance. The boundary depends on intended use, observable behavior, and whether the previous configuration remains recoverable.
How I can help with this decision
- Ask or talk (Free)
- I give my view on where upgrade and model-change breaks usually come from, and the test that would have caught yours.
- Review (Pay if it was worth it)
- I write an independent post-mortem on the failed upgrade or model change and a review of your testing, version handling and vendor terms. I recommend changes to the approval process and the contract.
- Retain (When it makes sense)
- I stay close through the next upgrades or provider releases to review test evidence before each go-live.