The model changed and our AI workflow became less reliable. What do we do?
Uncontrolled model-version changeTreat a model change like a dependency change with an evaluation and release decision. Compare representative outputs and failure rates against the prior version, review material regressions, and use an approved fallback where available. Keep version and configuration records for investigation.
What to do now
- Identify the changed model and settings
- Run representative regression evaluations
- Assess material quality and safety differences
- Approve rollout or use the controlled fallback
When to escalate
A regression compromises important decisions or user safety.
The AI service owner and relevant authorized reviewer
Example
A replacement model handles summaries well but fails the team's structured approval extraction tests.
Mistake to avoid
Assuming a newer model is automatically better for every workflow.
Prevent a repeat
Re-evaluate after changes to models, prompts, tools or source data; keep important decisions accountable to people.
Suggested ApprovedFlow setup
- Tasks
- Documents
- Approvals
Track evaluation and review tasks, and keep approved evidence with the work; do not claim the platform provides an unverified AI agent or automated diagnosis. See the platform
Related cases
- Can I use an AI estimate as the project commitment?Unvalidated model-generated estimate
- Should every project workflow become an AI agent?Unnecessary autonomy
- A shared API changes without notice. What should happen?Uncontrolled interface change
- Our acceptance criteria change after delivery. What should we do?Moving acceptance boundary