Our test suite passes, but production failures continue. What should we inspect?
Test relevance gapReview whether tests cover real failure modes, configuration, data, permissions, and operating conditions. A passing suite is useful evidence only for the behavior it actually checks.
What to do now
- Analyze recent production failures
- Map them to missing or weak tests
- Add representative verification
- Review release and observability gaps
When to escalate
Escalate when
Failures threaten serious harm, data integrity, or essential service.
Who decides
The technical owner and authorized release decision maker
Example
A failure caused by a production permission difference leads to role-specific integration tests.
Mistake to avoid
Adding more trivial tests to increase the test count.
Prevent a repeat
Review real execution evidence and maintainability alongside feature completion; do not treat an unrun test as passed.
Suggested ApprovedFlow setup
- Tasks
- Documents
- Approvals
Track delivery tasks and attach engineering evidence; keep source control, build execution and technical controls in their actual engineering systems. See the platform
Related cases
- How do we decide which tests to automate first?Automation priority
- The proof of concept works. What remains before production?Prototype-production gap
- The coding assistant says all tests pass, but no test log exists. Can I accept it?Unverified generated engineering claims
- An acceptance test fails because the requirement was wrong. What should happen?Requirement validation failure