Pipeline ignored when red
Engineers merge despite failures because it is probably the tests.
Test automation
How to stabilize automation
Stable automation is not about zero failures - it is about failures that mean something. When every red build might be the test, teams stop trusting the pipeline.
What it looks like on your team
Retry loops in CI
Configs retry three times by default, inflating duration without fixing root cause.
Slow feedback
Long unstable suites delay releases because teams wait for maybe green.
Manual regression alongside automation
Duplicate effort because automated checks are not trusted.
Root causes
-
No flake budget or ownership
Flaky tests accumulate because there is no metric or team accountable for stability.
-
Tests coupled to timing
Animations, lazy loading, and API latency are not modeled in waits.
-
Oversized end-to-end scope
Single tests cover too many steps, increasing blast radius of any failure.
-
Selector strategy absent
Each author picks different locator styles, compounding breakage.
What to do
-
Define stability SLOs
Track flake rate and time-to-fix; treat recurring failures like production bugs.
-
Split long flows
Shorter tests isolate failures and are easier to shard across CI jobs.
-
Standardize waits and selectors
Document patterns; code review tests like product code.
-
Schedule and monitor regularly
Nightly runs with alerts catch drift before release day.
Related reading
FAQ
How stable should UI automation be before gating releases?
Most teams target well under 2% flake on critical paths before blocking merges on automation.
What is the difference between flaky and brittle tests?
Flaky tests pass and fail without code changes; brittle tests fail predictably when the UI changes.
Can scheduling tests improve stability?
Regular runs surface environmental drift early and keep data fresh, especially for long-running suites.
Try QAlity on your hardest flows
Record, run in the cloud, and recover from UI changes with less manual work.