Introduction
A test fails. You rerun it. It passes. Nothing changed. If that sounds familiar, you have flaky tests. They are one of the most expensive problems in software delivery, not because any single failure costs much, but because they slowly train your team to ignore red builds. Once developers start hitting "rerun" instead of investigating, your test suite stops doing its job.
This guide covers what makes tests flaky, the six root causes behind most flakiness, how to detect flaky tests systematically, and how to stop them from entering your pipeline in the first place.
What Is a Flaky Test
A flaky test is a test that produces different results on different runs without any change to the code under test. Same commit, same test, different outcome.
The impact goes beyond wasted rerun time. Flaky tests create three compounding problems:
- Lost trust: When failures might be noise, developers stop treating them as signals. Real bugs slip through because someone assumed the failure was "just that flaky test again."
- Slower delivery: Reruns, investigations, and blocked merges add friction to every deployment. A pipeline that needs two or three attempts to go green doubles or triples your feedback loop.
- Hidden debt: Flakiness usually points to a real weakness, either in the test or in the product. Ignoring it means the underlying race condition or leaky resource stays in your codebase.
What Makes a Test "Flaky"?
The defining trait is non-determinism. A healthy test is a pure function of the code it tests; given the same inputs, it always returns the same verdict. A flaky test has hidden inputs, things like system time, network latency, execution order, or leftover state from a previous test. When those hidden inputs shift, the result flips.
This is why flaky tests are so hard to reproduce locally. Your laptop and your CI runner differ in CPU contention, network conditions, parallelism, and timing. The hidden input that flips the test on CI may never occur on your machine.
The problem exists at every scale. Google has published research showing that a meaningful share of its test suite exhibits some level of flakiness, and that flaky failures account for a large portion of test-to-fail transitions in its CI systems. Microsoft, Mozilla, and GitHub have all written publicly about dedicated tooling and teams built specifically to manage flakiness. If companies with that much engineering investment still fight this problem, no team should expect to avoid it entirely. The goal is management, not perfection.
The 6 Root Causes of Flaky Tests
Almost every flaky test traces back to one of six categories. Knowing them speeds up diagnosis considerably because you can check the likely suspects in order rather than guessing.
1. Timing and Async Issues
This is the most common category, especially in UI and integration tests.
Race conditions: The test asserts on a result before the operation producing it has finished. Under normal load, the operation wins the race. Under CI load, the assertion wins, and the test fails.
Fixed waits: sleep(3000) is a guess about how long something takes. When the environment is slow, three seconds is not enough, and the test fails. When it is fast, you burn three seconds for nothing. Fixed waits make tests both flaky and slow, which is an impressive combination.
Async/await problems: A missing await causes the test to continue before a promise resolves. Sometimes the promise resolves fast enough anyway, and the test passes. Sometimes it does not. These bugs are easy to write and hard to spot in review because the code looks almost correct.
The fix in all three cases is the same principle: wait for events, not for time. Wait for the element to be visible, the request to complete, the state to change.
2. Shared State and Test Dependencies
Test order dependency: Test B passes when it runs after test A because A leaves behind data B silently relies on. Run B alone, or run the suite in parallel, and B fails. Any test that cannot pass in isolation is a flake waiting for a scheduling change.
Shared resources: Two tests writing to the same file, port, or global variable will collide eventually, especially once you enable parallel execution.
Database state conflicts: Tests that assume specific row counts, IDs, or empty tables break as soon as another test, or a previous failed run, leaves the database in an unexpected state. Auto-incrementing IDs are a classic trap here.
3. Environment Inconsistencies
CI vs local differences: Different OS, browser version, locale, screen resolution, or installed fonts can all change behavior. "Works on my machine" is often literally true and completely unhelpful.
Resource starvation: CI runners are usually shared and often underpowered compared to developer machines. A test tuned against a fast laptop can time out on a busy runner.
Container limitations: Memory limits, missing system dependencies, and headless browser quirks inside containers all produce failures that never appear locally.
4. External Dependencies
Any test that calls a real third-party API inherits that API's reliability. Rate limits, maintenance windows, network timeouts, and DNS hiccups all become your test failures. The test is technically doing its job, reporting that something failed, but it is reporting on infrastructure you do not control and cannot fix.
The general rule: unit and integration tests should mock external services. Keep a small, separate set of contract or smoke tests that hit real dependencies, and do not let those block merges.
5. Resource Leaks
Leaks are sneaky because the leaking test usually passes. The victim is a later test that fails when memory runs out, the connection pool is exhausted, or the OS runs out of file handles. The failure appears in a test that has nothing wrong with it, which sends the investigation in the wrong direction.
Symptoms to watch for: failures that only occur in long test runs, failures that move around between runs, and suites that get slower the longer they run.
6. Non-Deterministic Elements
Random values: Unseeded random data means every run tests something slightly different. Occasionally the random input hits an edge case, or violates a validation rule, and the test fails. Seed your randomness so failures are reproducible.
Time zone issues: A test that passes in UTC and fails in the runner's local time zone, or vice versa, is comparing dates without controlling the zone.
Date-sensitive logic: Tests that break at midnight, on the 31st, at month boundaries, or on February 29 are all real and all common. Freeze the clock in tests instead of using the actual current time.
How to Detect Flaky Tests: A 4-Pillar Framework
You cannot fix what you have not identified, and gut feeling is a poor identification method. Teams consistently underestimate how many flaky tests they have because each individual developer only sees a slice of the failures. Systematic detection rests on four pillars.
1. Automated Detection Methods
Historical pass/fail rate analysis: Track every test's result across every run. A test that fails 3% of the time on unchanged code is flaky by definition. This is the cheapest signal you can collect because the data already exists in your CI logs.
Rerun-based detection: If a test fails and then passes on immediate rerun with no code change, flag it. This catches flakes at the moment they occur rather than in retrospective analysis. The caveat: reruns hide flakiness if you only record the final result. Record every attempt.
Statistical flip-rate analysis: Count how often a test transitions between pass and fail across consecutive runs of the same commit or branch. Genuine regressions fail consistently after a specific change. Flaky tests flip back and forth without correlation to code changes.
Setting practical thresholds: A useful starting point is the 2% rule: any test that fails more than 2% of runs on stable code gets flagged for investigation. Tighten the threshold as your suite improves. Whatever number you pick, the point is having an explicit, agreed threshold instead of arguing about each test individually.
2. CI/CD Integration for Detection
Detection works best when it is built into the pipeline rather than run as a periodic audit.
Track per-test metrics, not just per-build results. A build that passes 99% of the time can still contain a test that flakes constantly, hidden behind retries.
Cross-run analysis compares results for the same test across branches, commits, and runners. A test failing on one runner type but not another points at environment, not code.
Environment correlation means recording metadata with every result: runner ID, parallelism level, time of day, browser version. Flakiness that clusters around a specific variable hands you the diagnosis.
Failure pattern recognition groups failures by error message and stack trace. Fifty failures with the same timeout signature are one problem, not fifty.
3. Manual Identification Techniques
Automation catches most flakes, but people catch them earlier.
Developer reports: Make it trivial to flag a test as suspicious, ideally one click or one command. The developer who just hit a weird failure has context that no dashboard has. If reporting takes more than thirty seconds, it will not happen.
Code review red flags: Reviewers should treat these as flakiness smells: hard-coded sleeps, assertions on timing, dependence on test execution order, real network calls, unseeded randomness, and use of the current date or time.
Audit-based reviews: Once or twice a year, review your slowest and oldest tests. Flakiness concentrates in tests nobody has touched in years, written against assumptions that no longer hold.
Prioritization: Not all flakes deserve equal attention. Investigate first the tests that block merges, flake most often, and cover critical paths. A flaky test in a nightly optional suite can wait.
4. Monitoring and Observability
Detection tells you a test is flaky. Monitoring tells you whether the problem is growing.
Dashboards and trend tracking: A visible flakiness rate, suite-wide and per-team, keeps the problem honest. Trends matter more than snapshots. A suite going from 1% to 3% flaky over a quarter is a fire alarm even though both numbers look small.
Alerting thresholds: Alert when the suite-wide flake rate crosses your agreed limit, or when a previously stable test starts flipping. Route the alert to the team that owns the test, not to a channel everyone mutes.
Correlating spikes with changes: A sudden flakiness spike after a dependency upgrade, CI runner change, or parallelism increase usually is not a coincidence. Keeping deployment and infrastructure events on the same timeline as test results makes these correlations obvious.
Test metadata over time: Ownership, framework, last-modified date, and average duration all help surface patterns. If 70% of your flakes live in one legacy Selenium package, you have a migration argument, not just a bug list.
Proven Strategies to Fix Flaky Tests
Detection techniques let you know how many flaky tests you have. This section is about working through them.
1. The Quarantine Approach
Quarantine means moving a known-flaky test out of the blocking pipeline while keeping it running and tracked. It is the single highest-leverage practice for teams drowning in flakes, because it immediately restores trust in the main suite.
The rules that make quarantine work instead of becoming a graveyard:
- Quarantined tests still run on every build. You keep collecting data; they just cannot block a merge.
- Every quarantined test gets an owner and a deadline. Two weeks is a common limit. Miss the deadline and the test is either fixed, rewritten, or deleted with a documented decision.
- Cap the quarantine size. If the queue exceeds the cap, fixing flakes takes priority over new feature work until it is back under the limit.
2. Framework-Specific Solutions
Playwright: Rely on its auto-waiting and web-first assertions like toBeVisible() instead of manual waits. Use test.describe.configure({ mode: 'serial' }) only when order genuinely matters, and prefer isolated browser contexts per test. Turn on trace collection for retries so every flake comes with a full recording.
Cypress: Let its built-in retry-ability do the waiting. The most common Cypress flake source is cy.wait(ms) with a fixed number; replace it with intercepts and cy.wait('@alias') on actual network requests. Avoid conditional testing based on DOM state, which is almost always a race condition in disguise.
Selenium: Most Selenium flakiness comes from raw Thread.sleep calls and stale element references. Use explicit waits (WebDriverWait with expected conditions) everywhere, relocate elements after page changes, and pin browser and driver versions in CI so upgrades happen deliberately.
Jest and pytest: Enforce isolation: reset modules and mocks between tests, use fresh fixtures instead of module-level state, and seed randomness. Both ecosystems have plugins to detect order dependence by shuffling execution (pytest-randomly, Jest's --randomize). Run them regularly, not just once.
3. Root Cause Resolution Techniques
When a flake needs an actual fix, a repeatable workflow beats improvisation.
Reproduce first: Run the test in a loop, locally or in CI, until it fails. A hundred runs is a reasonable start. If it will not fail in isolation, run it alongside its full suite, in parallel, on a constrained machine. Matching CI conditions matters more than run count.
Collect artifacts on every failure: Screenshots, videos, browser console output, network logs, and application logs, captured automatically at failure time. Flakes are too rare to debug live; the artifacts are usually all you get.
Investigate systematically: Walk the six root causes in order of likelihood: timing first, then shared state, then environment. Compare metadata from failing runs against passing ones and look for the variable that differs.
Apply known fix patterns: Most fixes fall into a handful of shapes: replace a fixed wait with an event wait, isolate state with fresh fixtures, mock an external call, seed a random value, or freeze the clock. Document which pattern fixed which test. Your next flake probably matches a previous one.
Flaky Test Prevention Methods
Fixing flakes is necessary. Preventing them is cheaper. Here’s how to prevent flaky tests from reaching CI.
1. Code Review Checklists
A short, enforced checklist catches most flaky patterns before merge. Here are the essentials:
- No fixed sleeps. Waits must target a condition or event.
- Every test passes in isolation and in random order.
- No real network calls to services you do not control.
- Randomness is seeded; time is frozen or injected.
- No assertions on incidental details like element counts that depend on unrelated data.
Write the checklist down and link it in your PR template. Team agreements only work when they are visible, and "we all know not to do that" is not a policy.
One newer item deserves explicit mention: AI-generated test validation. Code assistants produce tests quickly, and they reproduce every anti-pattern in their training data, fixed waits included. AI-generated tests should get the same review scrutiny as human-written ones, plus a stability check: run them 20 to 50 times before merging, not once.
2. CI Configuration Best Practices
Resource allocation: Underpowered runners manufacture timing flakes. If your flake rate drops when you double runner resources, the tests were never the whole problem.
Test sharding: Split the suite across parallel runners, but shard by consistent grouping rather than randomly per run, so failures are comparable across builds. Sharding also exposes hidden order dependencies early, which is painful once and valuable forever.
Retry policies: Automatic retries are acceptable only if every attempt is recorded and flagged. A retry that silently converts a failure into a pass is how flakiness becomes invisible. Retry once, log it, and feed the data into your detection pipeline.
Smoke tests: Run a small, fast, ultra-stable subset first. If the smoke suite fails, skip the rest. This protects the full suite's signal and gives developers feedback in minutes instead of an hour.
3. Writing Resilient Tests
Design for diagnosability: A test that fails with "expected true, got false" wastes an investigation. Write detailed tests with messages, log context, and capture artifacts, so a failure explains itself.
Isolate properly: Each test creates what it needs and cleans up what it made. Unique identifiers per run, fresh database transactions rolled back after each test, and no reliance on anything another test created.
Wait on events: Worth repeating because it fixes the largest category of flakes: wait for the condition you actually care about, with a generous timeout, rather than guessing a duration.
Mock deliberately: Mock external services at the boundary, keep the mocks in sync with real contracts, and maintain a small separate suite that verifies the real integrations without blocking merges.
How TestFiesta Helps With Flaky Test Management
Every detection method, fix strategy, and prevention method we discussed in this guide can be built by hand with CI logs, scripts, and discipline. TestFiesta packages it into one workflow, so your team spends time fixing tests instead of building detection infrastructure.
TestFiesta tracks per-test results across every run, flags tests whose failure patterns match flakiness rather than regression, and correlates failures with environment metadata to point you toward the root cause. Quarantine workflows come with the ownership and deadline mechanics built in, so flagged tests do not disappear into a backlog. It works across Playwright, Cypress, Selenium, Jest, and pytest, and plugs into your existing CI pipeline.
Ready to see your suite's actual flake rate?
Start a free TestFiesta trial and get visibility into your test reliability from your very first pipeline run.
Sign Up for a Free Trial
Frequently Asked Questions
What's the difference between a flaky test and an intermittent bug?
A flaky test fails inconsistently because of a problem in the test or its environment; the product is fine. An intermittent bug is a real product defect that only surfaces under certain conditions, like a race condition in production code. The distinction matters because the fix lives in different places, and the diagnosis is the same in both cases: reproduce the failure and find the hidden variable. Never assume a flapping test is "just flaky" until you have confirmed the product is not the cause.
How many flaky tests is too many for a test suite?
As a working threshold, keep your suite-wide flake rate under 1% of test runs, and flag any individual test failing more than 2% of runs on stable code. More important than the exact number is the trend. A suite at 0.5% and climbing is in worse shape than one at 1% and falling. If more than roughly 5% of your builds need a rerun to go green, flakiness is actively slowing your delivery and deserves dedicated time.
Should I delete or fix flaky tests?
You should neither delete nor try to fix your flaky test as the first step. Instead, quarantine first. Remove the test from the blocking pipeline, keep running it, and set a deadline. Once you’re at the deadline, decide what you want to do based on value. If the test covers a critical path, fix it. If it duplicates coverage that exists elsewhere, or tests behavior nobody can explain, delete it and document why. Deleting a low-value flaky test is a legitimate engineering decision. Letting it rot in quarantine forever is not.
Can AI really help identify flaky test root causes?
Yes, AI can help with identification of flaky test root causes, but within limits. Pattern recognition across large volumes of test results is exactly what machine learning is good at: clustering failures by stack trace, spotting correlations between failures and environment variables, and matching a new flake against previously diagnosed ones. What AI cannot do is understand your system's intent, so treat its output as a strong hypothesis that a developer confirms, not a verdict.
