All articles

Flaky Tests in 2026: What Causes Them, How to Detect Them, and When to Quarantine

What makes tests flaky, what they really cost, how to detect them reliably, when retries help or hide bugs, and how to run a quarantine that gets tests fixed.

A flaky test passes and fails on the same code. One of them is an annoyance. A few hundred, in a monorepo shared by dozens of teams, turn every red build into a judgment call: real regression, or rerun and hope? Atlassian found flaky tests behind as much as 21% of master build failures in its Jira Frontend repository, and estimates that the reruns they force on the Jira backend repository waste over 150,000 hours of developer time each year (Atlassian Engineering, December 2025). The cost is not only hours. Once engineers learn that red often means nothing, they stop trusting the suite, and the suite stops protecting production.

Below are the questions engineering leaders most often ask about flaky tests, answered with recent research and the practices of teams that run tests at very large scale.

Key takeaways

  • A flaky test is one whose result changes without a code change, and its real damage is lost trust in the pipeline.
  • Most flakiness comes from networking, external dependencies, time, shared state and concurrency, and flaky tests tend to fail in clusters that share one cause.
  • Detection works best when failures are rerun under varied conditions and tests are scored on their history, not judged on a single retry.
  • Automatic retries are cheap and useful, but a retry that silently turns red into green hides real bugs.
  • Quarantine only works with evidence, an owner, a due date and a defined way back into the suite.
Illustration of an engineer reviewing a list of passed checks on a laptop next to a gear with a rerun loop, standing for automated test runs in a CI pipeline

What is a flaky test, and why does it matter?

A flaky test is one that can both pass and fail against the same code and configuration. The cause sits in the test, in the code under test or in the environment, not in the change being reviewed. That is what makes flakiness expensive: every failure has to be investigated before anyone knows whether it means anything.

DORA's guidance on test automation sets the bar. The goal it describes is that test failures always indicate a real defect in the product, and it is direct about the alternative:

"In particular, don't tolerate flaky tests."

DORA also suggests tracking how many test failures represent a real defect and how many come from poorly written tests. That ratio says more about a suite than its pass rate.

How much do flaky tests cost?

The useful numbers come from teams that measured. A five-year industrial case study of a commercial project with about 30 developers and roughly one million lines of code found that flaky tests consumed at least 2.5% of productive developer time, mostly spent investigating suspicious failures and repairing tests (Leinen et al., ICST 2024). Atlassian's figures show what happens when the same problem compounds across a large monorepo:

21%

of master build failures in the Jira Frontend repository caused by flaky tests

Atlassian, 2025

150,000+

developer hours a year lost to flaky reruns in the Jira backend repository

Atlassian, 2025

2.5%

of productive developer time, at minimum, in a 30-developer project

Leinen et al., 2024

Compute is rarely the cost that matters. Attention is.

What causes flaky tests?

A 2025 study of 10,000 test suite runs across 24 open-source Java projects analyzed 810 flaky tests and found that 75% of them fail as part of a cluster, with clusters averaging 13.5 tests that fail together because they share a root cause (Parry et al., EASE 2025). That changes the repair strategy: fix the shared cause, and a dozen tests stop flaking at once. The usual suspects:

  • Networking. The most common theme in the study: timeouts, ports already in use, services not ready when the test starts.

  • External dependencies. The second most common: third-party services, package mirrors and images that change or slow down underneath the test.

  • Time. Fixed sleeps, timeouts tuned on a fast laptop, and assumptions about the clock or time zones.

  • Shared state and order. Tests that pass alone but fail after another test leaves data, files or global state behind.

  • Concurrency and randomness. Race conditions, unordered collections and unseeded random data.

How do you detect flaky tests?

The naive method is to retry a failed test once and call it flaky if it passes. It works, but it misses a lot. GitHub reported that its earlier approaches, comparing builds of the same tree and retrying within a build, identified only 25% of flaky failures. Rerunning each failure in three different ways raised that to 90%, and cut the share of commits with a flaky red build from about 1 in 11 to 1 in 200 (GitHub Engineering, 2020). The main methods compared:

MethodWhat it catchesLimits
Retry in the same buildFailures that clear on an immediate rerun, often timing and races.Misses flakiness that needs another host, clock or order, and a lucky pass can hide a real bug.
Rerun under varied conditionsGitHub's three modes: same process for randomness and races, clock shifted forward for time assumptions, another host for order and shared state.More compute per failure, and CI that can control the clock and test placement.
Historical scoringTests that flip across many builds and branches. Atlassian scores each test between 0 and 1 with Bayesian inference over a moving window of runs.Needs stored results and volume, and is slow to judge a new test.

At scale, teams combine them. Atlassian's Flakinator, which handles more than 350 million test executions a day, retries in the build and uses the historical score to decide which tests to act on. Bitbucket now detects flaky tests by default for Pipelines users on its Standard and Premium plans, with optional auto-quarantine (Atlassian, March 2026).

Should you retry flaky tests?

Yes, as a measurement and as a stopgap. The industrial case study found automatic reruns very cheap and manual investigation of a failed pipeline far more expensive. But a retry policy has to record every flip. A test that fails, passes on retry and is reported as plain green has been turned from a signal into noise.

Never let a retry erase the failure

Keep the first failure in the test report, count it toward the test's flakiness score, and alert when a test needs retries more often than your threshold allows. A real regression that fails only some of the time, such as a race condition in production code, looks exactly like a flaky test until someone investigates it.

How should you quarantine flaky tests?

Quarantine means the test keeps running but no longer blocks merges or deploys. It is the right move for a test with a high flakiness score, and the wrong one if quarantine becomes a graveyard. The rules that separate the two, drawn largely from how Atlassian runs it:

  • Quarantine on evidence, not on one failure. Use the flakiness score or a minimum number of flips across builds, so a real bug is not quarantined the first time it appears.
  • Keep running quarantined tests. Atlassian collects their results from branch builds, scheduled jobs or a dedicated quarantine pipeline, so the data needed to fix and release them keeps coming.
  • Assign an owner and a due date. Atlassian opens a Jira ticket for the owning team, with a pre-decided due date, as soon as a test is detected as flaky.
  • Define the way back. A test that stays healthy for a configured period leaves quarantine automatically.

How do you fix flaky tests for good?

Detection and quarantine buy time. Fixing is what gives the suite its value back. GitHub found that only 0.4% of its flaky tests failed 100 times or more, so it focused its effort there and routed each issue to the people who had most recently changed the test or the related code. Its principle: "the only person bothered by a flaky test is the person who wrote it." Put together, a repeatable loop looks like this:

  1. Rank by impact

    Score each flaky test by its failures and the branches, developers and deploys it blocked. Work the top of the list.

  2. Route to the right owner

    Assign it automatically to whoever last changed the test or the code it covers.

  3. Fix the shared cause first

    Because flaky tests cluster, look for the common dependency: one test container, one mock server, one fixed sleep.

  4. Reproduce, then fix the cause

    Rerun with the clock shifted, in another order or on another host until it fails on demand. A longer timeout is not a fix.

  5. Delete what is not worth fixing

    DORA's guidance is that an expensive, unreliable suite can be pruned. A test nobody trusts protects nothing.

Where to start

Measure first: the share of red builds that turn green on rerun without a code change, and the number of tests that flip in a month. Then turn on retries that record every flip, quarantine with owners and dates, and fix the biggest clusters. For regulated teams the payoff is not only speed. A pipeline whose red builds always mean something is the evidence that your test gates really control what reaches production.

Sources

  1. Taming Test Flakiness: How We Built a Scalable Tool to Detect and Manage Flaky Tests - Nitish Malik, Atlassian Engineering, December 8, 2025
  2. Introducing Flaky Test Detection in Bitbucket Tests - Rajkumar Singh, Atlassian, March 26, 2026
  3. Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures - Parry, Kapfhammer, Hilton, McMinn, EASE 2025, arXiv, April 23, 2025
  4. Cost of Flaky Tests in Continuous Integration: An Industrial Case Study - Leinen, Elsner, Pretschner, Stahlbauer, Sailer, Jürgens, IEEE ICST 2024, May 2024
  5. Capabilities: Test automation - DORA, last updated July 17, 2025
  6. Reducing flaky builds by 18x - Jordan Raine, The GitHub Blog, December 16, 2020, updated December 20, 2021

Where to start

Start with your last release

The free Release Check shows where your last release spent its time and what that costs a year, in under 5 minutes and without a call.

Keep reading