Test Automation Interview Questions: Flaky Test Diagnosis

This is the defining characteristic of a flaky test: non-deterministic behavior where the same code under test and the same test code yield different outcomes across runs. The statement says exactly this, so it is true. The instability is typically caused by factors external to the code under test, such as timing dependencies, shared mutable state, or reliance on unordered operations.

Hard-coded sleeps make results depend on machine load and scheduling, shared mutable state makes outcomes depend on the order tests run in, and live external services add network and availability variability. Pinning a seeded random generator does the opposite: it makes inputs deterministic and so reduces flakiness rather than causing it.

Flaky tests are non-deterministic: they pass or fail across runs without a code change. The two most frequently cited causes are shared mutable state (including test-order coupling, shared fixtures, or leftover data from earlier tests) and timing or synchronization problems (asserting before an asynchronous or UI action has settled). Either pair of these answers names two real, recurring root causes, which is why the question is open-ended and graded by keywords rather than a single keyed choice.

A flaky test is defined by its non-deterministic behavior: it passes and fails for the same code and inputs, with no deterministic cause traceable to the code under test. The term 'flaky' is the standard name for this property of a test.

The term 'flaky' describes a test whose pass/fail result is not deterministic for the same code and inputs. Such a test is named a flaky test, and the blank is filled with that single word.

The errors come from the test acting before the element reaches the state it needs, which is a race between the test and the application. Explicit, condition-based waits, such as WebDriverWait with expected conditions for clickable, visible, or stale elements, block the test until the application reaches that state, so the same step succeeds regardless of machine speed or load. A fixed sleep guesses a duration and is fragile in both directions, and a retry hides the flakiness without removing the race. Pinning hardware does not fix the timing dependency and is not a real remedy.

A test that passes locally but fails intermittently in CI is usually caused by environmental non-determinism rather than a logic error. Shared mutable state lets one test's side effects change another test's inputs, and CI's different ordering or parallel execution exposes dependencies that a sequential local run hides. Timing and race conditions occur when an assertion runs before an asynchronous operation such as a network call, render, or database write completes; slower, more loaded CI runners widen that window. Identifying which family is responsible is the first step toward a deterministic fix.

Once the test passes alone but fails in the full suite, the cause is order-dependence from shared mutable state left by an earlier test. The procedure is to run the failing test right after each preceding test one at a time, or to bisect the list by repeatedly running the failing test after each half and keeping the half that still reproduces the failure, until a single predecessor is isolated. A good answer also confirms the culprit by running that predecessor directly before the failing test and identifying the shared state it leaves behind.

A test-ordering dependency is confirmed when its outcome changes with the set of tests executed before it, and when the failure can be reproduced deterministically by a specific predecessor sequence. Randomizing the order across repeated runs exposes whether failures track particular predecessors, and bisecting that predecessor list isolates the minimal set that reproduces the failure. Resetting or inspecting the shared in-process state those predecessors modify then verifies the mechanism. The fresh database container and absence of network calls rule out external sources of flakiness, leaving execution order and shared state as the variables under test.

The three-step approach mirrors standard flaky-test diagnosis. First, running the test alone establishes whether it is inherently unstable; since it never fails in isolation, the cause is external to the test body. Second, shuffling the execution order across runs tests the ordering-dependency hypothesis: if failures correlate with specific predecessors, the flakiness is caused by state those tests leave behind. Third, reviewing the code for missing teardown identifies the specific shared state (DOM, cookies, localStorage, page location) that leaks between tests, which is what needs to be reset.

The defining symptom, passing in isolation but failing in the suite, is the classic signature of a test-order dependency. An earlier test leaves shared mutable state (a database row, a global variable, a file, a singleton) in a condition the flaky test implicitly relies on. When the test runs alone that state is absent or default, so it passes; when a specific preceding test runs first, it pollutes the state and the test fails. A missing explicit wait is ruled out by the stated explicit-synchronization constraint, external API variability is ruled out by mocking, and floating-point errors are deterministic and would fail in isolation too, so none of those depend on interactions with other tests.

The two causes have different triggers, so each needs its own controlled experiment. Order dependency is isolated by single-threaded runs with randomized order and a different seed per iteration; holding concurrency at one removes parallel load as a variable, so failures that cluster around specific predecessor tests reveal shared mutable state. Resource contention is isolated by varying the thread count while keeping the order fixed; holding order constant removes sequence as a variable, so failures that rise with concurrency and coincide with CPU, memory, or connection-pool exhaustion reveal contention. Since each experiment changes exactly one factor, the pattern of results in each identifies which cause is responsible.

The symptom (passes alone, flakes only in the full sequential suite) points to two families of cause: state left behind by earlier tests, and timing-dependent bugs that appear under full-suite load. The diagnostic that separates them is per-test state isolation (fresh database or separate process) run in the same order, because it removes cross-test state while keeping the suite's execution pattern. If the flake vanishes, pollution is the likely cause; if it persists, timing or async is more likely. A full reset between tests is not a clean test, since it alters timing and can hide a race and produce a false conclusion.

Three signal-to-cause mappings are correct. First-attempt failures that pass on retry, with no parallelism correlation, point to warm-up or resource-initialization race conditions in the test environment — the environment needs time to stabilize after initial resource allocation. Failure rates scaling with the number of concurrent test processes indicate contention over shared mutable state or implicit ordering dependencies between tests, because more parallelism increases the probability of interleaving. A constant failure rate invariant to parallelism, time of day, and execution duration points to genuine nondeterminism inside the product code itself, such as an application-level race condition, because no external environmental variable explains the variance. A test that fails 100% of the time on a specific OS/browser combination and never elsewhere is a deterministic compatibility defect, not a flaky test; quarantining and auto-retrying it masks a real bug rather than addressing intermittent nondeterminism, so that mapping is invalid.

Three signal-to-cause mappings are correct. First-attempt failures that pass on retry, with no parallelism correlation, point to warm-up or resource-initialization race conditions in the test environment — the environment needs time to stabilize after initial resource allocation. Failure rates scaling with the number of concurrent test processes indicate contention over shared mutable state or implicit ordering dependencies between tests, because more parallelism increases the probability of interleaving. A constant failure rate invariant to parallelism, time of day, and execution duration points to genuine nondeterminism inside the product code itself, such as an application-level race condition, because no external environmental variable explains the variance. A test that fails 100% of the time on a specific OS/browser combination and never elsewhere is a deterministic compatibility defect, not a flaky test; quarantining and auto-retrying it masks a real bug rather than addressing intermittent nondeterminism, so that mapping is invalid.

Timestamped logs at the trigger, the job's commit, and the assertion expose the real event ordering in the failing CI runs. If the assertion's log precedes the commit's log on a failing run, the test read the row before the async write landed, which directly confirms the race. A retry loop only hides the symptom and does not reveal ordering. Snapshotting updated_at is unreliable because the job's own write overwrites that field, so it yields no evidence about whether the write beat the assertion. Swapping in a synchronous mock changes the system under test and cannot confirm a race in the real database path.

A sound answer reproduces the flake many times in the failing environment to measure its rate and failure mode, collects at least two signals (such as the stack trace or error type and the preceding test execution order), names a structural anti-pattern such as fixed sleeps, shared mutable state without teardown, or uncontrolled external dependencies, and closes with a minimal reproduction that makes the failure deterministic, which proves causation rather than correlation.

The diagnostic pattern (passes alone, passes single-worker, passes locally, flakes only with several workers) isolates the variable to concurrent worker processes. Nothing internal to the failing test changes between passing and failing runs, and it has no external dependencies besides the file. The failing test reads a fixed path that a separate test creates and deletes in its hooks, so when a sibling worker runs that other test at the same moment, the reader can find the file missing or in an unexpected state. The answer is shared filesystem state at a fixed path accessed by parallel workers, i.e. a race between tests that are not independent. Isolating the path per worker or per test removes the race.

A strong answer names concrete diagnostic techniques that expose the cause of non-determinism: isolating the test to detect order dependence or state leaks, running serially with --runInBand to rule out parallel-worker interference, looping the test to reproduce the failure, and using git bisect to locate the introducing commit. It must also explain that retries or larger timeouts are not diagnostics because they mask the symptom, keep the underlying non-determinism in the codebase, and do not identify why the test fails. The question asks for an open description and justification, so no single option is correct and it stays a short-answer question graded by keywords and the rubric in the sample answer.

The question asks for concrete diagnostic techniques for flaky tests, so a correct answer must name at least one method that actively surfaces non-determinism, such as running the test many times in isolation to reproduce timing-dependent failures, or randomizing test order to expose state pollution between tests. Each technique should also state the failure mode it reveals: repeated isolated runs point to races or timing dependencies, while shuffled ordering points to shared mutable state or missing cleanup. Verbose logging and disabling CI retries are complementary because they capture the interleaving behind the flake and confirm whether the failure is genuinely transient. The sample answer covers repeated isolated execution, randomized ordering, and logging, each paired with the specific failure mode it reveals.

The evidence points to shared-connection state pollution. In 48 of 50 failures, Test T was the first test to receive a connection that another test had recycled while still holding an open, uncommitted transaction. Test T assumes autocommit mode and never verifies or resets connection state, so it inherits the dirty state and hits the 'connection is in a transaction' error. This also explains why it passes in isolation, where it always gets a fresh connection, and why it passes in CI when preceded by tests that roll back properly. A race in Test T's own teardown would reproduce in isolation, which it does not, and pool exhaustion or a short timeout would not produce a transaction-state error.

This scenario is a textbook test-isolation violation. The test does not control or reset its initial conditions — specifically the transaction state of its database connection — and is therefore vulnerable to pollution from other tests that share the same connection pool. Proper test isolation requires each test to set up and tear down its own state defensively, including verifying connection state, rather than assuming clean infrastructure.

The variable that changes between the two runs is the presence of other tests, so the flakiness comes from the execution context rather than from the production code's logic. This points to shared mutable state (database rows, global variables, file system artifacts, or environment variables) left dirty by a prior test, which is a breach of test isolation (independence). A genuine production race condition is inherent to the code's concurrency logic and would typically reproduce when the test runs alone too.

The test escapes detection only when all 10 runs yield the same outcome, either all pass or all fail. Because runs are independent with per-run failure probability 0.05, P(all 10 pass) = 0.95^10 ≈ 0.599, which dominates. P(all 10 fail) = 0.05^10 ≈ 9.8 × 10^-14 is negligible. The escape probability is therefore approximately 60%, so a fixed 10-run detector misses most tests with a 5% true flake rate.