End-to-End Tests You Can Trust

An end-to-end suite has exactly one asset, and it is not coverage. It is that when it goes red, somebody believes it. Lose that and the suite is a tax: it still costs minutes on every pipeline, and nobody reads its output.

The failure is gradual. One test fails intermittently, someone re-runs it and it passes. That happens enough times that re-running becomes the first response to any failure. Then a real regression fails the suite, gets re-run twice, passes on the third attempt because the flakiness went the other way, and ships.

Flakiness is a defect, not weather

The framing that matters: an intermittent test is a bug in the test, and it should be treated with the same seriousness as a bug in the application. The alternative — a quarantine folder of known-flaky tests that everyone ignores — is just a slower way of deleting them, with the maintenance cost retained.

Practically: when a test fails intermittently, either fix the cause or delete the test. A deleted test is honest about the coverage you have. A flaky one is not, and it consumes attention every week.

Nearly all flakiness comes from three sources.

Waiting for the wrong thing

The single largest cause:

await page.click("#submit");
await page.waitForTimeout(2000);      // hope
expect(await page.textContent(".status")).toBe("Saved");

A fixed sleep is a bet that the operation finishes within a duration. It is simultaneously too long — every run pays it even when the app is fast — and too short, on the loaded CI machine where everything takes twice as long. That combination is why suites are slow and flaky rather than one or the other.

Wait for the condition instead:

await page.click("#submit");
await expect(page.getByTestId("status")).toHaveText("Saved");  // retries until true

Modern assertions retry until they pass or time out, so the test takes exactly as long as the app takes and does not care about machine speed.

The subtler variants worth knowing:

• Asserting on an element that exists before it is ready. A button rendered but not yet hydrated accepts a click that does nothing. Assert on the enabled or interactive state, not merely on presence.
• Racing a background request. The assertion passes against stale content that is about to be replaced. Wait for the response or for the resulting state, not for the element.
• Animations. An element moving toward its final position produces click coordinates that miss. Disable animations in the test environment; there is no value in testing the transition and considerable cost in tolerating it.

Test data that other tests can see

The second cause is shared state. Tests pass individually, fail when run together, and fail differently depending on order — which is the signature.

A shared seeded account edited by one test breaks another. A test asserting “three items in the list” breaks when a parallel test adds a fourth. Two tests registering the same email address collide.

Each test creates the data it needs, with unique values. Not shared fixtures — its own records, namespaced by a random identifier. Then parallel execution is safe, order does not matter, and a failure means what it says.

Create that data through an API rather than the UI. Logging in through the login form on every test is slow and makes every test depend on the login page. Set up state via a request, inject the session, and let the test spend its time on the thing it is actually about — with exactly one test covering the login form itself.

And prefer asserting on relative facts over absolute ones. “This item appears in the list” survives a parallel test adding another; “the list has three items” does not.

Selectors that survive a redesign

The third cause is coupling to markup. A selector like .container > div:nth-child(3) button breaks when someone adds a wrapper, and the resulting failure implies the feature is broken when only the structure moved.

In order of preference:

• Role and accessible name — getByRole("button", { name: "Save" }). This is what a screen reader uses, so the test breaks when the user-visible meaning changes, which is exactly when it should. It also means an accessibility regression fails your tests, which is a genuine bonus.
• A dedicated test id for things with no accessible name. Explicit, stable, and a clear signal to anyone editing the markup that something depends on it.
• Visible text, acceptable but couples to copy, which changes for reasons unrelated to behaviour.
• CSS structure — avoid. Classes exist for styling and will be changed by people who do not know a test reads them.

What belongs in end-to-end at all

These are the most expensive tests you own — slowest to run, slowest to debug, most likely to break for unrelated reasons. They should be spent on the few paths where an outage is unacceptable:

• Sign up and sign in
• The core transaction — checkout, booking, publish
• Anything involving payment
• One pass through the primary navigation, to catch a broken build

Ten to twenty tests, running in a few minutes. Validation rules, error states, edge cases and formatting belong at a cheaper layer where they run in milliseconds and fail with a stack trace instead of a screenshot.

The instinct to grow the suite toward completeness is what produces the forty-minute pipeline that everyone ignores. Breadth is not what this layer is for.

Running them without doubling your pipeline

• Parallelise by shard. Test-created data makes this safe, and it turns a serial ten minutes into two.
• Split the run. A critical-path subset on every pull request, the full suite post-merge or nightly. Fast feedback where it gates, thoroughness where it does not.
• Capture trace and video on failure only. A recorded trace turns “fails only in CI” from an afternoon into two minutes, and storing them for every run is wasteful.
• Retry once, and treat it as a bug report. A retry keeps the pipeline moving; a test that needed one should be tracked and fixed, not silently tolerated. If your runner reports flaky passes, watch that number — it is the leading indicator of the suite losing credibility.

The short version

The only asset is that people believe a red run, so treat intermittency as a defect and delete what you will not fix. Never sleep — assert on conditions that retry. Have every test create its own uniquely-named data through the API. Select by role and accessible name. Keep the suite to the handful of paths that must never break, and push everything else down to a cheaper layer.