What a self-healing test run is actually good for

A self-healing nightly test run sounds like the end of flaky pipelines. The honest version is that it saved less triage time than I expected, and told us something far more useful instead.

I built one for a European energy trading platform. Overnight the suite runs, an agent looks at what failed, repairs what it can, and raises a bug for what it cannot. By the time the team starts work, the morning report is already sorted.

What the setup does

The mechanics are simple to describe. The regression suite runs on its normal nightly schedule. When it finishes, an agent takes each failure in turn and reads the evidence: the assertion that failed, the stack trace, the logs, and where available a screenshot of the state the test was in.

For each failure it reaches one of three conclusions.

1. 1.The test is wrong and the application is right. The agent proposes a fix to the test.
2. 2.The application is wrong. The agent raises a bug with the evidence attached, and leaves the test failing.
3. 3.It cannot tell. The failure goes to a person, with the agent's reasoning so far.

The third outcome matters more than it looks. An agent that is forced to choose between the first two will guess, and a confident guess in a test pipeline is worse than an honest question.

The repairs were the least interesting part

I expected the value to be in the repairs: fewer red builds, less time spent on the same broken selectors. There was some of that. It was not the point.

When the agent could fix a test on its own, the failure was almost always our own test debt. A selector that had moved. A wait that was too optimistic. A fixture nobody had updated since the last schema change. Cheap to fix, and it should never have reached a human in the first place.

When the agent could not fix it, the failure was usually real. Those are the ones worth someone's morning.

So the value was not the healing. It was that the pipeline started sorting its own noise from its own signal, and we could finally see the ratio between the two.

The ratio is the measurement

Most teams know their suite is noisy. Very few can say how noisy. Once every failure is classified, you can. Count the failures that turned out to be test debt and the ones that turned out to be defects, week by week.

• If test debt dominates, the suite has a maintenance problem, and adding more tests will make it worse before it makes it better.
• If the share of real defects is rising, something upstream has changed: a release cadence, a team, a dependency. That is worth knowing before the next release, not after.
• If the agent keeps fixing the same kind of failure, the fix belongs in the framework, not in the tests. Repeated repairs to waits usually mean a missing synchronisation helper.

None of this needs the agent to be clever. It needs the classification to happen every night, consistently, and to be written down.

The agent needs a hard boundary

This is the part I would insist on before anything else. The agent may change test code. It may never change application code, and it may never weaken or delete an assertion to make a test pass.

Without that rule you are not healing tests, you are deleting evidence. A test that fails because the application is wrong is doing its job. An agent rewarded for a green build will find that the quickest way to green is to agree with the application.

In practice the boundary is a short list of things the agent is not allowed to do, enforced by the pipeline rather than by the prompt:

• Change anything outside the test code.
• Remove, skip or disable a test.
• Change an expected value so that it matches the actual one.
• Loosen a tolerance, a timeout or a comparison to get a pass.

A prompt can ask for good behaviour. Only the pipeline can guarantee it. If a proposed fix touches a forbidden path or changes an assertion, it should be rejected before a human ever sees it.

Every fix is a reviewable change

The second rule: every automated fix lands as a reviewable change, never a silent commit. Each one carries the failure it was responding to, the agent's diagnosis and the diff. Someone on the team approves it, or does not.

This costs a few minutes each morning, and it is the cheapest few minutes in the process. It is how you find out that the agent has misread a failure, and it is how the team keeps owning its own suite instead of slowly handing it to something nobody reads.

If you cannot see what it did last night, you do not have a quality gate. You have a rubber stamp.

Why this matters more once agents write the code

A self-healing run is a small example of a bigger problem. As coding agents write more of the application and more of the tests, the question of who is allowed to decide that something passed gets harder, not easier.

The same principle applies at every scale. Whatever produced the change should not be the only thing that verifies it. An agent that writes a feature and its tests will share its own misunderstanding across both. An agent that repairs tests without limits will share the application's misunderstanding instead. Independent verification, with boundaries the agent cannot move, is what turns automation into evidence.

Start with the boundary and the review. The healing is the easy part.