When test automation pays off

The rule: frequent runs, expensive failures

Test automation is an investment, not a quality measure in itself. An automated test saves money when it replaces a check that would otherwise be repeated by hand, and when the defect it catches would cost more than the test itself. Both conditions have to hold. A test that runs twice a year saves hardly any working time. A test for a flow whose failure nobody would notice prevents no expensive damage.

The same rule tells you what should stay manual: checks that run rarely, screens that change every week, and exploratory testing where a person looks for the unexpected. Automate those and you pay the one-off effort and then the maintenance without getting meaningful time back.

What belongs in the calculation

When I talk to heads of engineering, the calculation usually has two items: the effort to build the tests and the test time saved. An honest calculation has four.

The first three can be counted in hours. The fourth should at least be named: what does a defect in billing, in login or in an interface cost if it goes unnoticed for a month?

• One-off effort: framework, test environment, test data, the first tests and the integration into the pipeline.
• Ongoing maintenance: adapting tests to changed screens, investigating unstable tests, keeping test data current. This item is almost always underestimated.
• Test time saved: the hours a manual regression run ties up today, multiplied by how often it runs and by the share that can actually be automated.
• Residual risk: defects that slip through despite automation, and defects that reach customers without it. This item is rarely expressed in euros, yet it often decides the outcome.

A break-even formula with a worked example

The simplest version of the calculation leaves residual risk aside and asks one question: after how many months has the time saved paid for the build? Break-even in months equals the one-off effort in hours divided by the net monthly saving in hours. The net saving is runs per month times hours per run times the automatable share, minus maintenance per month.

An example with plausible figures for a small product team. The regression runs four times a month and takes 16 hours, at an internal rate of €90 per hour. Building the automation takes 120 hours plus 6 hours of maintenance a month, and 70% of the checks can be automated.

The same formula shows when it does not pay. If the regression runs only once a month, the net saving drops to 11.2 minus 6 = 5.2 hours a month, and break-even moves out to about 23 months. By then the software has changed enough that part of the tests would have to be rewritten anyway. Frequency decides, not technology.

• Manual effort today: 4 × 16 hours = 64 hours a month, or €5,760 a month.
• Automatable share: 70% of 64 hours = 44.8 hours a month.
• Net saving: 44.8 hours minus 6 hours of maintenance = 38.8 hours a month, or €3,492 a month.
• One-off effort: 120 hours × €90 = €10,800.
• Break-even: 120 hours divided by 38.8 hours a month = about 3.1 months. After twelve months, 465.6 hours of net saving (€41,904) stand against €10,800 of one-off effort.

The test pyramid as a cost lever

One-off effort and maintenance depend heavily on the level at which you test. Ham Vocke's article on martinfowler.com describes the test pyramid: many fast unit tests at the base, fewer integration tests above them, and a few tests through the user interface at the top. The higher a test sits in the pyramid, the more of the system it needs and the slower it runs. It also fails more often for reasons unrelated to the behaviour it checks.

For the calculation this means: a check that can be written as a unit or integration test costs a fraction of a UI test to build and maintain. Many teams start by automating what the manual tester sees, which is the user interface. It is often cheaper to protect the business logic underneath and to test only the few flows through the UI where the interaction of all parts is the actual risk. The automatable share in the example does not change, but the 120 hours of build and the 6 hours of maintenance go down.

Flaky tests and test data: the underestimated maintenance costs

A test that is sometimes green and sometimes red without any code change costs twice: someone has to check the result, and the team loses trust in the whole suite. Google reported on its Testing Blog that about 1.5% of all test runs there produced such a flaky result, which across millions of tests adds up to considerable effort. The countermeasures are known: mark the affected tests, quarantine them, retry selectively, and fix the root cause, usually timing dependencies, shared state or external services.

The second maintenance driver is test data. Tests that run against a shared database with accumulated content break as soon as someone changes that content. Tests that create and remove their own data take more effort to write and considerably less to maintain. If you want to keep the 6 hours of maintenance from the example, plan for both points from the start.

Hotfixes and the Cyber Resilience Act

So far this has been about hours saved. There is a second driver that the formula does not capture: the time between finding a defect and shipping the fix. If every fix needs the full manual regression run, a hotfix takes at least as long as that run, two working days in the example. With automated coverage of the critical flows, it takes hours.

For manufacturers of products with digital elements this is becoming an obligation. Regulation (EU) 2024/2847, the Cyber Resilience Act, requires in Annex I that vulnerabilities are addressed and remediated without delay, including through security updates. Under Article 14, actively exploited vulnerabilities must be reported within 24 hours. Starting a multi-day test run at that point is a problem. The ability to ship a small change quickly and safely is no longer a convenience but a requirement. Which products are in scope and from when is a question for legal counsel. The technical capability can be built now. In the calculation it belongs on the residual-risk side, and it weighs heavily there.

What this means for your decision

The calculation is not an argument for or against automation. It is a filter applied to each check. Put in your own figures: How often does the regression run, and how long does it take? What share can be automated at reasonable effort? What does a defect cost in the flows that fall outside? Candidates with many runs and a high failure risk come first. Candidates with few runs and low risk stay manual, and that is not a weakness.

To run the numbers for your own situation, use the calculator on this website. It uses the same formula as this article and shows break-even and annual saving for your figures.