Little consideration is given to the opportunity cost of automation. What are the implications to the quality of software being delivered? What do we lose by focusing on the automated/coding skills of testers and ignoring their “true testing skills”? Automation is an important cog in the delivery of software but there are limits to its benefit which aren’t always fully recognised. When making decisions about automation, we all need to understand its limits and associated costs; it’s opportunity cost.

Opportunity cost can be defined as “a benefit, profit, or value of something that must be given up to acquire or achieve something else. Since every resource (land, money, time, etc.) can be put to alternative uses, every action, choice, or decision has an associated opportunity cost”.

Opportunity Cost 1 – The Decline of testing skills

I think there’s an unhealthy obsession with testers needing developer like coding experience. We should be better coders but we must be better testers. Testing in an agile environment puts problem-solving at the heart of it; something the waterfall/v model test approaches never did. We have successfully carried over our valuable testing intuition and healthy scepticism of developer’s guidance/advice from the old testing world but these skills can only help us so much. 

True Testing Skills

When I talk about good “true testing skills” I’m not referring to knowledge of how to exercise the boundaries of an input field or having the foresight to check failover scenarios. These are the basics for any half decent tester. We need to bring more testing skills/value to our teams than just the basics. Good testing skills are knowing that more exploration of a particular feature is required because, when asked a few simple questions about a story, the developer’s answers seemed indecisive and unsure. Good testing skills are understanding the limitations of running the same manual checks multiple times. Here’s an interesting article about the negative impact automation has had on airline pilot’s flying skills.

Good coders without testing skills are just automated checkers. Good testers without coding skills won’t be able to keep pace with executing an ever-expanding suite of regression tests. There needs to be a balance between both sets of skills. Automation skills shouldn’t be the only testing skill promoted and encouraged.

Opportunity Cost 2 – The Pesticide Paradox

The Pesticide Paradox simply means the more often you run the same checks, the less effective they become. James Whittaker describes this phenomenon by saying ‘pesticides will kill bugs but spray the same field enough times with the same poison and the remaining bugs will grow immune’. There is obvious value in running your automated checks regularly but their associated problems need to be examined too.

Firstly, some testers will start excluding features from their exploratory test sessions because they believe related automated checks already cover what they are about to test i.e. they see this as duplication of work. In the short-term this attitude may be fine but over the long-term if overlooked, it can develop into an unhealthy habit. Automated checks should never mean certain features are off-limits for exploration. Automated checks should be seen as a mechanism for improving the effectiveness and speed of your complimentary exploratory tests.

Secondly, the data we use to run our checks with is quite often test generated data. Ideally, we should use production data but sometimes we need consistent reliable data. This seemingly innocuous trade-off can allow bugs to creep into production. As we have created a sanitized set of data for testing purposes, we have inadvertently removed all potential variation those automated checks could have used. If you have to use test data, make sure to regularly review it to mitigate such risk. This also ties in nicely with my third opportunity cost; maintenance.

Opportunity Cost 3 – Maintenance Time

Pretty obvious one. The more checks you write, the more code you need to maintain, the less time you have for testing. Also as your product evolves so too should your automation suite. You will need to spend more time comparing the benefits of each automated check with their associated maintenance costs. Sometimes automation can be flaky. If that occurs, 1 of 2 things will follow. One, you will spend more and more time investigating why your checks randomly failed. Or two, you stop trusting your automation results (“ah sure 95% of them passed, that usually means it’s all fine”) and people start commenting out checks. Both are a reality of automation but tend to be readily accepted as par for the course. We need to be very careful about what, why and when we automate.

The Automation Opportunity 

Automation can be a powerful tool but unless people recognise its limits, the expectation and reality of what automation can bring will never match up. People need to understand automation on its own is not a silver bullet. When used properly it will dramatically reduce the amount of manual testing time required for each release while increasing the robustness of the software. When used ineffectively, automation will dramatically increase the amount of test maintenance time for each release while decreasing people’s confidence in the software. Use automation wisely or the opportunity cost may be greater than you think!