Michael Bolton first highlighted the difference between testing and checking. The output of checks are binary results (true/false, on/off, black/white). Checks can be executed by humans or automatically by machines. The output of testing is questions, facts and information. Tests can only be executed by humans. Unfortunately, a lot of people continue to use these words interchangeably which can lead to confusion and mistakes. To become better testers, I believe we need to start with the basics; our vocabulary.

Wikipedia defines regression testing as “a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes such as enhancements, patches or configuration changes, have been made to them”. It’s critical everyone on the team, testers and non-testers alike, understand there is a difference regression checking and regression testing.

So what exactly is regression checking?

I define regression checking as the examination of an existing behaviour, either manually or automatically, that results in a binary pass or fail outcome. Can you sign up for an account? Yes. Can you login? No. Can you make a payment? Yes.

If you have scheduled releases (weekly/fortnightly/monthly) and you rely solely on automated regression checks, you can unwittingly develop bad habits amongst your testers which can lead to complacency and mistakes. If your team is lucky enough to have a mature release process, where the code is pushed to production with every commit, then you won’t face this “regression checking” problem.

Unfortunately, automation can lull testers into a false sense of security. As they observe automated regression checks successfully passing, they start to naively believe that human/sapient testing activities, prior to scheduled releases, are no longer required. This creates a complacent mindset amongst testers who are now under the misguided illusion that automation appears to cover much larger swathes of functionality (because they see everything always passing!) than what it covers in reality. The more often tester see the automated checks successfully passing, the less benefit they see in manually exploring and interacting with features. When working with scheduled releases, relying solely on regression checks, with zero regression tests, is a dangerous idea because testers will eventually stop using their most valuable asset; their brain!

Variation and Information

Regression testing, on the other hand, is a far more creative and engaging activity. The input is variation, the output is information. Good regression tests require an intimate knowledge of the automation suite. A comprehensive understanding of what your automated checks cover and ignore is a non-negotiable prerequisite. Only then can you start to identify regression testing opportunities. Variation is a key component to quality regression testing. Figure out where and how to add it in. This approach not only helps exercise more of the code base but also makes the whole regression testing experience a far more valuable and thought-provoking activity. Turn the creation of regression tests into a game and challenge yourself to create paths through the application which you have never taken before.

Time, and not quality, has become the single biggest motivating factor in a testers’ decision-making process and as a result, regression testing needs to be time-boxed. If you regression test every check you’ll never get anything out the door! Choose features that are most at risk because of the release and focus your regression testing on those particular areas. If you have to, run your tests in parallel with your automated regression checks. If the automation takes 30 minutes, then give yourself 30 minutes to prep and run your tests.

Injecting Variation

As an example, let’s look at creating regression tests for an e-commerce site.

1. Change your starting point. Let’s assume there is an automated check that logs a user into the website from the homepage. As an alternative, try manually logging in when prompted to as you perform an action i.e. posting a review or comment. This would make a useful regression test as it adds variation around how users log in to the site. Habits are great at allowing you to make decisions faster. However, this habit, of going on auto-pilot when starting each of your tests, can have a detrimental outcome to the quality of your work.

2. Change the sequence of your steps. Again let’s assume there is an automated check that logs a user in and then adds an item to their shopping cart. Instead, while manually regression testing the feature, why not add something to your shopping cart first, and then log in after. Maybe this will accidentally clear your cart after you successfully log in or it could redirect to a wrong page.

3. Change the data. One of the biggest advantages and disadvantages of automation is its continual reuse of test data. When regression testing you should try to exploit this weakness. For example, assume we have a check that proves a user can successfully purchase a product. However, let’s also assume accounts can transition between different states. Maybe a users’ account could be in a suspended or unverified state. Try purchasing a product with an account in each state and see what happens.

Familiarity breeds contempt

Familiarity breeds contempt and regression checking is fraught with it. If you work with native mobile apps, internal legacy applications or a website that doesn’t push code to production with every commit, then you will continue to have scheduled releases. You will continue to face this “regression checking” risk. The best way to minimise this risk is to supplement your regression checks with regression tests. Never allow automation to create a complacent attitude towards regression testing.

By simply introducing the terms “regression testing” and “regression checking” to a team’s lexicon it can reduce mistakes and misunderstandings. You will no longer misinterpret the product owner when they ask “have the regression tests passed?”. At the end of the day, these people only care that behaviours haven’t unintentionally changed and features haven’t broken. When decision makers need to deploy or delay a release, “information and facts”, rather than “binary results”, become their most powerful asset.