Exploratory testing is more than just randomly tinkering around with the system. In fact, we can use tools & processes to give structure to our exploratory tests. Some of the most commonly used ones are charters, heuristics and oracles.
However, many tech folks have not heard of these terms before despite using the tools & processes themselves. This might explain the popularity of the topic being covered at tech meetups.
This post sheds light on these common tools/processes, finally putting a name to the testing activity being carried out.
The Magna Carta, one of the most famous charters in history, set out the rights and privileges for the King's subjects. In software testing terminology, a testing charter sets out the aims & objectives of an exploratory testing session.
Exploratory testing is used when trying to validate a user story or a new product release. It can take place alongside automated tests or be a sole activity on its own, sufficient enough to validate a test result. By having a testing charter, the session is kept focussed and time-boxed. Although, keeping the session time-boxed is far trickier! For example, one might come across a bug that appears in one browser and additional time would be needed to check other browsers.
One way to keep the session time boxed, would be to use Session Based Time Management (SBTM) when chartering. Like the name suggests, one would test for a specific period of time. It could be quick 10-minute check or take up the whole afternoon.
A charter is not meant to be a detailed plan. -- Jonathan Bach 1
The charter is drawn up indicating what areas need to be covered and any potential bugs to look out for.2 The charter itself can be something as simple as a user story on an agile board, which the whole team can view and add details to.
|Session Charter||Basket Calculation User Story|
|Browsers||Chrome, Safari iOS|
|Potential Bugs||Validate promo codes|
|Backlog Item||Q1S4 - 2342|
Bugs and potential pitfalls are recorded, and at the end of the session there is a team debrief where results of the session are discussed. This could be over a communications channel such as Slack or at the next daily stand-up.
Any member of the team can generate a charter be it the developer or the tester. But team members should review the generated charter – remember it's not a detailed plan! Generating a charter shouldn't be too difficult i.e. 'What should I test?'. The charter might come from a bug that has just been fixed or from the "interaction among a collection of stories". 3
Using charters avoids the duplication of exploratory tests and provides "valuable information to move the project forward". 3
Ok, so you have got an awesome charter which the team has reviewed and contributed. You are now ready to start your afternoon of bug hunting. You know what feature to test, but how exactly will you test it? What sort of things will you look out for?
A heuristic is a provisional and fallible guide by which we investigate or solve a problem; it’s also a method by which learning takes place as a result of discoveries informed by exploration -- Michael Bolton 4
You can apply heuristics to assist your exploratory testing session. For instance, you can check to see if an email address field accepts certain characters. Or if an application appears consistent on different mobile devices. These are all test heuristics that you can check the product against.
The FEW HICCUPPS mnemonic lists what the product should be consistent with. Coined by Michael Bolton, with the original list coming from James Bach. It provides the most commonly used heuristics when chartering a testing session.
F - Familiar Problems
When testing one should look out for problems encountered in previous sessions. But this can be misleading. As Bolton states, "problems that are significant in one product may be less of a priority in another". 4 So context is crucial.
E - Explainability
Misunderstanding of what the product should do wastes time.
This is made worse by reporting a problem that is expected behaviour or a 'known issue' that apparently everyone but the person testing knows about. A thorough understanding of the system is essential for all those involved in the project. The system's functionality should be well documented, so that even a newcomer would understand.
W - World
Bolton states that "the product itself should be consistent with things that we know about or can observe in the world". A lack of this would mean the product is inconsistent with its purpose. 5
H - History
Consistency is key. Providing there is no new functionality, the behaviour of the system should be the same as previous versions.
I - Image
Image is everything. Misspellings and grammar should be checked. The branding of the product must the reflect quality of workmanship.
C - Comparable products
The quality of workmanship should be reflected throughout all the products. When developing a new product, comparisons should be made against products in the same category and those which aren't but have the same code base.
C - Claims
The product should match up to the original specifications. If something has changed, documentation should be updated and everyone should be informed. This includes those involved in separate projects and those selling the products!
U - Users’ expectations
The system under test should behave in a way that is "consistent … what users want (and) … their reasonable expectations". 4
Meaning when developing we should always have the end-user in mind. For instance, consider the page loading of a site. The page may load super-fast when you are using the office’s wired Ethernet Internet connection. But end users may not have access to similar speeds and could see things differently, paving the path for unknown bugs.
P – Product
"The behaviour of a given function should be consistent with the behaviour of comparable functions or functional patterns within the same product unless there is a specific reason for it not to be consistent". 4
Let's use an example. Say you have a feature that lets you change the quantity of an item. You would need to make sure that both mouse & keyboard actions yield the same result. Unless certain actions have been disallowed.
P – Purpose
"The behaviour of a feature, function, or product should be consistent with its apparent purpose". 4
Using the example above, if the quantity of the item is changed and the user navigates away from the page, the updated quantity of the item should be retained.
S – Statutes
"The product should behave in compliance with legal or regulatory requirements". 4
For example, conforming to RFC standards, UX guidelines etc.
In addition to the FEW HICCUPS heuristics, there will be some specific to your domain. For instance, you might have a car hiring site that allows you to hire cars within a 5-mile radius of your address. You would need to make sure that miles within that radius and outside of it are tested.
The word oracle may conjure up an image of a fortune teller gazing into crystal ball, hoping to seek the results of your latest test run. Okay, maybe not the last part.
An oracle is a principle or mechanism by which we can tell if the software is working according to someone's criteria; an oracle provides a right answer—according to somebody. -- Michael Bolton 4
This 'right answer' can come in many forms.
There are many sources that we can use to validate test results. Examples include: documentation, written/ verbal confirmation of what is expected etc.
But a test oracle can also be an automated tool that, given an input, can determine whether the result is correct or incorrect. This is done be having a known set of accepted values. Some tools automatically generate the expected result, which is useful when it comes to regression testing. 2
So what do you do if you don't have a test oracle?
Data Science is a field that is growing in popularity, covering areas such as Artificial Intelligence and Machine Learning.
With scientific problems, it can be difficult to predict what the outcome should be. To overcome this obstacle, techniques can be applied that give you an idea, on whether the result was the one you expected.
These include doing a boundary range check i.e. if the result falls into the range of accepted values, then the result is deemed as expected. Or comparing the results to another system that "overlaps in functionality”.7
You can see how the tools tie together. You need a charter to define the purpose. Once you have got that, you can use heuristics to test different aspects of the system. You can then consult oracles to see if the behaviour of the system under test was expected behaviour.
Sitting at the top of the automation pyramid, exploratory testing is often overlooked in favour of automation. But curiosity is something that can't be automated and that curiosity often leads to unexpected bugs being found. A structured approach means that everyone is aware of what and how a feature is being validated. So it’s time to shed light on the top of the pyramid.