Using Charters, Heuristics and Oracles

Not many tech folks have heard of these testing tools & processes before despite using them. Intrigued? Read on...
By Shamyla Sidd Posted 16 December 2016

Using Charters, Heuristics and Oracles

Exploratory testing is more than just randomly tinkering around with the system. In fact, we can use tools & processes to give structure to our exploratory tests. Some of the most commonly used ones are charters, heuristics and oracles.

However, many tech folks have not heard of these terms before despite using the tools & processes themselves. This might explain the popularity of the topic being covered at tech meetups.

This post sheds light on these common tools/processes, finally putting a name to the testing activity being carried out.

Charter

The Magna Carta, one of the most famous charters in history, set out the rights and privileges for the King's subjects. In software testing terminology, a testing charter sets out the aims & objectives of an exploratory testing session.

Magna Carta - The most famous charter of them all

Why use a Charter?

Exploratory testing is used when trying to validate a user story or a new product release. It can take place alongside automated tests or be a sole activity on its own, sufficient enough to validate a test result. By having a testing charter, the session is kept focussed and time-boxed. Although, keeping the session time-boxed is far trickier! For example, one might come across a bug that appears in one browser and additional time would be needed to check other browsers.

Session Based Time Management

One way to keep the session time boxed, would be to use Session Based Time Management (SBTM) when chartering. Like the name suggests, one would test for a specific period of time. It could be quick 10-minute check or take up the whole afternoon.

A charter is not meant to be a detailed plan. -- Jonathan Bach 1

The charter is drawn up indicating what areas need to be covered and any potential bugs to look out for.2 The charter itself can be something as simple as a user story on an agile board, which the whole team can view and add details to.

Exploratory Session  
Session Charter Basket Calculation User Story
Notes /
Browsers Chrome, Safari iOS
Potential Bugs Validate promo codes
Environment CI
Backlog Item Q1S4 - 2342


Bugs and potential pitfalls are recorded, and at the end of the session there is a team debrief where results of the session are discussed. This could be over a communications channel such as Slack or at the next daily stand-up.

Who should generate charters?

The team.

Any member of the team can generate a charter be it the developer or the tester. But team members should review the generated charter – remember it's not a detailed plan! Generating a charter shouldn't be too difficult i.e. 'What should I test?'. The charter might come from a bug that has just been fixed or from the "interaction among a collection of stories". 3

Using charters avoids the duplication of exploratory tests and provides "valuable information to move the project forward". 3

Heuristics

Ok, so you have got an awesome charter which the team has reviewed and contributed. You are now ready to start your afternoon of bug hunting. You know what feature to test, but how exactly will you test it? What sort of things will you look out for?

Taking a heuristic approach to bug hunting

A heuristic is a provisional and fallible guide by which we investigate or solve a problem; it’s also a method by which learning takes place as a result of discoveries informed by exploration -- Michael Bolton 4

You can apply heuristics to assist your exploratory testing session. For instance, you can check to see if an email address field accepts certain characters. Or if an application appears consistent on different mobile devices. These are all test heuristics that you can check the product against.

A case for the HICCUPPS

The FEW HICCUPPS mnemonic lists what the product should be consistent with. Coined by Michael Bolton, with the original list coming from James Bach. It provides the most commonly used heuristics when chartering a testing session.

F - Familiar Problems

When testing one should look out for problems encountered in previous sessions. But this can be misleading. As Bolton states, "problems that are significant in one product may be less of a priority in another". 4 So context is crucial.

E - Explainability

Misunderstanding of what the product should do wastes time.

This is made worse by reporting a problem that is expected behaviour or a 'known issue' that apparently everyone but the person testing knows about. A thorough understanding of the system is essential for all those involved in the project. The system's functionality should be well documented, so that even a newcomer would understand.

W - World

Bolton states that "the product itself should be consistent with things that we know about or can observe in the world". A lack of this would mean the product is inconsistent with its purpose. 5

H - History

Consistency is key. Providing there is no new functionality, the behaviour of the system should be the same as previous versions.

I - Image

Image is everything. Misspellings and grammar should be checked. The branding of the product must the reflect quality of workmanship.

C - Comparable products

The quality of workmanship should be reflected throughout all the products. When developing a new product, comparisons should be made against products in the same category and those which aren't but have the same code base.

C - Claims

The product should match up to the original specifications. If something has changed, documentation should be updated and everyone should be informed. This includes those involved in separate projects and those selling the products!

U - Users’ expectations

The system under test should behave in a way that is "consistent … what users want (and) … their reasonable expectations". 4

Meaning when developing we should always have the end-user in mind. For instance, consider the page loading of a site. The page may load super-fast when you are using the office’s wired Ethernet Internet connection. But end users may not have access to similar speeds and could see things differently, paving the path for unknown bugs.

P – Product

"The behaviour of a given function should be consistent with the behaviour of comparable functions or functional patterns within the same product unless there is a specific reason for it not to be consistent". 4

Let's use an example. Say you have a feature that lets you change the quantity of an item. You would need to make sure that both mouse & keyboard actions yield the same result. Unless certain actions have been disallowed.

P – Purpose

"The behaviour of a feature, function, or product should be consistent with its apparent purpose". 4

Using the example above, if the quantity of the item is changed and the user navigates away from the page, the updated quantity of the item should be retained.

S – Statutes

"The product should behave in compliance with legal or regulatory requirements". 4

For example, conforming to RFC standards, UX guidelines etc.

More than a few hiccups?

In addition to the FEW HICCUPS heuristics, there will be some specific to your domain. For instance, you might have a car hiring site that allows you to hire cars within a 5-mile radius of your address. You would need to make sure that miles within that radius and outside of it are tested.

One can have many heuristics specific to their domain being tested so the list is endless. Elizabeth Hendrickson's Test Heuristics Cheat Sheet lists some of the commonly used heuristics. 6

Oracles

The word oracle may conjure up an image of a fortune teller gazing into crystal ball, hoping to seek the results of your latest test run. Okay, maybe not the last part.

An oracle is a principle or mechanism by which we can tell if the software is working according to someone's criteria; an oracle provides a right answer—according to somebody. -- Michael Bolton 4

This 'right answer' can come in many forms.

Many sources of truth

There are many sources that we can use to validate test results. Examples include: documentation, written/ verbal confirmation of what is expected etc.

Test Oracles

But a test oracle can also be an automated tool that, given an input, can determine whether the result is correct or incorrect. This is done be having a known set of accepted values. Some tools automatically generate the expected result, which is useful when it comes to regression testing. 2

So what do you do if you don't have a test oracle?

The problem with Oracles

Data Science is a field that is growing in popularity, covering areas such as Artificial Intelligence and Machine Learning.

With scientific problems, it can be difficult to predict what the outcome should be. To overcome this obstacle, techniques can be applied that give you an idea, on whether the result was the one you expected.

These include doing a boundary range check i.e. if the result falls into the range of accepted values, then the result is deemed as expected. Or comparing the results to another system that "overlaps in functionality”.7

Get charter, use your heuristics & consult the oracle

You can see how the tools tie together. You need a charter to define the purpose. Once you have got that, you can use heuristics to test different aspects of the system. You can then consult oracles to see if the behaviour of the system under test was expected behaviour.

Why should you use them? (Besides the obvious time & effort saved)

Sitting at the top of the automation pyramid, exploratory testing is often overlooked in favour of automation. But curiosity is something that can't be automated and that curiosity often leads to unexpected bugs being found. A structured approach means that everyone is aware of what and how a feature is being validated. So it’s time to shed light on the top of the pyramid.

Shamyla Sidd
Shamyla Sidd
Developer in Test