Category Archives: Test Approaches

A Test Matrix Approach for Organizing Your Testing

For photo credit click here. This article was first published in June 2011 in Testing Circus. I have revised my original article for this posting.

To prepare for testing, some testers will translate the functional requirements to formal test cases defining the input and expected output in order to cover different types of tests such as: positive, negative, and boundary. Test cases can be time consuming to write and maintain plus will not always provide the flexibility to introduce different testing scenarios. Over time the test cases can provide diminishing returns since we may be retesting the same keystrokes whereas exploring different testing paths provides an opportunity to find new bugs.

Testers can perform exploratory testing to explore areas of the application they feel may be at risk. At times a more structured approach is needed to help the tester organize her testing to ensure they adequately cover the features while providing the flexibility of exploratory testing.

The goal of the test matrix is to provide organization and not become a checklist that is blindly followed. The value of creating a test matrix must outweigh the time to create and maintain if it is to be reused. A test matrix can be beneficial for regression testing and when testing a feature across multiple browsers or testing areas. In the end, how a tester organizes her tests can be a personal preference or company-driven especially in regulated environments.

What is a Test Matrix?
A testing matrix can be created for existing features to facilitate regression testing or for new features that require multiple rounds of testing. The goal is to reduce the need to gather this information each time the feature is tested allowing the focus on generating test ideas and performing testing. This approach is also helpful for features that are tested across multiple browsers or modules within the application.

A testing matrix can be created using a spreadsheet or table format to take advantage of the row and column relationship providing a way to keep track of completed testing. The rows define the feature’s options and the columns represent the testing areas or browsers to test. This approach reduces the time to create and update the testing matrix through consolidating the feature’s options across testing areas. The tester can bundle multiple tests into one test scenario or she can test them individually based upon her test ideas, experience level in testing, and knowledge of the feature. This allows her to change her approach when retesting a feature and allows other testers to take different testing paths providing better overall testing coverage.

Creating a Test Matrix
If you are testing a web-based application and you are testing the ability to print a report using different browsers, your testing matrix may look as follows:

Chart 1 test matrix
The above matrix provides initial testing guidelines defining the browsers to test. The columns can be modified to add or change browsers. If multiple reports need to be tested, this matrix can be copied for each report or additional columns can be added. It really is up to you on the best way to organize the testing. With this information coupled with knowledge of the application and history of fixed bugs, testing ideas can be generated. This matrix can be customized to include columns to document the testing ideas and notes. A testing matrix should be created based upon the needs of the features being tested, the risk level, and how much documentation is helpful.

Variation of a Test Matrix
The above test matrix example defines specific feature options to be tested allowing the tester to generate their test ideas. Another variation would allow the tester to document their testing ideas. Your testing matrix may look as follows:

Chart 2 test matrix

Benefits of This Approach
This approach can be helpful when there are a define set of features that must be tested such as in regression testing or when the same set of options are tested across multiple browsers or testing areas. While the tester is executing her testing ideas, this approach can help her organize tests and document what has been tested so she can review progress. Because it is not always possible to perform all test ideas across all browsers or testing areas, the tester can assess how much testing to perform. This information can be reviewed with stakeholders or other testers to discuss test coverage.

Disadvantages of this Approach
There is a cost associated with creating the test matrix by ensuring the correct information is captured and maintaining it when the features change. There is the risk that a tester will test only within the constraints of this structure and could miss important tests. Training needs to occur to ensure testers use the testing matrix as a way to organize their testing and not constrain the testing.