Reduce Risk With Exploratory Medical Device Software Testing
By David Borcherding, Seapine Software, Inc.
More and more medical devices have some sort of software component to them. The teams responsible for testing this software often rely on scripted testing, both manual and automated, to decrease the risk of defects in a product under development. Scripted testing is documented in test plans or test protocols with test cases or test procedures with the documented evidence of test results or test runs.
The problem with this approach is that scripted testing is not meant to identify error conditions in scenarios that significantly deviate from the design or requirements, even if a comprehensive risk management plan is followed. To find these hidden or divergent risks, you need to go off script, and that’s where adding exploratory testing can help.
Exploratory testing — sometimes called usability testing, use testing, user testing, or formative evaluations — is the hands-on, simulated-use testing done to discover and explore unanticipated hazards.
It’s almost impossible to plan tests that cover every variation in data, configuration, interaction, sequence, timing, and so on. Scripted tests are designed to ensure that the product meets the requirements (using new feature test cases) and to mitigate the risk of new features breaking existing functionality (via regression test cases). Experienced testers can anticipate issues that might occur, but it may be too costly or time-consuming to write a test case for every scenario that comes to mind.
In her book Explore It! Reduce Risk and Increase Confidence with Exploratory Testing, Elisabeth Hendrickson says the best test strategy answers two core questions:
- Does the software behave as intended under the conditions it’s supposed to be able to handle?
- Are there any other risks?
Scripted testing can answer the first question, but to find those potentially critical “other risks,” you need to explore.
Push The Envelope
Exploratory testing puts the thinking back in the hands (or rather, the head) of the tester. As an exploratory tester, you design the test, you execute it immediately, you observe the results, and you use what you learn to design the next test. You’re not just following steps in a test case someone else created; you’re pushing the application or product to its limits to gain a better understanding of how it works and where it breaks. You see it from the user’s point of view, rather than the developer’s.
Ultimately, you get a more complete view of the product — including its weaknesses and hidden risks.
Discover More Defects
On average, 11 percent more overall software defects are discovered through exploratory testing vs. scripted testing. For defects that should be immediately obvious, such as a missing button in the user interface, exploratory testing discovers 29 percent more vs. scripted. When it comes to “complex” bugs (bugs requiring three or more user actions to cause an error or failure), it jumps to 33 percent more defects found. (Source: Defect Detection Efficiency: Test Case Based vs. Exploratory Testing)
The reason you find more defects when using an exploratory method is because you have more latitude to try different types of tests, using your past experience and knowledge of the product. Scripted testing, on the other hand, limits you to only the steps outlined in test cases, which, in turn, limits your ability to consider other test scenarios.
There are numerous reasons why test cases don’t always lead to finding bugs, such as how well the test case was written (did the analyst understand the requirement?), who wrote the test case (is the analyst writing the test case knowledgeable about how the product works?), how well the requirements document described new functionality, and so on.
Even if you had perfect test cases, exploratory testing would still find more defects over the course of a release, for several reasons:
- You tend to find a good number of defects when “testing around” functional areas while verifying defects. Fixing an issue often breaks something else.
- If a defect exists and is not found while executing the initial test run (following the test cases steps), it is unlikely that the next tester running the same test will find the defect. However, exploratory testing in the same functional area may reveal the bug.
- Exploratory testing allows you to think outside the box and come up with use cases that might not be covered in a test case. For example, you might perform one test and then ask yourself, “What if I tried this? What if I didn’t do that?”
- Some defects, typically the hard ones to find, are dependent on a sequence of events. If you don’t have really deep test cases, you can miss finding defects that exploratory testing can find in a less-structured, longer test session.
Find The Most Important Defect In The Shortest Time
Because you are not bound by the test case steps, exploratory testing makes finding important defects faster. Essentially, it allows you to cover more ground and focus on testing the “what ifs.”
For example, let’s say you’re assigned to test the “Edit Patient Name” functional area in a product. There are two possible scenarios for your assignment:
- Execute a test run, or
- Perform exploratory testing on the “Edit Patient Name” functional area.
In the first scenario, you would follow the steps outlined in the test case to verify that each step works, reporting any bugs that are found. But what if the most important bug — the one that crashes the application and deletes the user’s data — doesn’t occur during these steps? You won’t find it.
In the second scenario, you explore or “test around” editing the patient name. At first, you might perform the same steps as at the beginning of the test case. As testing progresses, however, you might ask, “What happens when I click Edit, delete the name, and try to save the patient name with a blank field?” And boom, the application crashes and deletes all of the user data. You just found the most important bug by exploring something that wasn’t a step in the test case.
Exploratory Testing Feeds Scripted Testing
When exploratory software test sessions are recorded, they can easily be converted into repeatable test cases or regression tests without the need for you to take notes in a separate document or notebook.
A test session recording application records all activity and builds a detailed history of the test session, including descriptions of the user interface controls used and a screen shot of every step with the relevant graphical user interface (GUI) element automatically highlighted.
Once you’re done exploring, the app generates a step-by-step written script of the test steps in plain English. You can then save this script as a test case in your test case management solution.
Test session recording apps simplify the defect reporting mechanism for risks found in exploratory test sessions, because they make it easy for you to go back and reproduce the error. Steps and screens are automatically captured, so there’s no need to do all of that manually when submitting an issue to development and documenting a risk mitigation with a test case.
Fill The Gap
As corporate belts tighten and cause reductions in testing budgets and staff, you might be tempted to settle for a documented test approach because you know it will satisfy your company’s quality standards or auditors. All that really does, however, is open the door to risk.
Combining exploratory testing with mandated documented scripted testing can help fill the gap left by shrinking resources to keep your test effort strong and reduce the risk of issues in the final product.