The Case For Formative Human Factors Testing

When thinking about human factors, many device manufacturers tend to be most familiar with the all-important validation testing. This isn’t surprising, since it is mandated by the FDA for devices that could pose safety risks if used incorrectly. Because the validation test is required  human factors testing, it can be tempting to skip over preliminary human factors activities during the development process.

However, this approach is problematic from both a device usability and a safety standpoint. Omitting the formative stages of evaluation increases the probability of discovering device flaws when it is too late to make design changes. Although minor changes to design can be acceptable, any updates made in response to safety-related errors, discovered in validation, will likely require a repeat of the validation study. You don’t want to be in a position in which you have to decide between losing time and money by going back to the design and development phases, or taking your chances with the FDA and attempting to justify a poor design. This may be the most obvious motivation to bite the bullet and start human factors early, but there are a few additional , equally important reasons you may not be considering.

Flexibility In Methods

The FDA human factors guidance, Applying Human Factors and Usability Engineering to Medical Devices, highlights specific expectations for validation testing. This includes how to identify and test critical tasks, expectations for use scenarios, incorporation of training and use of instructional materials, and expectations for subjective follow-up with participants after the completion of testing. To meet the FDA’s expectations for human factors, it is important that these guidelines be followed.

Alternatively, one of the benefits of formative testing is the possibility for a flexible approach. Although following the validation methods will satisfy the FDA, these methods may not always be best for gathering the type of feedback — during formative stages — that is intended to produce design inputs. For example, utilizing a cognitive walkthrough approach, in which users provide feedback and commentary on a device and its instructional materials while performing tasks, is not permitted at the validation stage. However, utilizing this method during the formative period can provide much more insight into a user’s thoughts and help to identify points of confusion.

Formative stage testing also provides an opportunity to incorporate more robust evaluations of the instructional materials and labeling. Users cannot be required to use instructional materials during validation testing, but designing a formative study that forces users to rely on the materials can result in feedback that greatly improves design. Do your design a favor and take advantage of utilizing less-stringent methods while you have the chance!

It’s Not As Intuitive As You Think

With all of the great data that can be collected and utilized to improve product usability before reaching the validation stage, why is it that device developers choose to omit formative stage evaluations? Part of the problem often lies in the development team being overly-familiar with its own product. When a device takes years to develop, it becomes almost impossible to see the device from an outsider’s perspective. I know what it’s like to see jaws drop when user testing begins and performance doesn’t live up to everyone’s assumptions about how easy the device should be to use.

Immersion in the development process can produce a false sense of what will be intuitive to a user, since device operation has become intuitive and obvious to those who have been involved in development from the beginning. Gathering feedback and formative data using prototypes that aren’t finalized can help the development team detach from the design before reaching a point where they are less willing to let go.

Unanticipated User Errors

Developing a robust risk analysis for your product is key not only to identifying the critical tasks for user testing, but also to anticipating what types of errors users could perform during those tests. For all of the information that can be gathered via brainstorming, consulting with subject matter experts, and digging through archival safety event data related to similar products, it still is extremely difficult to predict what users will do once they get their hands on your device.

Skipping formative user evaluations means that unanticipated use errors will make their first appearance in validation testing, and no one wants that kind of surprise so close to an FDA submission. Discovering potential errors earlier not only provides an opportunity to design them out of the system, but also prepares you for what to expect during validation if use errors cannot be fully mitigated.

Inform A Better Pre-Submission

When performed correctly, formative human factors activities are going to improve your final submission to the FDA by demonstrating that design changes were made based on feedback, both from human factors experts and target end-users of your device. However, the ways in which formative evaluations can improve FDA pre-submissions are more often overlooked.

For human factors studies, a major benefit of the FDA Pre-Submission Program is receiving feedback on the validation study protocol and ensuring the approach to study design is appropriate. Formative studies ensure that the protocol is more robust, because findings from these studies can be incorporated into the protocol and inform the validation design. This also decreases the likelihood that the FDA will question your choice of use scenarios, approach to the simulated setup, or data collection and analysis methods. Making changes based on formative results can validate that these choices are correct, leaving less room for regulatory criticism.

If you forego formative testing and are not able to present a protocol that is accurate and robust, your approach may need so many modifications after the pre-submission that it is not possible to move forward confidently without completing another pre-submission. Few developers can afford to do multiple unplanned submissions, and waiting to pre-submit until at least a portion of the formative work has been completed can prevent the need for repeated work.

Avoid An Awkward Situation

A pet peeve of all medical human factors consultants is seeing our clients elect to forego formative evaluation, and then attempt to pressure us into using our data to make the device design appear good enough to pass muster with the FDA. Although our job is to help clients meet the FDA’s human factors expectations, our ultimate goal needs to be ensuring that those devices that do make it to market are safe.

Insisting on manipulation of the human factors data (however subtle it may be) in an attempt to show the FDA better user performance forces us to decide between compromising our ethics and having a dissatisfied client. Consultants and developers need to be working towards the same goal of achieving optimal safety and usability, and involving us earlier in the process decreases the chances of both parties having to make uncomfortable decisions later on.

Practice Makes Perfect

Even if you are confident that your design is optimized and you’ve identified all possible sources of risk, going into a validation study cold is going to make things more difficult. It’s rare that protocols are perfect the first time around, and it is pretty common to discover that updates are needed to make the evaluations run smoothly.

Common things we discover include users not understanding directions from the moderator, certain details of the study being more difficult to capture than we thought, or the setup of the study space not promoting natural task performance for users. Although minor changes to the protocol often can be made without issue, any significant changes made after the validation study begins likely will mean having to throw out data and start over with new users.  

Having the validation study as your first round of user testing can also cause difficulty for the study moderator. Conducting a study often requires flexibility, and without the benefit of formative testing, it can often be difficult to adapt to unanticipated situations and user behavior. Although pre-validation dry runs can help alleviate some of these issues, you don’t want to waste time trying to perfect things when time is of the essence — when you could have utilized a formative study as a practice run and reaped all of the other design input benefits at the same time.

Conclusion

The formative human factors evaluation period is too often overlooked, due to constraints of time and money, and this can lead to unfortunate consequences for medical device manufacturers. We need to remember all of the benefits of starting early with human factors, and making the right choices early on will actually save time and money by ensuring the design is optimized earlier and that multiple attempts at validation will not be required. Human factors consultants and device developers need to work together to make sure the right decisions are made, resulting in safer, more usable devices getting to market more quickly.

About The Author

Natalie is the Program Manager for the Usability Services division of the National Center for Human Factors in Healthcare. She oversees the technical and quality aspects of usability projects conducted  both for the medical device industry and within MedStar Health. She is involved in all aspects of planning and executing usability tests, and also leads an initiative to incorporate usability testing in medical device procurement. She has a special interest in ensuring that safe and effective products are brought to market through successful FDA submission.

Natalie holds a master’s degree in industrial engineering, with a focus on human factors and ergonomics, from the University of Wisconsin, where she was mentored by Dr. Ben-Tzion Karsh. Some of her previous work involved research on primary care redesign for the aging population and implementation of process improvement efforts in the ambulatory care setting.

Natalie can be contacted through http://www.medicalhumanfactors.net.