Medical Device Usability Testing In The Age Of COVID-19
By Charles L. Mauro, CHFP, Mauro Usability Science
Testing the human factors performance of medical devices, especially those undergoing submission for FDA approval, is a complex and often poorly understood process, even in the best times.
A recent review of FDA guidance with respect to medical device human factors testing criteria indicated serious structural problems that may contribute to the extensive number of issues reported by users of such devices. Note that this review, focusing on methodology, was produced before the global COVID-19 pandemic.
Nonetheless, corporations striving to validate their medical devices’ human factors performance have a massive new problem layered on top of testing programs already under attack for improper testing methods and overzealous risk mitigation practices. However, like other major historical changes in medical device development processes and validations, the impact of COVID-19 is more tactical than strategic.
Device development teams now must contend with a complex issue that will fundamentally change the way human factors engineering (HFE) studies are planned, executed, and reported. However, to be totally clear, user testing is going to proceed and will do so soon.
Under COVID-19, device development teams must first proceed based on robust and explicit procedures to maintain development deadlines and reduce risk when engaging in user testing programs. User testing is going to continue, but under a different set of priorities and far more restrictive professional guidelines.
Below, we share our standard operating procedures (SOPs) for addressing new problems presented by COVID-19. This is not a comprehensive list, but it provides development teams with a jumping-off point for user testing programs.
Break Down Your User Study Pipeline Into Three Categories
The COVID-19 pandemic will not stop all user testing, but it will be necessary to manage user testing by breaking down current user testing studies into three categories. If you have user studies that fall into these categories, trim your testing vendor list down immediately. Retain only vendors with extensive, demonstrated experience in business-critical testing and who have a robust SOP structure for how to proceed in light of COVID-19.
Some firms have had major disruptions before, or have conducted complex user testing under adverse conditions with difficult user population requirements. This is not a time for cost containment, but for risk mitigation, in terms of study execution.
Category 1: Study Must Proceed As Soon As Possible With Actual Respondents In A Lab-Based Setting — Critical HFE validation studies are going to continue, just as important clinical trials will still move forward. Scheduled device or combination product validation studies under the FDA will proceed, but under much stricter procedures, and conducted only by firms with experience in complex user testing and validated SOPs that address the special problems presented by COVID-19.
Complex user testing studies have been conducted within the time domain of COVID-19 in the United States without virus exposure to respondents, facility staff, or the research team. However, special procedures are required to mitigate COVID-19 risks that cannot be overlooked. Research vendor selection is vitally important if you must move ahead with a critical summative study. As noted above, this is not a time for cost reduction in user testing.
Category 2: Study Can Be Postponed, But Still Requires Respondents In Lab-Based Setting — Most studies likely fall into this category. These include formative studies, early development cycle concept testing, IFU optimization, and ongoing sequential formative studies. These studies will eventually require more robust testing methods (as described in Category 1), but can also benefit from the gradual reduction in risk associated with improvement in the overall COVID-19 emergency.
However, you should still proceed with preparation of requests for proposal (RFPs) and related planning for these studies. In this time of confusion and increased risk, it is vitally important to keep your testing vendors and teams aware of the status of studies still in the pipeline, and generating RFP documents is a good way to achieve this.
It warrants mention that there has been talk of moving medical device user testing sessions to online methodologies. The adoption of online methods will not produce the level of insights required to objectively map critical use errors to root cause. Without this linkage, which is provided in lab-based user testing, such user testing will be mostly a waste of time and resources. Plan Category 2 projects for later execution dates to reduce risk and manage costs. Do not waste time attempting to migrate such studies to online-based methodologies.
Category 3: Study Objectives Can Be Met Using Other HFE Research Methods That Do Not Require Lab-Based Testing — This third category is an opportunity for device development teams to develop a deeper understanding of testing methods outside of lab-based testing, including third-party heuristics analysis.
Indeed, many human factors problems encountered in formative and even summative studies can be resolved earlier in device development through the integration of HFE heuristics analysis. Now is an ideal time to pull together early stage medical device concepts currently in progress and work with human factors professionals to adopt heuristic analysis or other new testing methodologies. Explore whether such practices could reduce your operation’s development cycles and costs associated with later user testing.
Now also is an excellent time to objectively parse the critical HFE performance variables related to your medical device concepts and determine whether existing research literature, combined with heuristic analysis, can improve device HFE performance prior to any user testing.
In summary, now is a critical time to optimize your user testing and HFE research pipeline, vendor relationships, and development methodologies. Major lab-based studies are going to proceed, as is formative research.
However, the best way to think about testing in the age of COVID-19 is that it will require far more professional human factors expertise. Additionally, integration of professional HFE expertise early in the medical device development cycle may end up benefitting programs in major ways, including increasing device safety and decreasing user testing costs.
About the Author
Charles L. Mauro CHFP, is Founder and President of MAURO Usability Science founded in 1975. Today, his firm specializes in advanced human factors research and optimization for medical devices and related technology. Over his 47-year career, Mr. Mauro has managed more than 3,000 HFE research and development projects. He is a certified human factor engineering professional and has received awards and citations for human factors research from Human Factors and Ergonomics Society, Industrial Designers Society of America, NASA and the Association for Computing Machinery. His clients include many leading medical and pharmaceutical entities and startups. Mr. Mauro has testified as an expert in product development and human factors science in more than 75 major cases. He has been an invited speaker at the USPTO, FDA, NASA, MIT, Stanford, UPenn and many other leading academic and research institutions.