Understanding Costs And Risks For HFE Usability Studies — Part 1: Testing In-House
By Charles L. Mauro, CHFP, Chris Morley, and Kim Dalton, Mauro Usability Science
This article is the first of a two-part series; read part two here.
The U.S. Food and Drug Administration (FDA) is increasing demands[i],[ii] for drug delivery device developers to demonstrate the human factors engineering (HFE) effectiveness of their devices and related labeling systems. Therefore, rigorous HFE usability testing is required to ensure a given medical device meets HFE standards prior to submission for FDA approval.
In light of these increasing demands, medical device developers and their clinical teams are faced with two critical questions: First, how to objectively understand and manage the costs and risks of such studies and, second — given the costs, risks, and complexity of such studies — whether they should conduct HFE studies internally or retain a formal HFE testing agency. The answers to these two questions are more complex than most development teams understand.
The first question device developers must ask themselves is whether to conduct a formative or summative study. At the highest-level, only formative studies should be considered for assignment to internal HFE testing resources; it generally is considered industry best practice to employ an independent HFE testing agency to carry out summative studies for FDA submission. Therefore, the issue of whether to test internally or outsource is essentially limited to early development formative studies. Below is an objective description of the costs, risks, and benefits of conducting robust formative studies internally.
Factors Related To Executing HFE Formative Studies Internally
Well-understood requirements impact studies executed internally, which should be balanced against the cost of retaining an outside HFE testing agency. Most device development teams have minimal awareness of the following critical factors impacting internal HFE study execution:
Controlling For Respondent Bias Based On Study Location — HFE studies conducted at any facility that can be identified with your company name are immediately subject to an increased likelihood of respondent bias; study data can be skewed positively or negatively based on the respondents’ internal expectations and/or opinion about the source of the device being tested. This factor is well-understood to impact the quality of study data.
HFE studies conducted with source identification and/or at corporate locations generally are considered invalid from a research protocol perspective. Therefore, you should only test in an anonymous location, such as a certified testing center, with the research sponsor blinded to respondents. Studies executed at an on-site HFE study lab will be subject to a wide range of legal and technical requirements.
HFE studies conducted with source identification and/or at corporate locations also are likely to receive significant pushback from the FDA or other governing bodies. Top-tier independent testing agencies work only with approved and fully-vetted testing facilities designed to maintain anonymity and data security. Such testing facilities are costly and require constant management by independent HFE testing firms.
Thus, if you decide to conduct in-house HFE testing at an outside lab, you will need to allocate considerable time to coordinating with the testing facility, as they are mostly staffed by inexperienced, low-paid administrative staff.
Proper Insurance For Testing With Human Subjects — With internally run HFE studies, it is critical that the device development team and those assigned to HFE testing with actual respondents possess all forms of required insurance related to testing with human subjects. Such policies normally are not included in the insurance riders for corporations, and these highly specialized forms of insurance can require special brokers. Never execute any HFE testing without fully verified and documented insurance.
It probably goes without saying, but make sure your accounting department did not let your insurance lapse. This is especially important if you test infrequently and do not have alerts set for your corporate insurance policies.
This form of insurance also is a major expense for independent HFE testing agencies and is fully required under all master services agreements between corporate clients and HFE agencies. Verification is required every year.
Corporate Legal Approvals — Most development teams considering internal HFE studies fail to secure full legal approval, leading to serious problems with their legal department. Thus, secure full approval from your corporate legal department, or outside counsel, with respect to all respondent release forms, recruiting criteria, non-disclosure forms, and related testing methods. You also must have in place Health Insurance Portability and Accountability Act (HIPAA) and respondent confidentiality systems and procedures.
Data Storage And Data Security — This includes management and storage of all respondent demographic information, recruiting criteria forms, video recordings, and related data capture files. If you have a corporate legal team that understands FDA and government data integrity requirements, you will have to put in place an approved data storage system that meets such requirements.
For example, you cannot store respondents’ informed consent forms, testing session video recordings, or other respondent data on your laptop computer, office server, or a cloud-based service (e.g., the consumer-facing Dropbox). Establishing systems that objectively meet such standards adds to the cost and complexity of maintaining independent testing agencies.
Establish And Maintain Approved HFE Testing Standard Operating Procedures (SOPs) — Internally run studies depend on a comprehensive set of HFE testing SOPs directing the execution of said studies; these SOPs include how to deal with all matters related to respondent safety, confidentiality, data capture, data storage, analysis, and reporting, to list only the main categories. Your corporate risk management and compliance team will have a specific set of directives related to these requirements.
These SOPs must be maintained and updated on fixed schedules and should be audited by an external source for compliance to relevant state and federal guidelines. Maintenance of SOPs may seem unnecessary, but should any problem or question arise related to your internal studies, your SOPs will be audited immediately. For independent testing agencies, the creation and maintenance of SOPs is a complex and time-consuming process. All major clients who employ top-tier testing agencies audit their selected agencies’ SOPs.
Creation and Approval of Study Protocols — Studies that effectively assess a given drug delivery device or related medical product’s HFE performance must be based on professional testing protocol documents created and approved by management and the clinical team. Such documents require explication of all study aspects, and can be complex and time-consuming for those who do not routinely create them.
IRB approval is not possible without a professional HFE testing protocol. Even simple formative studies should have a management-approved protocol that has been reviewed for risk by the compliance team. Independent testing agencies that work on large, complex formative and summative studies normally are skilled at creating and managing study protocols — a natural by-product of them frequently creating such protocols.
IRB Approval Required — Formal IRB approval for HFE studies, even at the formative stage, generally is considered required professional research practice. Still, some device development teams frequently ignore this requirement — at great risk. Should there be an incident during testing, IRB approval would be the first line of defense for your team and your employer.
The tasks required to set up, manage, and execute all variables related to IRB approval involve a complex, rigorous process that requires filing and closure procedures. Internal HFE testing teams that require IRB approval must have a formal relationship with an outside IRB agency (professional HFE testing agencies all have formal relationships with IRB entities), and have all related corporate signatures and approvals in place prior to submission for IRB review. Finally, IRB approval is nearly impossible without a robust testing protocol document.
Properly Trained HFE Moderation Staff — HFE study quality is, in large measure, determined by the quality of your study design and the training of your professional moderation team. Simply allowing a device engineer to perform HFE moderation and data capture generally is considered poor research practice, and often leads to low-quality data. Therefore, all internal HFE study team members should have formal education/training in HFE science and related HFE moderation methodologies. This is an essential aspect of obtaining reliable data.
Professional Recruiting Of Respondents — It is critical that respondents testing the device are as similar as possible to the intended device user(s), ensuring that observed behaviors are representative of real-world users’ behaviors. Additionally, if the device has multiple user groups, effort must be made to sample respondents from each user group, giving testers a comprehensive understanding of the types of errors/issues that may occur with different user groups.
Recruiting the right respondents can be extremely time-consuming, especially if pools of potential respondents have not already been identified. Thus, it often is necessary to contract a professional recruiter, as they typically have a head-start on knowing where to find the right respondents. If a professional recruiter is brought in, your development team will have to provide the recruiter with sample sizes and screeners for each user group, as well as a desired data collection schedule.
The recruiter then will locate and compensate the respondents, but this process is costly, with fees accumulating for recruitment, respondent incentives, and referral fees (in the case of particularly difficult-to-find user groups). Nevertheless, such costs often are necessary to obtain the right respondents, reduce the time internal staff must commit to recruitment, and maximize the quality and generalizability of study data.
Never Test with Colleagues, Friends, or Family — This well-understood requirement is inflexible if you hope to have reliable data. Using any internal employee as a test subject generally is considered poor research practice. Even in the quickest studies, one should never test with friends, family, or colleagues.
Emergency Procedures — Proper emergency procedures are critical to complying with human subjects research best practices, as well as gaining IRB approval for your study. Moderators must be properly trained in minimizing the likelihood of emergencies, and handling emergencies when they do occur. Moderators always should be in a position to intervene if a participant somehow places themselves at risk for harm during a session.
If an accident — such as a needle prick — does occur, moderators must know the appropriate actions to ensure the respondent’s safety. First-aid kits should be in the testing room at all times to deal with such mishaps, and SOPs should be established for handling both minor and more serious emergencies.
HFE testing with a live-drug involves a massive, complex set of additional concerns for ensuring respondent and moderator safety, such as arrangement of on-site medical support, personal protective equipment, and hazardous waste disposal. Establishment of comprehensive emergency procedures is essential to proper study execution.
Time Required To Conduct Data Analysis — This factor often is overlooked by teams planning internal HFE studies. Internal staff review of session videos to verify recorded behaviors, and subsequent data analysis, will take time away from other job functions once the study data has been obtained. This can take staff members away from their primary roles for a significant period of time, or delay generation of the final report if the study becomes a side project conducted only when staff members have spare time.
For example, data analysis for 45-minute sessions with 20 respondents can take between 40 and 80 hours, depending on the complexity of analysis and study design. Device development teams must consider whether they have sufficient internal bandwidth to execute data verification and analysis during the time period in which study results must be delivered.
Reporting Study To The FDA — Although both formative and summative studies may be submitted to the FDA for review, summative studies are explicitly conducted with the intention of demonstrating a device’s HFE effectiveness for FDA approval. Study reports submitted to the FDA should detail who conducted the study, the qualifications of the team conducting the study, and the facility where the study was conducted.
It is possible that the FDA will question the objectivity of studies conducted internally. Thus, development teams executing studies internally must ensure — to the extent possible — that all study stages are conducted without bias, and that all HFE testing procedures adhere to best practices.
Inherent Bias Of The Device Development Team — An inherent bias exists when internal staff conduct HFE studies, due to their extensive knowledge and history with the product and its development. This bias can lead internal staff to misrepresent observed behaviors during coding of respondent behavior, root cause of error analysis, and reporting of results. Designing the study protocol and moderator’s script to objectively and neutrally assess device performance, as well as utilizing moderators properly trained in HFE moderation methodologies, are critical steps to minimizing such bias.
Summary
Executing HFE studies internally often is far more complex and time-consuming than device development teams realize. Many factors must be considered to maximize cost-effectiveness and produce high-quality data appropriate for decision-making and HFE optimization.
As you consider the complexity, risk, and cost trade-offs between internal execution of HFE studies and utilization of an independent HFE testing agency, it is critical to understand that properly run HFE studies, using internal resources, are unlikely to lead to significant cost savings, unless you are willing to take equally significant risks — risks likely to be flagged by your corporate legal department or the FDA when reviewing your data. The bottom line is simply that there is no discount methodology for producing high-quality HFE research data.
If you decide to retain an independent HFE testing agency, there are numerous factors affecting the cost. In part two of this series, we will describe in detail how device development teams can best understand and manage these costs, should they decide to retain an independent testing agency.
About The Authors
Charles L. Mauro, CHFP, is Founder and President of MAURO Usability Science, a consulting firm specializing in advanced HFE research and optimization of complex medical devices and related technology. He is Chairman of the Design Protection Section for the Industrial Designers Society of America and is the IDSA liaison to the US Patent and Trademark Office. He has lectured on product design and HFE research at MIT, Stanford University, UPenn, and other leading human factors research and engineering programs. Over his 47-year career, Mr. Mauro has managed more than 4,000 HFE research and development projects. He holds a BS with distinction in Industrial Design from The Los Angeles Art Center College of Design and a master’s degree in Ergonomics and Biomechanics from New York University. At NYU, he was appointed NIOSH Research Fellow at the RUSK Institute of Rehabilitation Medicine.
Chris Morley is a Senior Human Factors Engineer at MAURO Usability Science. At MAURO Usability Science, he has managed several complex medical device HFE research studies for top-tier device development teams. Mr. Morley holds a master’s degree in Experimental Psychology from Old Dominion University and specializes in experimental design, statistical analysis, and human factors psychology.
Kim Dalton is a Research Associate at MAURO Usability Science, specializing in various data-collection methods, including online surveys, eye-tracking, galvanic skin response (GSR), and micro-expression facial analysis. Ms. Dalton holds a BA in Neuroscience from the University of Colorado.