Navigating The Ethics Of Product Validation

Medical device development is a complex and costly process affecting many stakeholders, including the human factors consultants tasked with assessing a product’s safety and usability. Competing priorities can potentially lead to ethical conflict between consultants and device manufacturers, especially when development is near-complete. Consider, for example, these cases — all sample situations I’ve experienced, to some degree, while working in the human factors consulting industry.

Case #1: A manufacturer, having completed no previous human factors work, approaches a human factors consultant for assistance with a validation study. Upon initial review of the company’s device and documentation, the consultant identifies potential use-related hazards not previously considered. Because the manufacturer cannot tolerate lengthy delays and the high costs associated with redesign at this stage, they want to move forward with a study protocol that omits testing of some potentially hazardous use scenarios.

Case #2: A human factors consultancy intends to validate a complex device that involves user training provided by the device manufacturer. After beginning the validation study, the consultancy observes a failure rate for certain high-risk scenarios. To keep the project moving, the manufacturer decides to adapt its training program to emphasize the failed scenarios (in an attempt to reduce the failure rate).

Case #3: A product’s human factors validation study has been completed, and the minimum standard of 15 users has been exceeded. Some team members suggest omitting data from two of the worst-performing participants to bring the number of users down to the required minimum.

Case #4: A previously undetected safety-critical error is discovered during the validation testing pilot run. Believing the error is unlikely to occur during the actual test, the manufacturer elects to move forward. After completing the study without occurrence of the error, the human factors consultant is unsure whether to include discussion of this potentially safety-critical problem in his/her reporting to the FDA.

Unfortunately, these types of ethical dilemmas arise with some regularity. Human factors consultants serve as a bridge between the FDA and medical device manufacturers, utilizing industry knowledge to help the manufacturer navigate the human factors regulatory process, as well as optimize designs for safety and usability. Ethical conflicts can arise when there is a disconnect between the study plan that the manufacturer believes is reasonable to achieve sufficient validation, and what the consultant believes is needed to conduct a robust study without sacrificing the integrity of the human factors process. To confront these issues and think about how to avoid them, we first need to think about the causes.

Mutual Misunderstanding

A lack of familiarity or mutual understanding regarding the human factors process can increase the probability of conflict between human factors professionals and medical device developers, and these dilemmas can start before the human factors work even begins. Manufacturers new to the process sometimes fail to see human factors as an integral part of development, viewing it instead as a quick validation stamp just before approval. Ideally, every stage of development should include human factors activities to maximize usability and ensure a new device fits well into the eventual work system. Some of these activities include interviews with end users, ethnographic studies, heuristic evaluations, usage-related risk analyses, and formative stage user testing.

When a device is market-ready, a human factors validation study is intended to be the final step in a robust human factors process. However, because validation is the only strict FDA requirement, the above activities — which should be considered vital to the process — can be viewed as unnecessary, time-consuming, and not worth the cost. When manufacturers come to human factors consultants at this late stage, it is possible that potentially critical usability issues remain embedded in the design. Then, when problems are discovered by consultants, ethical questions arise between meeting a paying client’s needs and ensuring that a device is going to be safe and usable on the market.

Contributing Factors To Conflict

The FDA released its final guidance on incorporating human factors and usability into medical device design in 2016. Although the ideas highlighted throughout the guidance are extremely useful, the content can also be a contributor to ethical conflicts. Because much of the document is written in a non-specific, recommendatory manner, it is open to interpretation by both the FDA and device developers, and this can cause confusion and potential conflict.

For example, in Case #2 above, the manufacturer wants to use the device’s training program to mitigate previously observed failure during the validation study. Despite the guidance discussing this as a poor technique for error mitigation, unless this approach is explicitly described  in the human factors report, the FDA would have no way of knowing that trainers emphasized certain tasks in a manner inconsistent with real-world training in an attempt to “pass” the study. Band-Aid fixes like these can hide a design deficiency enough to complete a successful validation study, but can leave a consultant conflicted between professional and ethical obligations.

Unsurprisingly, these problems are exacerbated by timeline constraints and development costs. When a critical usability issue is discovered so late in development, a manufacturer may not be able to afford the delays and costs associated with design modifications. Although both the consultant and the manufacturer are concerned about patient safety, pressure from stakeholders can lead things down the wrong path.

With inflexible time constraints and investor deadlines often dictating decisions at this point in the process, people on both sides may be forced into undesirable compromises. Consultants can be put in the uncomfortable position of deciding between allegiance to the device company engaging their services and the safety and efficacy of the device. Such pressure makes it easier to convince ourselves that the device is safe enough, and that these errors are unlikely to occur once the device reaches the market.

How Manufacturers Can Avoid Conflict

The easiest way for manufacturers to avoid such conflict is to thoroughly understand the benefits of human factors. Although that could be considered biased advice coming from a human factors engineer, the field is rapidly emerging as a vital part of development that the FDA takes very seriously. Rather than considering it another painful step in the process, the manufacturer should think of human factors as a vital tool to create the safest and most usable design.

Going the extra mile from a usability perspective can give a device company the edge over its competition, as well as reduce cost by increasing the probability that safety and usability issues are caught early, before designs are finalized and the costs of redesign skyrocket. Even if faced with a decision that leads to delays, manufacturers can use the time productively, rather than pushing forward with a suboptimal product. Conducting additional formative testing, further evaluating instructional materials, and refining training programs all are good ways to spend redesign time.

Another key to avoiding ethical conflict is understanding the constraints of human factors consultants, and approaching consultants with an expectation of honesty. If manufacturers want to make sure that studies are run within acceptable ethical boundaries, and meet FDA expectations, they need to understand that some of the adjustments described above are unacceptable. Although some consultants may bend to demands, working with this type of firm won’t necessarily result in a positive outcome.

Finally, it is important to recognize that the understanding of constraints goes both ways. Consultants should be sensitive to the vast amounts of time and money invested in product development, and attempt to meet the needs of the manufacturer, as long as any compromises are transparent to both parties, as well as the FDA.

Avoiding Conflict for Consultants

Human factors consultants must not forget that we have an obligation to patient safety and a shared responsibility for how devices operate once they reach the market. The best thing we can do in these situations is to hold our ground, refusing to compromise ethical standards that, if bypassed, could result in patient harm. That can be easier said than done, and the ethical line isn’t always clear.

When we look at some of the examples above, situations like Case #1 (omitting evaluation of high-risk scenarios) and Case #3 (omitting unfavorable data) are more cut-and-dried regarding the standards that should be upheld. Case #4, however, presents a more ambivalent situation in which a consultant has been hired to execute and report on a study, but has knowledge outside of the study results that could be critical. Although taking a stand in these situations could result in severing a relationship with a client, consultants who take on work that crosses ethical boundaries hurt the industry as a whole.

If a consultant refuses work due to ethical concerns, a device manufacturer willing to overlook safety issues likely will find another human factors firm to complete the work in the manner they are requesting, and the new consultant may be unaware of past ethical concerns. Even good human factors consultants can miss these critical issues if they aren’t discussed openly. By requesting review of risk assessments, previous user studies, and other documentation before conducting work with companies looking for validation studies only, consultants can try to determine for themselves whether the manufacturer legitimately is ready for validation.

Human factors firms inexperienced in FDA requirements may be less apt to catch these red flags, so they should consult with more experienced industry experts to ensure potential problems are identified. One thing to remember is that, while it might be tempting for consultants to take these contracts, running a poorly designed study and helping to secure approval for an unsafe product has the potential to hurt all parties (and the profession) in the end.

As unfortunate as these situations can be, it is important to note that the majority of medical device manufacturers are committed to creating safe products. As more companies experience the validation process and the human factors requirements become ingrained in development, we can hope that conflict occurs less frequently. The keys to avoiding conflict are a mutual understanding of shared safety and usability goals, and honest communication. If both parties can understand the expertise and constraints of the other side, together we can generate optimal designs without compromising ethical principles.

About The Author

Natalie is the Program Manager for the Usability Services division of the National Center for Human Factors in Healthcare. She oversees the technical and quality aspects of usability projects conducted  both for the medical device industry and within MedStar Health. She is involved in all aspects of planning and executing usability tests, and also leads an initiative to incorporate usability testing in medical device procurement.

Natalie holds a master’s degree in industrial engineering, with a focus on human factors and ergonomics, from the University of Wisconsin, where she was mentored by Dr. Ben-Tzion Karsh. Some of her previous work involved research on primary care redesign for the aging population and implementation of process improvement efforts in the ambulatory care setting.