Guest Column | February 8, 2021

ISO 14971:2019 — Clarifying Benefit, Risk, & Benefit-Risk

This is the third in a series of articles on the changes in the medical device risk management standard ISO 14971 that were released in December of 2019 and supported by guidance in the ISO TR 24971:2020 technical report or guidance on the medical device risk management standard. The standard and the technical report were reorganized and clarifications of the concepts of medical device risk management were included in the two documents. Few changes in the process resulted, but this article covers the change in the process for benefit-risk covered in Clause 8 of the standard. The first article in this series covered the risk acceptability criteria and policy. The second article covered production and post-production information.

The Evolution Of The Terms “Benefit” And “Risk” In ISO 14971

Discussion of the term “benefit” first appeared in the first edition of ISO 14971, which appeared in 2000. The requirement of controlling risk allowed the use of “benefit” as an alternative method of releasing a product when a residual risk was evaluated as “unacceptable.” Residual risk is that risk remaining after risk controls are applied, so, in this case, all possible risk controls were applied, and the residual risk still could not reach acceptable levels. The product should not be put on the market if the benefit does not outweigh the risk in a documented benefit-risk analysis. It would be up to regulators to ask the question, but you should not move product forward if the benefit does not outweigh the risk or until further risk reduction could be made.

One of the dramatic examples of the use of risk-benefit evaluation was the original mechanical heart, which had an associated mortality rate of 40% when it was used. This risk was considered unacceptable until measured against the benefit. The benefit was that the mortality rate was 70% without the use of the device. Clearly, the improved benefit outweighed the risk, and the device was ultimately cleared for use by FDA. However — and here is a consideration that must be part of postmarket review (see previous production and post-production article) — in less than a year, what was considered “state of the art” (another ISO 14971:2019 defined term in 3.28) changed and a competing product was released with an associated 30% mortality rate. Now, the original product was no longer acceptable — not because it was defective or the risk changed, but because the state of the art changed and a safer product was available. The original product had to be withdrawn until changes could be made to reach the new state of the art.

Note that the term “risk-benefit” was used in the 2000 edition and the 2007 edition of the standard. In the 2019 edition of the standard, you will find the term is now “benefit-risk” because regulators felt that manufacturers were only looking at benefit as an afterthought and thus wanted to emphasize benefit first instead of risk. Nowhere in regulation or standards will you find this term defined; however, you can find definition of “risk” in ISO 14971 (now at 3.18 in the 2019 version) going back to the original edition. In the standard, the definition is and has been “combination of the probability of occurrence of harm and the severity of that harm.” This definition was adopted by the regulators and appears in the EU’s Medical Device Regulation (MDR) and In Vitro Diagnostics Regulation (IVDR).

The FDA Quality System Regulation (QSR), 21 CFR 820, appeared several years before ISO 14971 and did not have “risk” as a defined term in the regulation. However, FDA was aware that the state of the art was changing. In the Preamble to the QSR in Comment 83, FDA indicated it was participating in ISO Technical Committee 210, working in the area of medical device risk. The FDA is currently considering updating its regulation to align with ISO 13485, in which the term “risk” carries the ISO 14971 definition.

In Essential Principles of Safety and Performance of Medical Devices and IVD Medical Devices, IMDRF/GRRP WG/N47 FINAL:2018, the International Medical Device Regulators Forum (IMDRF) uses the same “risk” definition as ISO 14971 in 3.35.

So far, we have half the equation, “risk,” well defined. The term “benefit,” however, had been undefined by regulators or in standards until ISO 14971:2019. Here, a couple of things happened:

“Benefit” is now defined, in 3.2, as “positive impact or desirable outcome of the use of a medical device on the health of an individual, or positive impact on patient management or public health.” A Note 1 to Entry expands information to help clarify the implications of “benefit.”
Since then, regulators, including FDA (see the four guidance documents in Refs. 1–4) and the European Commission (EC), adopted the term “benefit-risk” instead of “risk-benefit” as was previously used in the standard. ISO TC 210 JWG1, the authors of the risk management standard (ISO 14971) and technical report (ISO TR 24971), updated the standard and technical report to the new term for the new releases in 2019 and 2020, respectively.

We now have defined terms updated to current regulatory thinking, in the ISO 14971 standard and ISO TR 24971 technical report, so we now understand what we are talking about. Of course, we defined “risk” and “benefit” individually, and not as a combined term, but we have some understanding of the terms.

The 2019 edition of the standard indicates in 7.4 Benefit-Risk Analysis that “benefit-risk analysis” may be used to address unacceptable individual residual risks. It refers to ISO TR 24971’s discussion (via guidance, not requirements) of some aspects and possible methods of this analysis. This section only deals with the individual risks, and the standard (7.4) only requires that individual unacceptable residual risks undergo this analysis. Some regulatory bodies may require that all individual risks, regardless of acceptability, undergo this analysis. This is not in conflict with the standard, so you must apply this regulatory requirement in addition to the requirement of the standard if you are serving a market where this additional requirement exists. How to perform these analyses is beyond the space allowance we have here, but the discussion referred to earlier in 24971 is more than we could successfully perform here.

For individual risks, then, the 7.4 requirement in the standard only addresses individual unacceptable risks, not all risks. Regulators may impose additional requirements. However, we need to go further in the standard to uncover additional requirements for “benefit-risk analysis.”

How To Evaluate Overall Residual Risk

In ISO 14971’s Clause 8, Evaluation of Overall Residual Risk, we find a universal requirement applicable to all devices, regardless of risk acceptability. The standard requires, “the manufacturer shall evaluate the overall residual risk posed by the medical device, taking into account the contributions of all residual risks, in relation to the benefits of the intended use…” (emphasis mine). This means for every device, an overall residual risk evaluation shall consider the benefit compared to the risk of using the device. The benefit should outweigh the risk. Fortunately, ISO TR 24971:2020 includes 2.5 pages of guidance on methods you can use to perform this analysis. The technical report in 8.3 a)–f) provide some possible approaches, indicating there is no preferred approach. The evaluators should be experienced in the use of the device and the environment of use, which are both critical aspects of such an evaluation. One of the key tools I have used in this type of risk evaluation is a visual tool presenting all of the risks of the device simultaneously on a risk chart so that it may be easier to understand the device’s risk profile in such an analysis. ISO TR 24971 8.3 b) explains this approach to using visual representations of residual risks to understand where problems may exist in the overall residual risk of the device and where efforts can be applied to give the best results on reducing residual risk.

ISO TR 24971’s Clause 8.1 discusses that there is no preferable way to perform overall residual risk analysis, and that judgment is involved. As a result, it is essential that professionals with knowledge, experience, and authority (including application specialists and clinical specialists or users) perform this evaluation. In my experience, a team that includes medical professionals current with device experience, product liability insurance experts, experienced design members, and risk management experts not associated with the project would provide the needed expertise to perform this evaluation. Adequate preparation, including review of the entire risk management file for the device prior to conducting the overall residual risk evaluation and review of documentation of comparable devices, is necessary and may provide valuable information. Additionally, the overall residual risk acceptability criteria provided for the device might be different from the individual residual risk acceptability criteria. This criteria must be included in the risk management plan for the device prior to conducting the evaluation. ISO TR 24971 Clause 8.2 provides information on the inputs for the evaluation.

An important aspect of the requirements in Clause 8 is the requirement that “the manufacturer shall inform the user of significant residual risks, and shall include the necessary information in the accompanying documentation.” A limited discussion of the rationale about what is significant for this disclosure appears in ISO 14971 Annex A.2.8.

It is important to understand that only Clauses 1–10 of ISO 14971:2019 contain requirements. All other parts of the standard and all parts of ISO TR 24971 are only guidance to provide aid in constructing your risk management system. Those sections are not requirements. This misunderstanding has caused a great deal of problems for those implementing the standard in the past.

That covers the risk side. Now for the benefit side of the equation.

How To Evaluate Benefit

While engineers love numbers and want to assign values to such an analysis, it is not possible to do this numerically for benefit, because it consists of many independent factors. This effort should involve medical/clinical risk management team members, as they are more qualified to perform benefit analysis. Additionally, the evaluation team must consider any alternative treatment modalities, including pharmaceuticals and biologics, when considering benefit-risk. If the alternative treatments provide the benefit at a lower risk, then the necessary benefit-risk may not be achieved to place the device on the market. ISO TR 24971:2020’s Clause 7.4 discusses benefit-risk analysis, with some additional information is provided in Clause 8.3 a) and c).

First, for a more complete understanding of benefit, the definition of the term in ISO 14971:2019 Clause 3.2 is essential:

benefit
positive impact or desirable outcome of the use of a medical device on the health of an individual, or a positive impact on patient management or public health

Note 1 to entry: Benefits can include positive impact on clinical outcome, the patient’s quality of life, outcomes related to diagnosis, positive impact from diagnostic devices on clinical outcomes, or positive impact on public health.

ISO 24971:2020 7.4.2 mentions factors necessary to consider when evaluating benefit, such as:

Expected device performance
Expected clinical outcome at that performance level
Competing devices
Alternative treatment modalities.

Confidence of the benefit estimate is important and is based on reliability of information. One problem is trying to compare different outcomes (pain versus mobility) and short-term versus long-term effects. Other considerations include:

Type of benefit
Magnitude of benefit
Probability the patient will experience the benefit
Duration of the benefit.

Availability of clinical data to support benefit determinations impacts benefit estimates (remember, data prior to getting real data from users after release is just an estimate and has associated uncertainty). ISO TR 24971:2020’s Clause 7.4.5 has three examples of benefit-risk analysis; although they are provided for individual benefit-risk, they can be extrapolated to the necessary overall benefit-risk requirement in Clause 8.

How To Evaluate Benefit-Risk

Notice that in all the discussions of benefit-risk in Clause 7.4 and Clause 8, there is no attempt to perform these steps using numerical values. There are several aspects of benefit, for instance, that cannot be compared on a numerical basis, just as overall residual risk, which has the same issue. These evaluations are provided in a textual rationale in the supporting documents in the risk management file. While the EU MDR mentions “benefit-risk ratio” in one place, this ratio does not exist (if one assumes that ratio means numerical). FDA has written four guidance documents^1–4 on benefit-risk and has provided examples of how the evaluation might be done in each. The examples are all text-based rationale statements with no overall numerical value assigned to the entire benefit-risk. In some areas of the evaluation, some numerical values appear, such as in uncertainty of the data based on sample sizes. FDA does not recognize a “benefit-risk ratio” as a numerical statement.

Once the device is on the market, you need to continue to monitor the “state of the art” to ensure no new device (or other treatment modality) has been released with a greater benefit that would cause your device to no longer have an acceptable benefit-risk based on an improved state of the art.

While “risk-benefit analysis” has been part of ISO 14971 since the 2000 edition, adequate information on what this analysis might look like has not been part of the standard or technical report until the most recent revisions of the two documents, and included the change in terminology to “benefit-risk analysis.” This change was prompted by the regulators trying to emphasize benefit over risk. Now, we have better explanation of the requirement in the extensive guidance provided in ISO 24971:2020. With this expanded information in ISO 24971:2020 about the requirements in ISO 14971:2019 Clauses 7.4 and 8, you should have a better understanding of what activities you need to perform to get your devices through design and development, and then on the market.

References

About The Author

Edwin L. Bills has been a member of ISO TC 210 JWG1 for more than 20 years. This is the ISO group responsible for medical device risk management and the creation and maintenance of ISO 14971:2019, the risk management standard for medical devices, and ISO TR 24971:2020, the accompanying risk management guidance.

This website uses cookies to ensure you get the best experience on our website. Learn more