Risk Assessment, Risk Acceptance Matrix

Each medical device comes with risks. Manufacturers must determine which risks they deem acceptable and which unacceptable. This is usually expressed in the form of a matrix of risk acceptance (or risk assessment matrix).

In this article you can read more about:

  • The regulatory requirements for the risk assessment matrix
  • The typical difficulties and errors when creating the risk assessment matrix
  • Notes on video training in our e-learning library
  • Tips for creating standard-compliant risk assessment of the risk / acceptance Matrix

Regulatory Requirements for the Risk Assessment Matrix

First thing to note is that there is no regulatory requirement for manufacturers of medical products, neither in Europe nor in the United States, to create a risk assessment matrix. However, the risk assessment matrix has been established as a tool to express the risk policy.

Requirements of the Medical Device Directive MDD (93/42 / EEC)

The MDD requires that the manufacturer risks may only be defined as acceptable by medical devices if the benefits of the product outweigh the risks or damages. The benefit is determined by a comparison with an alternative approach such as the non-application of the product, the use of a competing product or an alternative product.

Requirements of the Medical Device Regulation MDR

The requirements of the MDR are more specific and match those of the ISO 14971:2012 much closer as the requirements stated by the MDD.

ISO 14971 and risk assessment matrix

The ISO 14971 does not require a risk assessment matrix, even if it displayed one in older issues of the informative Annex. In these older editions of the ISO 14971 it differentiated between acceptable risks, unacceptable risks, and risks related to ALARP ("As low as reasonably practical"). Since 2012 edition of ISO 14971, there are no fixed acceptable risks anymore. The subdivision into acceptable, unacceptable and ALARP risk is thus obsolete. Risks should be "as low as reasonably possible".

Fig. 1: Risk Acceptance Matrix expresses the Risk Policy

Read more about these changes in the contribution to the ISO 14971: 2012, and the annexes ZA.

Difficulties and Typical Errors While Creating the Risk Acceptance Matrix

Already during the first step in risk management, namely the definition of risk policy (Expressed in the risk assessment matrix as a distinction between red and yellow areas), numerous errors occur for  medical device manufacturers that absurdity lead to all the other activities.

1. Error: Risk Assessment Matrix in the QM manual

The risk-benefit analysis and thus risk assessment matrix must be product-specific. Therefore, it generally does not make sense to define the risk assessment matrix in the QM manual or in a "risk SOP management".

2. Error: Risk Acceptance is Not Derived Quantitatively

Many manufacturers instinctively set risk politics. A typical indication for this is when they determine the acceptance criteria based on a risk priority number. But that does not make sense for the most part because

  1. The number that separates red and yellow areas, cannot be systematically and quantitatively derived. 
  2. The separation of red and yellow areas would have to be discussed in the risk assessment matrix for each severity. It applies for each severity class to estimate the benefits and risks separately.

A risk priority number, which can incorporate three parameters (e.g. severity of damage, likelihood of errors, probability of detection) contradicts the definition.

3. Error: The Risk Policy and Risk Assessment Matrix are Not Updated

One of our clients created a stand-alone software that runs on genetic data of patients and from literature data derived based on concrete treatment recommendations. And we came across a rare phenomenon:

The risk acceptance matrix must define acceptable and unacceptable risks on qualitative criteria, as with all manufacturers as defined in ISO 14971. The acceptance depends on whether the process and the product are better or worse than the alternatives. In this specific case it would be the treatment in ignorance of the literature situation that can handle only one software in the existing amount.

However, since the literature situation constantly changes - and improves in this case - better data for decision treatment is always available for the physicians. That means that the software must be measured with an increasingly sophisticated alternative. If it doesn’t keep up, risks caused by the software would be increasingly less acceptable.

In other words, an ISO 14971-compliant risk acceptance matrix of this manufacturer must be very dynamic and be colored from upper right to red without continuous improvement of the product. The challenge for us in risk management is to quantify this improvement.

Training Video on Risk Management

The E-Learning Library shows step by step how to create a risk acceptance matrix:

  1. Define severity axis, specify criteria for severity axes
  2. Define probability axis and probability classes
  3. Formulate risk policy and derive acceptance criteria quantitatively

Tips for Creating the Risk Assessment Matrix

Severity axis

In theory it is easy to determine a risk assessment matrix. Two axes, one for each the severity and likelihood of damage. For both axes a couple of categories are drawn and then the right upper area is defined as an unacceptable risk and the bottom left as acceptable. And that’s that!

Unfortunately, in practice this is not quite that simple. Already in defining the severity axis there are some challenges to be observed. For example, it is not enough, to write a few descriptions such as insignificant, light, heavy, critical and catastrophic. You need to have clear classification rules.

For example:

  • Death (yes or no)
  • Life critical damage (yes or no)
  • Medical intervention required (yes or no)
  • Reversible injury (yes or no)

Furthermore, there are still open questions: What is worse, life-threatening-non-reversible damage or non-life-threatening-non-reversible damage? A lost thumb or a permanently disabled leg? Plague or cholera?

Once again it becomes clear: The definition of these ethical principles is not a task of development, but a management. This is exactly what the  ISO 14971 calls for. 

Probability axis

When we at the Johner Institute create a risk assessment matrix, we almost always include five or six categories for the probability axis, each of which comprises of two orders of magnitude.

For example, observe the following definitions:

TermDescriptionFrequency (per treatment)
OftenOne- or more times per treatmentx > 1
ProbablyMay occur with normal use10-2 < x < 1
OccasionallyOccurs in irregular intervals several times per month/year10-4 < x < 10-2
Remotely possible One or several times per lifetime of the medical product10-6 < x < 10-4
UnlikelyNot during the lifetime of the medical product10-8 < x < 10-6
ImpossibleOnly achievable through high forcex <10-8

The explanation, as to why there are two orders of magnitude (i.e. a factor of 100), is the following: It is often necessary to cover eight and more orders of magnitude. And in five or six categories you just go to these 100 factors. 

But since the workshop on risk management in the MedConf I know from a subscriber (derived from a notified body) a second, almost simple justification: the factor 100 indicates the precision with which we can appreciate. The participants made it clear that with the example of a hard disk: If you ask a group of people, how long it takes on average, for a hard disk to be defective, the estimates vary between 2 years and 10 years. But everyone realizes that this average is greater than one month and less than 10 years. And between these two values is about the 100 factor.

Classification of the Axes of the Risk Assessment Matrix

These models are suitable only for the members of my narrow self-help group "Risk Management".

I have already said that I currently get a lot of risk management documents for testing. One of my first glances often applies to the risk acceptance matrix. So I look for example, how broad the categories defined for severities and probabilities and, for example, which probabilities are discussed.

The severity and probability axis of a risk acceptance matrix must be "wide" enough.

If our consultants notice that a probability class includes a certain magnitude or a lower one, we will suspect that the manufacturer has not discussed the complete relevant probability range. Let's explain that with one extreme example:

  • Common: p> 10E-3
  • Uncommon: 10E-4 <p ? 10E-3
  • Rare: 10E-5 <p ? 10E-4
  • Inconceivable: p ? 10E-5

First, we calculate how many applications there are. Assuming these are 10E8. Then we know that the supposedly unimaginable event happens 1000 times or less. This means that the manufacturer is not able to differentiate between damages, for example, with fatal consequences that statistically occur 1000 times or 0.0001 times. This is absurd. On the other hand, they would be able to differentiate between probabilities that occur 10E-4 and 10E- 5 times. Mostly one cannot approximate so accurately. 

Graphically expressed (see Fig.), One could say that these manufacturers buy themselves an unnecessary ability to differentiate with average probabilities (analogous grades) of the "border areas".

An exaggerated formulation would be that such files stem from a certain lack of understanding of risk management tools. Let us know if we can help you with advice or with a quick review of your risk acceptance matrix.

Contact and More Information

Simply use the contact form in order to get to a fast and free assessment. You will be contacted back within 48 hours.

Contact us now

Author:

Prof. Dr. Christian Johner

Starter-Kit_rot_dunkel

A quick overview: Our

Starter-Kit

Learn More Pfeil_weiß
blog_rot_dunkel

Always up to date: Our

Institutejournal

Learn More Pfeil_grau