PMS trend analysis - a complex issue where you cannot fail

Trend analysis is a legal obligation of all medical device manufacturers, especially in "post-market surveillance."

Manufacturers must not fail in the selection and application of suitable statistical methods for trend analysis. This is because the focus of authorities and notified bodies is increasingly shifting to monitoring post-market activities.

This article provides a quick introduction to this complex topic, outlines the legal obligations, and offers seven concrete tips for implementation. It also lists the seven most common mistakes that manufacturers should avoid at all costs.

1. Summary for readers in a hurry

a) Whom the trend analysis in post-market surveillance concerns

The following roles need to help identify, assess, and respond to trends in the post-market phase:

• Individuals responsible for post-market surveillance, e.g., in product management and customer support
• Risk management staff
• Quality management representatives
• Persons Responsible for Regulatory Compliance (PRRC)
• Employees in medical or clinical affairs

b) Why trend analysis is important in PMS

Trend analysis is relevant for medical device manufacturers in post-market surveillance (PMS) to

• meet the regulatory requirements for it, respectively
• prevent regulatory hassles in this context and, if necessary, delay "approval" of the devices,
• to identify problems with its own devices at an early stage,
• thereby ensuring patient safety, and
• to avoid a loss of reputation or even penalties.

c) What the roles need to know and do

These roles must

• understand the regulatory requirements,
• specify what data they collect and at what frequency, and
• what statistical methods will they use to analyze this data and calculate trends,
• set threshold values for the trends and check if they are exceeded, and
• decide whether reporting to the authorities is necessary.

In addition, manufacturers must specify these activities in procedure and work instructions and document the outputs of these activities, e.g., in reports.

2. Regulatory requirements

a) Overview

Many regulatory documents have requirements for trend analysis (see Tab. 1).

Tab. 1: Regulatory requirements for trend analysis

b) MDR / IVDR

The legal requirements of the EU MDR and IVDR regulations are very precise.

Obligation to report already in case of statistical increases leading to unacceptable risks

Manufacturers shall report (…) any statistically significant increase in the frequency or severity of incidents that are not serious-incidents or that are expected undesirable side-effects that

1. could have a significant impact on the benefit-risk analysis (…) and
2. which have led or may lead to risks to the health or safety of patients, users or other persons that are unacceptable when weighed against the intended benefits.

Article 88 MDR

The MDR and IVDR thus oblige manufacturers to find out whether the assumptions in the risk management file are still correct. If this is no longer the case, they might have to report even non-serious incidents (Fig. 1).

Obligation to perform static analysis and set thresholds

MDR and IVDR require manufacturers to plan for post-market surveillance.

b) The post-market surveillance plan shall cover at least:
– suitable indicators and threshold values that shall be used in the continuous reassessment of the benefit-risk analysis and of the risk management as referred to in Section 3 of Annex I

Annex III MDR,Annex III IVDR

These two requirements result in the following:

1. Manufacturers must constantly check whether their risk management assumptions are still correct on the basis of the collected PMS data.

2. They must set threshold values in accordance with the risk policy and check whether these are exceeded and, if necessary, take corrective and preventive measures.

3. If these values are exceeded to the point where the benefit-risk ratio is no longer acceptable, they must report this identified trend to the authority.

Thus, several decision steps are necessary (see Fig. 1):

1. is the "increase in severity of non-serious incidents or expected undesirableside-effects" significant?

2. is the impact on the benefit-risk ratio significant?

3. is the risk-benefit ratio still met?

Which methods are helpful in making the decision and whether changes are significant is described in this article below.

3. Definitions of terms

a) “trend“

MDR and IVDR use the term "trend" but do not define it. Therefore, it is helpful to use other definitions of the term.

b) “statistically significant“

In contrast, EU regulations define when an increase is "significant:"

The significant increase shall be established in comparison to the foreseeable frequency or severity of such incidents in respect of the device, or category or group of devices, in question during a specific period as specified in the technical documentation and product information.

Article 88 MDR

This, in turn, means that one should compare the actual values for probabilites and severities of risks with the originally assumed values in the risk analysis. How large this difference may be and when measures are taken must be specified by the manufacturers via the threshold values. There are several methods to specify these thresholds. They are described below.

4. Procedure for trend analysis

a) Determine what data to collect

Set data sources

EU regulations, ISO 13485, ISO 14971, ISO 24971, and ISO/TR 20416 provide more specific guidance to manufacturers. Manufacturers should not only collect and evaluate customer feedback; other data sources include:

• Authority databases
• Log files
• Scientific literature
• Standards and guidelines
• Installation and service logs
• Customer surveys

Set attributes

Manufacturers should determine not only the data sources but also the attributes they want to capture per record. For example, customer complaints can be documented using the following attributes:

Tab. 2: Examples of attributes for capturing data

Which attributes the manufacturer records depends on the objective. This should be described in the respective standard operating procedure, e.g., in the complaint management process.

Specify frequency of data collection

Manufacturers are required to specify the frequency of data collection. This depends on the type of device, the technologies used, and the questions to be answered (see Tab. 3).

Tab. 3: Examples of the frequency of data collection

b) Set objectives of the evaluation

The laws already determine the minimum set of questions for which the manufacturers must evaluate the data.

• Are the assumptions regarding the probability and severity of harm (e.g., adverse reactions) correct?
• Are there hazards that were overlooked in the risk analysis?
• Do the risk acceptance criteria still reflect the state of the art?
• Are the risks still acceptable, given the benefits?
• What are the causes of the problems? (Design flaws, production flaws, material flaws, suppliers, problems with storage, transportation, installation, maintenance, etc.)
• What are the usability problems (e.g., use errors)?
• Is there an off-label use of the device?

For these questions, manufacturers must also monitor trends and decide if there is a statistically significant increase.

c) Specify methods for evaluation

General

In order to be able to detect statically significant changes, manufacturers must use appropriate methods (plural!).

Based on the diversity of the medical devices in the market it is not meaningful to define a single trending procedure valid for all devices.

Source: Manufacturers Trend Reporting of Adverse Events, IMDRF/GHTF

Many methods are available for trend analysis:

• Regression
• Weibull analysis (recommended in ISO 24971)
• Nelson Rules (recommended in ISO/TR 20416)
• Mann-Kendall test
• Neumann trend test (for normally distributed populations)

The choice of method(s) always depends on the research question.

In the pharmaceutical environment, such trend analyses have been common practice for decades. Accordingly, one will find what one is looking for in the literature there, such as in the book Static Methods of Trend Analysis.

Here is an example from the "Nelson Rules": The fifth rule states that a trend exists if there are two or even three points in a series that are more than two standard deviations away from the mean in one direction (see Fig. 2).

Determine if there is a reportable increase

According to the MDR/IVDR, significant increases in the frequency or severities of potential harms must be reported if they "have a significant impact on the benefit-risk-determination and could lead to unacceptable risks" (see also Fig. 1).

A change is considered significant if it causes severities or probabilities to fall into a class other than that assumed (all arrows in Fig. 3).

Such a change is reportable if the associated risks move into the unacceptable range (see black-filled arrows in Fig. 3).

If enough data is available, averages (e.g., monthly) can help smooth out short-term fluctuations.

Check with the PMS experts at the Johner Institute to see how they can review your PMS plans and give you specific advice for improvement. They will also help you evaluate your data.

Contact us right away via the web form with a suggested appointment for a free, non-binding consultation.

5. Dealing with special cases

a) There are only a few devices on the market

When products are new to the market, or there are only a few instances of a device on the market, there is no data basis for many statistical procedures. Trends can hardly be derived from individual data points.

Then, depending on the risk, manufacturers have to make an increased effort to collect data:

• Participatory observations of usage in the field
• Active questioning of users
• Evaluation of log files at short intervals
• Analysis of comparable devices and technologies

b) Devices are already discontinued

If devices are discontinued, evaluations based on sales figures are no longer useful. It is generally better to standardize the values on the basis of usage.

If no information is available on this, the question arises as to whether the manufacturer is adequately fulfilling its obligation for post-market surveillance.

6. Recommendations for trend analysis

a) Seven valuable tips

The following seven tips will help manufacturers with statistic evaluation and trend analysis.

Tip 1: Use the concept of continuous improvement

Don't overcomplicate anything. It is better to start with only one method for evaluation than not to start at all. For example, you can largely meet the regulatory requirements with the procedure described in the second part of chapter 4 c).

Once you have established this evaluation procedure, you can add more data, methods, and evaluations to learn more about your devices (and users).

Tip 2: Think from behind

Many manufacturers feel overwhelmed by the amount of data. If you don't know which data to collect and evaluate with which priority, think "from behind," i.e., starting with the patients.

To do this, think in terms of a fault tree analysis of which events/hazards could endanger the safety of the patients. In this way, you get an idea which data give an indication of these events/hazards. Start with this data.

Also, look at the clinical evaluation recommendations: here, there should be an indication of what data are essential to capture in post-market surveillance or post-market clinical follow-up.

Tip 3: Use the IMDRF codes

The IMDRF has developed numerous codes that you can use to code problems, causes, etc., across the organization. These codes are required in reports (e.g., the PSUR). Therefore, use these codes from the beginning of your data collection and processing.

Tip 4: Learn more about your company and your devices

Don't think of collecting and analyzing data as a chore required by law. Trend analyses help you learn more about your company, your devices, and your customers and support necessary decisions.

You'll get relevant answers to questions like:

• How do your products and product variants differ from each other?
• How intensively are they used?
• Which phases of the product lifecycle (development, production, material purchasing, transportation, installation, maintenance) should you prioritize to maximize your impact on the safety, performance, effectiveness, and acceptance of your devices?
• Which suppliers cause the most problems?
• Which production facilities, which production processes, and which departments deliver the best outputs?

Tip 5: Collect data in early phases and evaluate it in good time

In order to answer these questions, you need to collect the necessary data (e.g., about suppliers, materials, and production facilities).

You should not start to think about this data only when the devices are on the market. Rather, an early and continuous evaluation is helpful and also required by ISO 13485, for example, in the context of process validation and monitoring.

Tip 6: Adjust sampling intervals

The intervals at which you collect and evaluate data should be variable.

1. The intervals should be dependent on the type and significance of the data (see Tab. 3).

2. The intervals should depend on the amount of data. As the amount of data increases, it is possible to perform evaluations and gain insights at shorter intervals.

3. The intervals should be in line with the risk that you may be able to capture or assess through this data.

Tip 7: Automate as much as possible

All of this assumes that you automate as many activities as possible:

• Collection of the data
• Processing the data (e.g., dealing with incorrect and missing data, conversions, categorizations, etc.)
• Analysis of the data, calculation of statistics and trends
• Alarming
• Report generation

b) Avoid typical mistakes

During audits, technical documentation inspections, and (FDA) inspections, seven typical mistakes stand out that manufacturers should avoid.

Mistake 1: Not all devices considered

The obligations of manufacturers through MDR and IVDR apply to all devices. Manufacturers cannot invoke the transition periods. These have long since expired with regard to post-market surveillance.

Mistake 2: Too little data analyzed

Limiting data collection and analysis to customer complaints does not meet regulatory requirements:

• Manufacturers need to analyze other data sources (as presented in chapter 4 a)).
• Customer feedback (and not just customer complaints) contains information that manufacturers need to look at.

Even if manufacturers analyze multiple data sources, they may still miss trends if they do not collect and evaluate the necessary attributes.

Mistake 3: Post-market surveillance viewed as an isolated process

Post-market surveillance and data evaluation are processes that must interact closely with other processes (see Tab. 4).

Tab. 4: Examples of the interaction between the evaluation of post-market data and other processes

Mistake 4: Too large observation periods selected

More than one year of evaluation periods are usually not accepted by authorities and notified bodies. But even periods of up to one year can be too long for statistical evaluation and trend analysis: trends are detected too late (only after one year) or not at all (because they average out over the year).

Tip 6 (see above) helps to define suitable intervals.

Mistake 5: Competencies not ensured

Auditors easily embarrass manufacturers with the following questions:

• Where have you defined the competencies for creating the PMS plan?
• What competencies do the people who select and apply the statistical methods need?
• Where is the evidence that these competencies are in place?

Mistake 6: Systems not validated

Without computer support, the data can neither be collected nor evaluated in a meaningful way. Therefore, Excel, self-written scripts, ticket systems, and statistical programs are used.

All these are computerized systems (or parts of them), which are subject to the obligations of ISO 13485 according to computerized systems validation. Do you have these validation documents?

Mistake 7: Wrong method and wrong application of the methods

The manufacturers make other mistakes when selecting and applying the methods:

• Many methods may only be used for normally distributed values. This requirement is often not fulfilled.
• Most methods may only be used on ratio and absolute scales but not on ordinal scales.
• Manufacturers make decisions based on regression lines but do not consider confidence levels. The statement that there is no trend is not very robust at a very low confidence level.

7. Summary and conclusion

Many manufacturers are not aware that they must report not only serious incidents but in certain cases also "statistically significant increases in the frequency or severity of non-serious incidents”.

Statistical evaluation and trend analysis, however, are more complex than most manufacturers realize. They are required by law and are under increasing review.

Therefore, manufacturers should dig deeper into the subject and build the necessary competencies.