Banks have been using machine learning for several years already. Other industries, such as medical device manufacturers, can learn from their experiences. This also applies to testing organizations, such as regulatory authorities and notified bodies.
Based on the parallels between the two industries, we have drawn up five best practices and recommendations for saving a lot of costs and avoiding unnecessary trouble with auditors.
A contribution and video from and with Dr. Daniel Lohner and Dr. Christian Johner
Daniel Lohner (d-fine) explains how banks are using machine learning and discusses with Dr. Christian Johner how medical device manufacturers can learn from it.
a) Importance of risk management
Risk management plays a decisive role in both industries. Banks have to manage financial risks, such as credit and market risks, and medical device manufacturers have to manage risks caused by unsafe medical devices.
In other words, both industries have to manage enormous risks. In one case, the stability of banks and, therefore, national economies must be guaranteed, and in the other, the physical integrity of patients, users and third parties.
In addition, both are required to analyze and minimize regulatory risks.
b) Strong and complex regulation
This is because both industries are heavily regulated. The regulatory requirements are constantly increasing.
For the banks, this regulatory tightening took place at the start of the 2000s (Basel II) and the regulations tightened further as part of the process of getting to grips with the banking crisis in 2009, which was partly caused by drastic misjudgments of credit risks.
New developments as well as modifications are now subject to a strict authorization process and have to be validated regularly.
This regulatory structure is complex and varies even within the EU. However, a start was made on harmonizing these requirements, for the bigger banks at least, in 2014.
Bank regulators have even created financial incentives for banks to develop their own internal risk models and to look at the content of their investments in greater detail.
Regulation of medical device manufacturers
Both industries are included in the current technology hype (fintech and medtech/biotech, respectively) and have the increased use of machine learning in common.
Customers and patients can look forward to innovative products. However, in addition to the positive expectations, algorithms, in particular machine learning, also represent major risks from the regulators’ perspective.
Find out more on the use of artificial intelligence (including machine learning) in medicine here.
a) Early digitalization in the banking industry
The banking industry was an “early adopter” of computer technology. It started using paperless transaction processing as early as the 1970s and banks have been offering digital products for decades.
Algorithms have also played an important role for many years. Examples of such algorithms include credit risk models for estimating the probability of customers defaulting on loans. Many of these models use machine learning methods.
b) Stricter regulation of algorithms
It is precisely because banks started using algorithms so early and because inadequate algorithms have already had catastrophic consequences (banking crisis) that banking regulators started drawing up and requiring rules on the use of these algorithms earlier than regulators of other industries.
These requirements relate to:
The Johner Institute’s AI Guideline contains an example of these requirements for the use of machine learning in medical devices.
As a result of these regulations, banks do not use a lot of machine learning methods. When it comes to machine learning, the banks have got used to:
Mandatory annual model validation
This section looks at five aspects where medical device manufacturers should learn from banks if they want to use machine learning.
For each aspect, we describe:
Lesson 1: view model development not as a project, but as a continuous process
A lot of banks initially expected that algorithms would be developed, approved and put into operation once. This meant they often saw model development as a one-off project activity.
As a result, they set up these projects within their organizations and only documented the final project result.
In retrospect, however, it is clear that this was not a good approach from either a technical or regulatory perspective.
Banks have to continuously validate algorithms, e.g., the machine learning algorithms, and adapt them to changing requirements, new data and new findings.
A lot of banks were not prepared for this, from an organizational and a technical perspective.
As a result, audits, ad hoc requests and the mandatory annual model validation represented massive challenges for a lot of financial institutions, leading to capacity bottlenecks. This is because they often had to rebuild their data preparation system and check consistency with old reports manually, which was a time consuming process.
The banks that were able to avoid costs and problems with the regulator were those that had set up clear organizational responsibilities and control processes, including data management, at an early stage and had automated standardized processes as much as possible.
The following has proven to be a sustainable solution:
Processes and infrastructure should be designed for iterative and incremental development right from the start.
An efficient technological and organizational infrastructure for a validation and development environment and data management does represent a high initial investment, but it pays for itself relatively quickly. In the end, these aspects were actually anchored in the regulatory framework following negative experiences.
Therefore, it is to be expected that medical technology companies that take the initiative and tackle these issues at an early stage will also be able to benefit from cost savings and better audit results in the medium term.
Lesson 2: submit data, code and models to configuration management
A lot of banks only saved the “finished end results”. This became a problem because the banking regulators repeatedly demanded comparative analyses of the development sample for rule validations in productive environments and, in particular, in the event of changes to models and procedures.
Therefore, banks had to justify, time and again, why they had used a particular machine learning method and not a different one or a classical computing method.
The financial institutions were unable to demonstrate that the results with current data were consistent with the results with the training data. They were also unable of compare the different models to one another and therefore unable to provide proof of improvements. This led to regulatory problems.
The origin of data and the data preparation should be fully documented and reproducible. This applies for the data sample used for model development in particular.
This is because the models have to be continuously developed for several reasons:
Although completely historicized data models would mean a higher initial investment, they have proven to be a sustainable solution in practice. Historicization includes version and configuration controls for
Read more on the validation of machine learning libraries here.
Lesson 3: justify the selection of models/algorithms
The selection of models, particularly with regard to model alternatives (e.g., other machine learning methods, other hyperparameters), should be documented and reproducible.
When approving new models and modifying models, banking regulators ask about alternative or historical model designs.
In the past, such requests were often regarded as special analyses and analyzed in an individual analysis alongside the actual model development.
Experience shows that such demands are made again and again and that, under time pressure, the risk of errors and inconsistencies increases significantly.
For this reason, and from a technical point of view as well, it is sensible to compare models with alternative models and techniques regularly during validation. Companies can significantly reduce time and costs by integrating alternative models and approaches into the model development and validation environment directly.
Companies should develop a model portfolio with “challenger” models that are run in parallel to the productive model using the same data. This makes it possible to always produce consistent results and it also helps demonstrate later why you selected the models you did and discarded others.
In the “Artificial Intelligence” seminar, manufacturers learn how to conduct and document these model comparisons without overheads and, thus, meet the regulatory requirements.
Lesson 4: the models and model results should be easy to explain
A major lesson of the financial crisis was that a lack of transparency about the risks and blind trust in “black box predictions” had led banks to do too much business with products that were supposedly safe but which were actually high-risk.
This was partly because of the rating agencies whose forecasts are still not transparent today.
Today, regulators pay special attention to the transparency and economic plausibility of model predictions.
In practice, this also means restrictions in the choice of machine learning model.
This was ultimately also a consequence of too much uncritical use of external model results before the financial crisis.
Since then, regulators have performed targeted individual assessments in which they scrutinize individual model results and their internal acceptance (known as a “use test”).
In addition, banks were unable to comply with their customers’ right of access to their data, which was not actually established by the GDPR but was in fact already required by the German Federal Data Protection Act.
Companies should select algorithms and models on the basis of their transparency and traceability as well as their performance (e.g., specificity and sensitivity). This is important for three reasons.
“Interpretability” should already be a factor in the choice of model. Users must be able to understand how the input data (feature) has affected the predictions made by a machine learning model. Regulators should be able to see that a model doesn’t just provide correct predictions at random or within narrow limits.
This forces manufacturers to use well documented and stable methods such as decision trees, logistic regression and scenario simulation models in preference to unnecessarily complex models.
In other words, “keep it simple” when it comes to selecting a model: companies should only use less comprehensible models, such as neural networks, if they provide demonstrable benefits. Banks are currently barely able to have such machine learning methods “approved”.
In addition, companies should use techniques to improve the interpretability of the models, in other words, to show auditors and users how the models work. Such techniques include, for example, LRP (layer-wise relevance propagation) and spectral relevance analysis (SpRAy).
Lesson 5: stay one step ahead of the regulator
A lot of banks waited until the last moment to adapt to regulatory changes.
This wait-and-see attitude certainly reduced time and effort in individual cases where regulatory requirements changed after industry consultations shortly before they came into effect, or when it had become apparent that certain aspects were not interpreted as strictly in practice.
But in a lot of cases this strategy led to considerable additional costs: Firstly, as a result of “last minute” implementation projects and, secondly, due to the missed opportunity to save costs and avoid trouble with the regulator in the long term through early implementation of a technologically and organizationally sustainable infrastructure.
Companies should do what is right and proper and not wait until an authority or regulator forces them to do something. They themselves benefit in several ways from such an approach:
Companies should actively, and independently of regulatory requirements, continuously improve their best practices regarding the use of algorithms, particularly machine learning algorithms. They should act more as trendsetters and not be one of those companies that only follows and is surprised by changes.
The medical technology industry is currently going through a change that started 10 years ago in the banking sector. Therefore, medical device manufacturers should take a close look at what they can learn from developments at banks.
The companies that follow the recommendations above will, like the “proactive” banks, generally always come out of audits well and benefit from a sustainable infrastructure for model maintenance.
Over-engineering, particularly of overly complex machine learning models, often leads to higher and avoidable follow-up costs and, in practice, often provides little added value compared to simpler models.
For the banks, the key to maximum efficiency is a critical cost-benefit analysis of model complexity while at the same time complying with the framework conditions such as transparency, reproducibility and clear responsibilities.
For medical device manufacturers, safety and benefits for the patients must be the top priorities. A more complex algorithm may increase the benefits for a lot of patients but compromise the safety of some patients.