The sooner your ongoing monitoring plan comes together, the less likely undetected problems can cause issues.
Ongoing monitoring: One of the strongest safeguards of model reliability
Establishing a robust, well-documented framework for ongoing monitoring is critical to identifying and preventing model deterioration. A framework that identifies issues will serve your organization more proactively than one that only reacts once an issue has been discovered through a longer period of use.
Although ongoing monitoring should be established during model development, many organizations lack the framework and understanding to establish a comprehensive plan. In a 2020 Crowe survey, 45.5% of webinar participants stated that their top model risk management (MRM) challenge is working with model owners to conduct ongoing monitoring.
The consequences of an inadequate or overlooked ongoing monitoring framework often lurk in the future. External evaluations might trigger flags, but the most direct damage can come when models are allowed to run for significant periods without their issues being identified. This oversight could lead to extensive backtracking, corrections and – if not caught in time – poorly informed business decisions.
What belongs within a comprehensive ongoing monitoring framework? Those elements might depend on the characteristics of each model, but the following considerations should be top of mind.
Get deeper insights into the essentials of an ongoing monitoring plan.
Consistent testing of model output
The actual product of an organization’s models should receive consistent testing through applicable means such as backtesting, benchmarking, sensitivity testing, data quality testing, vendor’s testing, review of changes, model processing testing, and assessment of internal and external changes.
- Backtesting. Backtesting is arguably the most important factor of ongoing monitoring. How did actual results compare to a model’s predictions, and what can this reveal about a model’s future predictive capabilities?
This form of testing can take different approaches depending on the type of model. A loan default model might directly compare actual defaults to the model’s predicted likelihood, while an anti-money laundering model might compare the number of false positives provided to a predetermined threshold.
- Benchmarking. Benchmarking is another form of testing to consider. Output can be compared to challenger models or external data sources. If an organization’s data deviates significantly from peers or different vendors, it might be worth further investigation.
- Sensitivity testing. Sensitivity testing, during which inputs are adjusted to understand change in output, can also help demonstrate whether models will hold up against potential future volatility.
- Data quality testing. Models have different inputs. Organizations should establish appropriate activities to assess whether the data is of proper quality. Then they should test, review, and document acceptable levels of data completeness and quality for every model.
- Vendor’s testing. Organizations should incorporate the model vendor’s own testing into their testing framework whenever appropriate. Doing so might include setting a reporting schedule that works for both parties.
- Review of changes. Organizations also should pay special attention to the record of overrides, overlays, and adjustments made to each model during output testing. A long history of changes might be a sign that a model should be redeveloped or replaced.
- Model processing testing. Inaccuracies can stem from unexpected changes within a model’s production environment or code. Periodic evaluations can help ensure that a model is working as intended. Running models in parallel on two systems and comparing outputs can help detect any changes that may need to be addressed.
- Assessment of internal and external changes. As the environment changes, so might the need to adjust models and key assumptions. Ongoing monitoring should regularly investigate how models respond to factors including:
- Changes in laws and regulations
- Changes in internal underwriting policies
- Changes in credit quality
- Mergers, acquisitions, or any other event that introduces another organization’s data to the models
- Changes to accounting standards
- New geographies
Secure standards: Critical for your ongoing monitoring process
Ongoing monitoring policies and procedures that are vague, difficult to execute, or not enforced will poorly serve model users. Everyone involved in the model process should have comprehensive, accessible answers to pertinent questions regarding monitoring activities, including:
- What is the activity?
- Who will execute it?
- What are the necessary tools and processes for execution?
- What is the schedule for execution?
- Who is accountable for setting targets for model use, accuracy, and reliability?
- What thresholds should be considered?
- What internal and external changes are tested as part of monitoring?
- What is the proper response when a threshold is breached or a flag is triggered?
- What is the escalation process?
- What is the remediation process?
- What information is included in a monitoring report?
- Who receives monitoring reports and when?
Get deeper insights into the essentials of an ongoing monitoring plan
Download our guide on the tests and considerations that could factor into an ongoing monitoring plan, including examples.
Captcha is required.
We provide insights beyond model validation
Ongoing monitoring evaluations and recommendations are already part of our standard model validation process, but we also offer direct services to establish or bolster your ongoing monitoring framework.
We can build policies, procedures, and templates designed to align teams and introduce technology that strengthens ownership, timeliness, and communication.
Gain a clearer perspective on risk
Crowe Model Risk Manager can make it easier to visualize your model risk management process.
Contact us today to discuss how we might help you achieve more effective monitoring of your models.
