ModelOps as Next-Gen for Model Risk Management

ModelOps as Next-Gen for Model Risk Management

Next-Gen AI transforming Model Validations from MLops to ModelOps

From the financial crisis to the end of 2019, financial institutions (FIs) racked up fines worth $36 billion globally for noncompliance with regulations. Financial institutions continue to face increasing pressure in a highly regulated landscape, including an ever-increasing and regular requirement for model validation in the context of new regulations such as CCAR, FRTB, and IFRS9 models.

Regulatory Changes and Model Risk Management

The changes in regulatory requirements and compliance adherence add to the complex working structure of Risk Management models. The digital footprint going to evolve compliance adherence quicker, better and controlled to fewer surprises for financial institutions. Impact of the changes due to COVID and upcoming changes in regulations has impact to crate below:

1. More Model Risk: Growing number of models across and keeping governance eyes with changes on models.

2. Enterprise-level Model Risk awareness and management: Awareness and reach out programs on the model risk.

3. Reporting: Comprehensive report out from Models, real-time analytics and predictable insights and decision-making abilities. (ModelOps)

4. Data-driven Insights: Models are now being more critical for data-driven insights. The value of the model with greater insights on the quantitative data will support the model risk operating system.

5. New in Model Risk: The new technology in model risk space has potential to report on risk clearly and succinctly.

  • Visualization: Greater eye for details is needed to have layered approach for unparallel information for best decision making for management teams at enterprise level.
  • Big data

6. Artificial intelligence and machine learning /MLOps- Continuous delivery of models: 

  • ML pipeline in production continuously services to new models on new data. The model deployment step is automated, which serves the trained and validated model as a prediction service

7. Automate and orchestrate with an AI governance workflow

  • Integrates with MRM systems, IT Systems, Business Systems
  • Includes MRM requirements in model life cycle
  • Collects and maintains MRM metadata for auditability

8. Continuous monitoring model performance against MRM-supplied conditions and controls

  • Automatically detects breaches to controls set by MRM team
  • Proactive alerts and notification to stakeholders
  • Remediate compliance and quality controls issues

Buoyed by technology advancement, AI-powered data science platform, increased appetite for risk mitigation and development of sophisticated statistical, predictive and prescriptive models have encouraged financial institutions to develop and deploy new risk models, measuring everything from capital adequacy, credit positioning, macro-economic conditions, liquidity, pricing and exposure, where each area have models requiring thorough model validation. This exponential growth in the number of models has led to even greater scrutiny from the regulators on the risks posed by the actual models themselves. Institutions now face the challenge of how to implement effective governance frameworks and model risk management systems to deal with this increased regulatory pressure.

Despite the increasing acknowledgment of model risk; accelerated by the 2008 global financial crisis we only have guidelines and lack globally defined industry standards on managing it. The current regulations are more agnostic in terms of overall financial stability rather than having a well-defined structure/steps in terms of deployment.

Banks are now building on the OCC guidelines with new policies and there is a growing interest in aggregating the model risk across the enterprise, as well as assessing the risk in isolation, based on models performing distinct tasks. In this context, it is essential that every financial institution has an adequate model risk management and governance framework in place.

Model Risk Framework

The model risk management framework must encompass the entire lifecycle and usage of the model. The framework cuts across the different business verticals covering a number of databases, systems, and business units and it is therefore important that the framework is controlled from a central point and that the business processes across the different functions are clearly understood with defined control points. The requirements for the framework include:

  • Governance, Policies and Control
  • Model documentation and inventories
  • Development & Testing of Model
  • Validation of model 
  • Performance monitoring and model usage
  • Real-Time feedback and Linkage to Risk Appetite

Figure 1: Key Attributes of Model Lifecycle

Figure 1 Key Attributes of Model Lifecycle{1}

Continuous model risk management requires development and validation, testing and documentation, requiring specialized resources (generally it’s an extra cost to the FIs). The retention of these resources on a permanent basis is placing a growing financial burden on many banks. Therefore, outsourcing solutions for model validation and governance are gaining popularity.

An outsourced MRM consists of SMEs, with the following key capabilities and domain expertise:

  • Assessment and validation of the existing model 
  • Assumptions and Hypothesis generation for new model validation
  • Model inputs and outputs
  • Building use cases and Testing
  • Risk Assessment
  • Documentation and Reporting

AI-Powered MRM

Generally, the financial institutions have rule-based risk classification which roles up into a similar enterprise-wide risk modeling. These rule-based models have multiple limitations:

  • It isn’t reflective of real-time risk and doesn’t include the changing market landscape
  • It takes a lot more time to validate a new model. By the time validation is complete the financial landscape could have completely changed resulting in reduced effeteness
  • Having a manual process increases the overall operating costs. The chances of error are more and the time required & cost of finding errors at later stage of model development increases exponentially
  • The current rule-based models cover the quantitative aspects but there is a lack of inclusion of qualitative measures (as this needs to be monitored in real-time)

AI-powered MRM provides the following enhancements on the current MRM frameworks:

1. Model Development:

    1. Traditional model produces a single statistical model as output, but AI/ML allows for the combination of different models to provide a more holistic picture
    2. Using AI-enabled workbenches by simply selecting a data set one can run multiple models to check the efficacy of model. This enables organizations to easily compare the performance of different candidate models and select the best one based on several automatically calculated metrics
    3. These advanced analytics models also possess many parameters with complex interactions that are determined by performance and judgment (qualitative) instead of just statistical procedures (quantitative)
    4. Ability to analyze unstructured and semi-structured data sets

2. Model Validation:

    1. Validate models in pre-production with advanced, customizable tests. These include detecting human bias and drift, calculating quality metrics for backtesting, and generating model explanations
    2. Automated real-time testing and reporting
    3. An interactive dashboard that compares model performance (existing vs new) on multiple parameters. Having this capability basically allows the organizations a pinpointed area where the new model needs improvement.

3. Reporting & Compliance:

    1. Automated smart report generation. In the current rule-based models, reports are generated periodically whereas using AI-powered MRM on can have real-time view of the situation
    2. Moving from Reactive to Pro-Active approach towards compliance. Currently, the rule-based engines show the result of what has happened. This leads to a reactive measure both from the FIs and the regulators in terms of fine. Having real-time update gives time to banks to mitigate the risk in a more pro-active manner
    3. Documentation and model management becomes streamlined and incremental changes can be recorder

{1}: Advantage Reply: MODEL RISK MANAGEMENT

The AI process in relevance of regulations – (Abstract -Chartis Research)

Img2

AI Taxonomy:

Img3

AI-powered Model Risk Management Conclusion

Model risk management is the Nucleus of the financial industry, with models providing the quantitative data required to infer informed decisions in real-time. The uses of these models are now becoming more varied and complex, and their potential appears never-ending. MLOps can really support the Model by increased automation within frameworks and improve both production and process quality, the idea is to improve collaboration between data analysts, data scientists, and process managers.

Model Risk Management with MLOps affects the entire life cycle of an ML project, including modeling, orchestration, and implementation as well as validation, diagnostics, governance, and business metrics. Model monitoring reports can provide insights into the accuracy and reliability of the ML models the company is using.

Model Risk Management with ModelOps: Model ops support the Model risk with Real-time monitoring of model performance, retraining and re-validation.

Model ops support the model-centric approach (enabling the MRM better), architecture to enable long-term, uniform deployment success independent from data science workbenches and execution platforms. The Model Ops supports the current state accessibility and also future state enabling the AI across the models not limited to one, this helps the several models with the model risk management to have greater interoperability to work seamlessly.

Img4

“AI model operationalization (ModelOps) is primarily focused on the governance and life cycle management of all AI and decision models (including models based on machine learning, knowledge graphs, rules, optimization, linguistics and agents). In contrast to MLOps, which focuses only on the operationalization of ML models, and AIOps which is AI for IT Operations, ModelOps focuses on the operationalization of all AI and decision models.” (Abstract Gartner)


This article is co-authored by Manorama Kulkarni, Director Financial Crime and Compliance (FCC Asia), Capgemini and Farhan Mohammed, Lead Analyst Financial Crime and Compliance, Capgemini