Large Bank Examination Workshop February 2026

First page Table of contents 1 Next page Last page

Large Bank Examination Workshop

February 10-12, 2026 Washington, DC

@ www.csbs.org ♦ @csbsnews

CONFERENCE OF STATE BANK SUPERVISORS 1300 I Street NW / Suite 700 / Washington, DC 20005 / (202) 296-2840

Large Bank Examination Workshop Washington, DC February 10-12, 2026

Tuesday, February 10, 2026 8:30 am – 9:00 am

Breakfast

Model Risk Management

9:00 am – 10:30 am 10:30 am – 10:45 am 10:45 am – 12:00 pm 12:00 pm – 1:00 pm 1:00 pm – 2:30 pm 2:30 pm – 2:45 pm 2:45 pm – 4:30 pm

Break

Model Risk Management

Lunch (on your own)

Model Risk Management

Break

Model Risk Management

Wednesday, February 11, 2026 8:30 am – 9:00 am Breakfast 9:00 am – 10:30 am

Liquidity Risk Management

10:30 am – 10:45 am 10:45 am – 12:00 pm 12:00 pm – 1:00 pm 1:00 pm – 2:30 pm 2:30 pm – 2:45 pm 2:45 pm – 4:30 pm

Break

Liquidity Risk Management

Lunch (on your own)

Liquidity Risk Management

Break

Liquidity Risk Management

Thursday, February 12, 2026 8:30 am – 9:00 am

Breakfast

Capital Adequacy and Stress Testing

9:00 am – 10:30 am 10:30 am – 10:45 am 10:45 am – 12:00 pm

Break

Capital Adequacy and Stress Testing

12:00 pm – 1:00 pm

Lunch (on your own)

Capital Adequacy and Stress Testing

1:00 pm – 2:30 pm

2:30 pm – 2:45 pm

Break

Capital Adequacy and Stress Testing

2:45 pm – 4:30 pm

Model Risk Management

December 9, 2025

Instructor: Rob McDonough

E-Mail: consult@gfmi.com Web page: www.gfmi.com Phone: +1 516-935-0923

3 The entire content of this course material is for student use only, subject to copyright and as such, may not be reproduced, modified or used in whole or part. Usage other than intended purpose is prohibited. 3

Course Objectives By the end of the course, participants will be able to: • Discuss the current regulatory guidance for model risk management at banks • Analyze the key factors involved in developing, implementing, and using models • Explain the role of model validation and the independence required for this process to be effective • Describe at a high level the necessary components of a model risk governance framework

‘All models are wrong; some models are useful.’

Source: Box, G. E. P., Hunter, J. S., and Hunter, W. G. Statistics for Experimenters: Design, Innovation and Discovery , 2 nd Ed., Hoboken, NJ: Wiley, 2005, p. 440.

Model Risk Management

Regulatory Guidance and Best Practices Session 1

Session 2

Model Development, Implementation, and Use

Session 3

Model Validation

Session 4

Governance, Policies, and Controls

Session Objectives By the end of the session, participants will be able to: • Discuss the Supervisory Guidance on Model Risk Management (Basel, OCC 2011-12 / SR 11-07) • Define model risk and how it impacts banks • Explain the concept of “effective challenge” • Identify where models and model output have a material impact on business decisions

Basel Model Risk Guidelines 1 • In July 2009 the Basel Committee on Banking Supervision issued a directive 1 requiring that financial institutions quantify model risk: 1. The model risk associated with using a possibly incorrect valuation 2. The risk associated with using unobservable calibration parameters. • Explicitly mandates a valuation adjustment that impacts Tier I regulatory capital to account for model risk.

1 Basel Committee on Banking Supervision: Revisions to the Basel II Market Risk Framework, July 2009.

Basel Model Risk Guidelines • Must have a robust framework for the subsequent ‘on-going’ validation activities that are required for continued regulatory compliance. • A consistent, and replicable model validation process must be implemented. • Validation function is performed by an independent group; independence and objectiveness are essential for a robust model validation process.

Basel Model Risk Guidelines • Confirmation of the conceptual soundness and initial risk quantification of the model’s design, concept, methodology, and assumptions • Confirmation of model’s operations, replicability, override and exceptions monitoring, monitoring of model KPIs, data integrity, and the use test • Annual examination of overall performance of the model  Backtesting  Benchmarking  Annual health check

Basel Model Risk Guidelines • Benchmarking is the examination of the performance of models relative to comparable models • Backtesting is the examination of the performance of the model based on its historical data comparing realized and predicted outcomes.  Backtesting is required for PDs, LGDs and EADs Basel language has “credit” focus, but concepts apply to market, operational, and other risk models.  Benchmarking performing loans  Benchmarking defaulted loans

U.S. Regulatory Guidance

• OCC 2000-16

 First comprehensive guidance • Elements of sound validation policy

 Independent review  Defined responsibility  Model documentation  Ongoing validation  Audit oversight • Focus on market risk - primarily interest rate risk

OCC 2000 – 16: Goals • Decision-makers understand the meaning and limitations of a model’s results. Where the models are too abstract for non specialists to understand the underlying theory, the bank must have a model reporting system in place that transforms the models’ outputs into useful decision-making information without disguising the model’s inevitable limitations. • Particularly when a model has been in use for a reasonable period of time, its results are tested against actual outcomes. • The bank should demonstrate a reasonable effort to audit the information inputs to the model. Input errors should be addressed in a timely fashion.

OCC 2000 – 16: Goals • The seniority of the management overseeing the modeling process should be commensurate with the materiality of the risk from the line of business in process. • To the extent feasible, model validation must be independent from model construction. • Responsibilities for the various elements of the model-validation process must be clearly defined. • Modeling software should be subject to change-control procedures, so that developers and users do not have the ability to change code without review and approval by an independent party.

U.S. Regulatory Guidance • Federal Housing Finance Agency Advisory Bulletins  2006-AB-02 focused on market risk models o Similar to OCC 2000-16  2009-AB-03 explicitly includes credit risk models and also addresses the validation of externally-managed vendor models, internally-managed vendor models, and the importance of validating and documenting the controls over models and their use. o “Model documentation and validation” should be read to include proper testing of related controls on the processes surrounding those models.

Interagency IRR Policy (Jan ‘10) • Goal: to remind institutions of supervisory expectations regarding sound practices for managing IRR. • Effective risk management processes especially important for institutions under earnings and capital pressure due to lower credit quality and market illiquidity. • Reiterates the importance of model validation for both internally developed as well as vendor-based models • Further emphasizes an expanded view on risk that goes beyond interest rate volatility to assess the impact of liquidity, credit, and other risk factors.

Interagency Guidance on Model Risk Management - 2011 • Most comprehensive guidance for banks on effective model risk management. • Rigorous model validation plays a critical role in model risk management; however, sound development, implementation, and use of models are also vital. • Encompasses governance and control mechanisms such as board and management oversight, policies and procedures, controls and compliance, and an appropriate incentive and organizational structure.

Federal MRM Regulatory Guidance

• Federal Reserve SR 11-7 (2011) - Defines model risk as the potential for adverse consequences from flawed or misused models. Requires banks to establish robust model risk management frameworks, including governance, validation, and ongoing monitoring. • OCC Comptroller’s Handbook: Model Risk Management (2021 update) - Provides detailed examination procedures for national banks and federal savings associations. Aligns with SR 11-7 principles, emphasizing independent validation, documentation, and board oversight. • FDIC Supervisory Guidance (FIL-22-2017, updated 2024) - Applies SR 11-7 principles to FDIC-supervised institutions. Highlights model risk in areas like capital adequacy, stress testing, and credit risk.

Federal Reserve SR 11-7 (2011) Provides comprehensive guidance on model risk management for banks. It emphasizes that models are inherently imperfect, so institutions must have strong governance, validation, and controls to manage risks from their use. • Definition of a Model A model is any quantitative method, system, or approach that applies statistical, economic, financial, or mathematical techniques to process input data into quantitative estimates. • Purpose of the Guidance Issued jointly by the Federal Reserve and the Office of the Comptroller of the Currency (OCC) , SR 11-7 sets supervisory expectations for how banks should manage risks associated with models used in decision-making.

OCC MRM Update (10/6/25) • Community banks have the flexibility to tailor their model risk management practices, including the appropriate frequency and nature of validation activities, commensurate with the bank's risk exposures, its business activities, and the complexity and extent of its model use. • The OCC's guidance on model risk management does not, and should not be interpreted to, require community banks to perform annual model validation. • The OCC will not provide negative supervisory feedback to a bank solely for the frequency or scope of the model validation that the bank reasonably determined to perform based on the bank's risk exposures, its business activities, and the complexity and extent of its model use.

OCC MRM Update (10/6/25) • The OCC will reinforce these points in communications with its community bank examination teams to ensure consistent messaging throughout the agency. • This bulletin is a first step as part of the OCC's broader review of model risk management guidance, practices, and examiner feedback at banks of all sizes.

Model Risk

‘The federal banking regulators define model risk as the potential for adverse consequences from decisions based on incorrect or misused models.’

Source: Federal Reserve SR 11-7

Exercise: Sources of Model Risk • Make a list of possible sources of model risk. • Try to do this without looking ahead at the next slide!

Sources of Model Risk

• Inappropriate specification • Incorrect parameter estimates • Flawed hypotheses and/or assumptions • Mathematical computation errors • Inaccurate, inappropriate or incomplete data • Improper or unintended usage • Inadequate monitoring and/or controls

Source: Office of the Superintendent of Financial Institutions (Canada), Enterprise-Wide Model Risk Management for Deposit-Taking Institutions, September 2017.

Discussion: Effective Challenge • A guiding principle for managing model risk is effective challenge: critical analysis by objective, informed parties who can identify model limitations and suggest appropriate changes. 1. Why might model users fail to offer effective challenge to a model?

2. What structures, processes and behaviors can a bank use to encourage effective challenge?

Key Areas Where Models Drive Decisions for Banks • Risk Management & Capital Adequacy  Models determine credit, market, and operational risk exposure , which directly influences how much capital banks must hold under Basel regulations.  Stress testing and scenario analysis help banks prepare for adverse economic conditions. • Credit Decisions & Lending  Credit scoring models assess borrower risk, distinguishing between “good” and “bad” borrowers.  These models guide loan approvals, interest rate setting, and provisioning for defaults.

Key Areas Where Models Drive Decisions for Banks • Pricing of Financial Products  Models calculate fair values for derivatives, mortgages, and structured products.  They help optimize interest rates, fees, and spreads to balance profitability with competitiveness. • Forecasting & Strategic Planning  Predictive models forecast liquidity needs, customer demand, and macroeconomic trends .  Banks use them to plan growth strategies, manage reserves, and anticipate regulatory changes.

• Investment & Portfolio Optimization  Quantitative models optimize asset allocation, balancing risk and return.  They are used in hedging strategies and to maximize shareholder value. • Fraud Detection & Compliance  Machine learning–based quantitative models identify unusual transaction patterns.  They support compliance with anti-money laundering (AML) and other regulatory requirements. Key Areas Where Models Drive Decisions for Banks

Model Risk Management

Regulatory Guidance and Best Practices Session 1 Model Development, Implementation, and Use Session 2 Model Validation Session 3

Session 4

Governance, Policies, and Controls

Session Objectives By the end of the session, participants will be able to: • Develop a statement of purpose to ensure that model development is aligned with its intended use. • Explain the importance of documenting the design, theory, and logic underlying the model • Describe the requirements for data integrity and the quality of assumptions for inputs • Assess the effectiveness of model use and management reporting

What is a Model?

‘A model generally refers to a methodology, system, and/or approach that applies theoretical and (expert) judgmental assumptions and statistical techniques to process input data in order to generate quantitative estimates.’

Source: Office of the Superintendent of Financial Institutions (Canada), Enterprise-Wide Model Risk Management for Deposit-Taking Institutions, September 2017.

Inputs

Processing Economic Financial Mathematical Statistical

Outputs Estimates Forecasts Decision Support Control

Data Assumptions Scenarios

Discussion: How Many Models? • Based on this definition and description, how many models do you think are used in a typical U.S. bank? a) 3?

b) 10? c) 50?

• Are all these models of equal significance for the operations, integrity and security of the bank?

• Is a spreadsheet a model?

• Should they all be subject to the same process of governance, risk management and regulatory oversight?

Discussion: Model Types • Make a list of applications or activities within a bank in which models play an important role.

• 1 • 2 • 3 • 4….

• Where possible, give examples of specific outputs or decision variables (e.g. prices, risk measures, etc.) generated by the models that you describe.

Rationale for Modeling

Modification/ Decommission

Model Development

Model Lifecycle

Ongoing Monitoring and Review (Validation)

Independent Review (Vetting)

Model Approval

Statement of Purpose • Model users should develop an economic or business rationale for developing a new model or for changing an existing model. • Model owners should understand the purpose of the model and ensure that modeling choices are properly documented . • This should be discussed and developed into a model “statement of purpose” containing:  why the model is being built,  what business problem it addresses, and  how it will be used. • It sets the scope, objectives, and intended outcomes of the model before any technical work begins • Evidence must be provided on the suitability of the model for the proposed purpose (e.g. by comparison with other candidate models that could be used or are used for the same purpose).

Model Development • The institution should have development processes for model owners to follow once the model choice has been made. • The objective is to implement a model that accurately quantifies the desired measures and reports them back to model users. • Documentation is an essential element of this process.

Inputs Critical assumptions Quantification of key parameters Data (relevance, quality, availability)

Computation Processes Methodology Programming of necessary code

Reporting Processes Communication of outcomes to permit effective use in decision-making

Model Development

Discussion: Documenting Development 1. Why is documentation an essential element of model development?

2. What should be included in this documentation?

Testing • Testing is an integral part of model development.

• The objective of testing is to determine whether the model is performing as intended.

• The tests and analysis performed will depend on the model and on the context in which it is to be used.

• Any single test is rarely sufficient, so banks should conduct a variety of tests to develop a sound model.

Testing • Tests should be conducted under a range of market conditions, including scenarios outside the range of normal expectations.

• Tests should encompass the range of products and applications for which the model is intended.

• Extreme values for inputs should be evaluated to identify boundaries of model effectiveness (stress tests).

• The impact of model results on other models that use those results as inputs should be evaluated.

Source: Board of Governors of the Federal Reserve System/Office of the Comptroller of the Currency, Supervisory Guidance On Model Risk Management (SR 11-7), April 2011.

Independent Review (Vetting) • Institutions should have an independent review (vetting) process to check that models under development are sound and fit for purpose.

Verification and Assessment Check documentation Review model owner’s model selection decision Evaluate the three components of the development process

Secondary Review Appraise conceptual soundness and model performance against criteria that reflect model purpose and product scope

Model Approval • Models used for regulatory capital inputs or internal risk assessment and control and related activities should not, unless an exception has been granted and documented, be approved for operational use without undergoing independent review .

• Institutions should have a dedicated model risk committee or model approver for approving new models for use.

• Models with identified weaknesses or limitations may be recommended for conditional approval provided that compensating mitigations are in place.

Validation • Once a model has been approved, its use should be subject to ongoing monitoring and review. • This process is called validation. • The objective is to verify that the model is performing as expected in line with its design objectives and business use. • Model owners and users have initial responsibility for validation. • All internal models and internal estimates should be subject to a thorough and consistent validation at least annually .

Discussion: Validation 1. For each of the elements of effective validation on the previous slide, what are some of the activities, techniques and methods that should be included in the process? 2. For obvious reasons, validation should not be done by those responsible for developing the model or those who have a stake in whether the model is determined to be valid. Why might this sometimes be difficult to achieve, and what steps can banks take to ensure properly independent validation?

Model Exceptions • Institutions should have processes and policies in place to manage model exceptions:  Use of unapproved models  Use of a model outside its intended purpose  Use of a model that has breached performance metrics  Back-testing shows model results inconsistent with actual outcomes • Exceptions should be escalated to the model risk committee or senior management, and the oversight authority should have the power to impose restrictions on use of the model

• Internal audit should maintain an ongoing review of the process.

Vendor Products • Banks are expected to validate their own use of vendor products. • Outsourcing model development does not eliminate the need to apply a similar vetting, approval, validation and decommissioning process as would be conducted for models developed in-house. • Restricted access to proprietary intellectual property (e.g. computer code) may mean banks using vendor product have to rely more on sensitivity analysis and benchmarking.

• But models should still be fully documented, including testing results, ongoing performance monitoring and outcomes analysis.

Backtests • Backtests compare actual outcomes with model forecasts during a time period not used for model development.  Using “out of sample” data • Model forecasts (usually expressed as an expected range or confidence interval around a central estimate) are compared with actual values for the back-test period, at a frequency and forecast horizon corresponding to the model’s intended use. • Differences between actual and forecast values are analyzed to see if they are significant in magnitude or frequency. • Variances do not necessarily indicate a problem with the construction of the model, as the future is impossible to accurately predict. • But – management should be able to explain and reconcile variances between the model forecast and actual events.

Modification • Models that perform poorly or become obsolete will be modified or decommissioned. Control of this process is essential.

• Modifications should be subject to the same level of vetting and validation as is involved with new model approval.

• No-one should have the authority to change a model or its use without re-approval through an appropriate process.

• Model changes should be carefully documented and justified.

Decommission • Decommissioning is not the end of the model risk lifecycle. • Recently decommissioned models may act as benchmarks for new models, and could be re-commissioned if the new model is not properly implemented to does not perform as expected.

Exercise • Let’s go to Exercise #2: Review sample model output under various scenarios to identify issues with report

• Exercises Pages 1-7

Model Risk Management

Regulatory Guidance and Best Practices

Session 1

Session 2

Model Development, Implementation, and Use

Model Validation Session 3

Session 4

Governance, Policies, and Controls

Session Objectives By the end of the session, participants will be able to: • Describe the importance of independence in the validation process • Discuss the strengths and drawbacks of internal vs. external validation teams • Assess the requisite knowledge, skills, and expertise of validation resources • Explain the importance of resolving validation findings and issues

Elements of a Sound Validation Program

• Data and Assumptions • Model Theory • Model Code and Mathematics • Model Reports

Validation Best Practices • The model validation process should optimally begin during the model implementation process in order to ensure that any model risks are identified  While model is in “beta” testing • Additionally, any changes to a model’s design or enhancements should be considered in light of existing assumptions and limitations around use and any updates that may need to occur. • Validation conducted on an ongoing basis at appropriate periodic intervals

Evaluation of Conceptual Soundness

Ongoing Monitoring Process verification Benchmarking against alternative internal or external data or models

Effective Validation

Outcomes Analysis Comparison of model outputs to corresponding actual outcomes

Source: Board of Governors of the Federal Reserve System/Office of the Comptroller of the Currency, Supervisory Guidance On Model Risk Management (SR 11-7), April 2011.

Elements of Sound Validation Program Audit Oversight • While larger organizations may have model-validation units within internal audit departments, model validation is often outside the scope of audit responsibilities. • Nevertheless, the formal policy should clearly specify that internal audit is responsible for ensuring that the model validation and model-validation units adhere to the formal policy.

From OCC 11-12/FRB 11-07 • A bank's internal audit function should assess the overall effectiveness of the model risk management framework. • Internal audit should verify that acceptable policies are in place and that model owners and control groups comply with those policies. • Internal audit also has an important role in ensuring that validation work is conducted properly and that appropriate effective challenge is being carried out.

Model Validation Policy • Firms that rely on financial models to measure, monitor, and manage risk should have formal model validation policies:  Model and model risk definitions;  Assessment of model risk;  Acceptable practices for model development, implementation, and use;  Appropriate model validation activities; and  Governance and controls over the model risk management process.

Scope of Policy

• Roles and Responsibilities • Model Inventory and Categorization

• Model Implementation • Model Documentation • Model Validation

• Reporting and Monitoring Validations • Retention and Safekeeping of Models

Elements of Sound Validation Policy

Independent Review • The personnel performing model validation should be as independent as possible from the personnel who construct the model. • When comprehensive independence is not practicable, the policy should explicitly provide for an effective communication process between modelers and decision makers

Independence • Validators may be internal staff, external consultants, or a combination of the two. • More complex models may warrant the engagement of third-party experts to supplement internal validations. • When internal staff validates a model, management must ensure that the staff is independent of the staff responsible for operating or using the model.

Elements of Sound Validation Policy

Defined Responsibility • The responsibility for model validation should be formally defined. Policies should specify that, before a model can enter production: a) the independent model-validation unit or external reviewer must document the model validation tests and the reasons for concluding that the model is valid, and b) internal audit must verify that no models enter production without formal approval by the validation unit.

Model Inventory: What Counts as a Model?

• Comprehensive set of information for models implemented for use, under development for implementation, or recently retired • While each line of business may maintain its own inventory, a specific party should also be charged with maintaining a firm-wide inventory of all models, which should assist firms in evaluating its model risk in the aggregate. • Any variation of a model that warrants a separate validation should be included as a separate model and cross-referenced with other variations.

Mission Critical Models • Institutions should thoroughly document and validate, at least annually , all mission-critical models, including:  pre-purchase models  hedging models

 market risk models  credit risk models  collateral “haircut” models  models used in preparing public financial disclosures.

Elements of Sound Validation Policy

Ongoing Validation • Most models are frequently altered in response to changes in the environment or to incorporate improvements in modelers’ understanding of the model’s subject. • However, model alterations can also help evade risk limits or disguise losses. • All changes in the modeling process should be documented and submitted for independent review.

Roles and Responsibilities • Risk Committee that has a delegation of authority from the Board usually responsible to oversee the model validation process  Can report to Enterprise Risk Management  Usually an Operational Risk function • Independent of all business units operating and utilizing the models.

Committee Role • Review and approve validation procedures • Review and approve model inventory annually • Review and approve classification annually • Review and approve a schedule for validation • Approve the selection of validators • Review the progress and results of validation and report this to Board

Business Unit Roles

• Assist in classifying each model • Assist in developing model inventory • Document the decision to develop or purchase new models or upgrade existing versions • Develop schedule for validation • Develop appropriate documentation as required under these procedures.

Business Unit Roles • Assist in developing and distributing each RFP for validation services for models • Assist in selecting the list of vendors to receive such RFP. • Review the responses to any RFPs and consult on the vendor to conduct any such validation. • Review the results of validation of any quantitative financial model and comment.

Internal Audit Role • Responsible for verifying compliance with and evaluating the adequacy of any work performed pursuant to these procedures • Review any subsequent remediation resulting from a validation. • Ensure that outstanding issues have been remediated during the annual audit cycle.  As a component of business unit audits  As a component of audit of validation process

Internal Audit Assessment • Verifying that selection of validation resources is appropriate  Identifying potential “selection bias” before selecting validator • Compare Engagement Letter (Scope) to Validation Report. • Compare current validation report with prior year validation report. • Inspect Organization’s model validation procedures and evaluate Validation Report for adequacy.

Internal Audit Assessment • Inspect relevant regulatory requirements and evaluate validation report for compliance. • Benchmark validation report with best practices. • Address any outstanding issues in the Management Response to the recommendations from each validation.

Mission Critical Models • In the case of mission-critical models managed externally by a vendor, the institutions should require the vendor to provide its model certification or quality assurance results. • Certification of math and mechanics of model different from a validation of a specific instance of a model - see handout example (Exercises Pages 8-10) • If the vendor considers such results too revealing of proprietary mathematics and code, it can provide evidence of the results and of the steps taken to generate them.

Model Inputs - Data • It is possible that data inputs contain major errors while the other components of the model are error free. • The model outputs become useless, but even an otherwise sound validation process will not necessarily reveal the errors. • Auditing of the data inputs is an indispensable and separate element of a sound model-validation process, and should be explicitly included in the organization’s policy.

Model Inputs - Data • The organization’s audit functions should ensure that internal data provided to the model agrees with the organization’s general ledger data, terms of outstanding contracts, etc. • Externally provided data can also often be checked against multiple sources. In addition, extremely effective and inexpensive procedures to spot errors include automated filters and the inspection of the inputs by experienced personnel.

Model Inputs - Data • If an organization decides that the model provides useful information despite data problems, the organization’s policies should specify that audit, risk management, and modeling personnel are independently responsible for apprising senior management of the data problems. • This alerts decision makers both that the model results may not be completely reliable and that there may be a need to devote more resources to providing quality data.

Mark to Market • Marking to market is the valuation of positions at readily available closing prices that are sourced independently. Examples of readily available closing prices include:  exchange prices  screen prices  quotes from independent brokers. • Institutions should mark positions to market prices where possible. • Addresses ASC 820 Fair Value Level I securities.

Mark to Model • Marking to model is an alternative to marking to market when market prices are not available. • It is defined as any valuation that has to be benchmarked, extrapolated, or otherwise calculated from a market input. • Addresses ASC 820 Level II and Level III securities.

Validation Issues - Data • Complex investment securities can be especially problematic to obtain values for • Biggest problems for most organizations:  Private label mortgage-backed securities  TruPS (Trust Preferred and TruPS CDOs)  Thinly traded ABS  Private credit instruments (loans or bonds) • Institutions should have Pricing Policies indicating how problematic securities should be treated  Such policies should have escalation procedures to a Pricing Committee when there is a conflict between front and back office on pricing.

Model Inputs - Assumptions • Prime examples include prepayment functions for loan-valuation models, “market-implied” interest-rate volatilities for derivative pricing models, and core deposit decay assumptions for asset-liability models. • These types of assumptions are generally determined by a separate model, which itself has inputs, processing and outputs that should be validated using the principles elucidated here.

Model Inputs - Assumptions • Many assumptions will be available in general form from publicly available sources at relatively low cost. • For example, many banks use the market-implied volatilities and mortgage prepayments that are available from the various vendors. • Core deposit decay behaviors also available from the various vendors.

Model Inputs - Assumptions • An organization may feel that it is better to derive its assumptions by studying its own customer base than by using general information about national or regional populations. • Interagency FAQ on Interest Rate Risk Management (2012) specifically encourages use of institution-specific assumptions. • An organization may feel that it has a special insight into market behavior, and that its assumptions about markets are superior to publicly available assumptions.

Validation Issues - Assumptions Biggest problems regarding assumptions – customer options: • Optionality in loans or securities

 Prepayments  Call features  Caps and floors • Optionality in funding sources  Core deposits  Callable/Putable FHLB advances

Best Practices for Developing Assumptions • Organizations should strive to use assumptions that reflect the institution’s profile and activities and generally avoid reliance on industry estimates or “default” assumptions.  e.g. national prepayment speed averages • Industry averages provide an approximation but are not a suitable estimate in every case. • An organization can contract with an outside vendor to assist with this process if necessary.

Model Processing • Model processing consists of the computer code and the theoretical models used. • The choice of theory is at least partly a matter of art rather than science: judgment comes into play in deciding what simplifications are acceptable. • The validation for the processing component of models should ensure that mathematics and computer code are error free.

Code and Mathematics • A number of procedures exist for testing code. Most models, such as those that operate in spreadsheets, have relatively simple code and equations, which can be cheaply tested by the independent construction of a similar model. • If the results of the two models agree precisely, it is usually highly unlikely that the two independently constructed models would contain precisely identical errors.

Code and Mathematics • For more complex models, independent construction of a similar model may be too costly. These situations require alternative practices. Some practices will: 1. Assign a modeling professional with the task of line-by-line proofreading of the code. 2. Compare model results to the results from a second, well validated “benchmark”

Code and Mathematics • An essential element of model validation is independent review of the theory that the organization uses. • Management should expect modelers to a) provide clear descriptions, in nontechnical terms, of the theory underlying the models; and b) show that the theory underlying the model has received recognition and support from professional journals or other forums.

Validation Issues – Model Code • Auditors/examiners are not responsible for evaluating code-level algorithms. • Code level review usually not conducted or required for annual validation process • Vendors usually have third party certifications of model code available – check to see if it is recent (3-4 years) and verify that there have been no major upgrades to the production model since the certification was conducted. • Internally developed models should have code level reviews periodically (but not annually unless major changes made)

Model Reporting • After processing the inputs, the model produces price or exposure estimates, or decision indices that will be used by decision makers. Obviously, the model validation process should assess the validity of those estimates. • However, it is equally important that the reports distilled from model output are clear and that decision makers understand the context in which the model results are generated. • Many of the procedures used to validate the input and processing components of a model are also useful for validating the model results. • At the time a model begins to produce outputs, model developers and validators should compare its results against those of comparable models, market prices, or other available benchmarks.

Backtests • Backtesting is another key element of model risk management.

• Backtests compare actual outcomes with model forecasts during a time period not used for model development.

• Model forecasts (usually expressed as an expected range or confidence interval around a central estimate) are compared with actual values for the back-test period, at a frequency and forecast horizon corresponding to the model’s intended use.

• Differences between actual and forecast values are analyzed to see if they are significant in magnitude or frequency.

Backtests • Significant discrepancies must be investigated to see if they are the result of omitted risk factors, misspecification or other errors.

• Effective, high-quality back-tests can help control model risk.

• But maintaining a rigorous separation between back-test data and data used to inform model development is not easy.

• The problem is made more difficult by the fact that, for financial markets, we are working with a limited amount of historical data.

• We are also usually building on the reported work of others.

Backtesting

• Issues

 How often?  Annually

 Semi-annually  Daily?

 Time horizon  Past Year

 Past Month  Past Week  Daily  Acceptable Variances?

Validation Issues - Backtesting • Backtesting will always indicate variances between model results and actual historical outcomes. • Institution must decide what variances are acceptable for key outputs and measures

 Values and prices  Earnings volatility

 Capital levels  Growth rates

Elements of Sound Validation Policy

Model Documentation • An inventory of models and their applications should be maintained. Policy should also require documentation for specific models that is adequate to facilitate independent review, training of new staff, and clear thinking by the model developer. • The most rigorous policies require documentation that is sufficiently detailed to allow the precise replication of the model being described.

Documentation • An organization’s model documentation policy should include the requirement for an executive summary that is made available to senior management. • The questions that models answer are invariably quite narrow in strict logical terms, so a clear statement of model purposes helps senior decision makers understand the limitations of the model.

Validation Issue - Documentation

• Technically, documentation is supposed to support the recreation of the model from scratch – usually not the case, especially for vendor models. • More importantly, documentation should be “desk level,” permitting someone relatively inexperienced with the model to generate accurate reports in the short term through a modeling cycle – screen shots, step-by-step “click-throughs”. • Protects against “key person risk”.

Reports • Independent review of a model’s underlying theory should always extend to the reports that transmit information from the modeler to the decision maker. An essential element of designing a model’s reports is ensuring that the results are communicated clearly and accessibly.

 Is the report right?  Is it the right report?

Record Retention • Policies and procedures for record retention that specify the retention requirements for:  Output reports  Model changes • Usually, record retention is addressed by an institution-wide policy to which the model validation policy refers  Data inputs  Assumptions

100

Stress Testing and Model Risk • But stress testing is also important in model risk management. • In particular, effective stress tests are an essential element of model development and model validation (essentially, ongoing monitoring and review). • For valuation models, two types of test are relevant:  Portfolio stress test: What impact would an unlikely but plausible adverse scenario have on the value of the portfolio?  Model stress test: checking performance over a wide range of inputs and parameter values in order to verify that the model is robust and understanding when it breaks down.

101

Model Stress and Stability Tests • The examples we have given so far are about portfolio stress tests for valuation models.

• At least, if not more, important in model development and validation are model stress tests for robustness and stability.

• How should such tests be conducted?

• What warning signs should we look for?

102

Model Stress and Stability Tests • Sensitivity analysis checks the impact of small changes in inputs and parameter values on model outputs.

• Unexpectedly large changes in outputs in response to small changes in inputs may indicate an unstable model.

• Simultaneous variation of multiple inputs may provide evidence of unexpected interactions. Can we show how these arise in the model, and can we explain them intuitively? • Stress tests should evaluate the model over a wide range of inputs and parameter values, including extreme values, to verify that the model is robust.

103

Applying Stress Test Results • Model stress tests establish boundaries for model performance.

• What is the acceptable range of inputs for the model?

• Under what conditions or for what parameter values or inputs may the model become unstable or inaccurate?

• What should we do if testing reveals that the model may be inaccurate or unstable in some circumstances?

104

Resolving Validation Findings • Any issues or findings from a model validation need to be assigned for resolution to specific parties by specific dates • Usually, the business unit that owns the processes are responsible for this resolution.  This can be the model developer if the finding is related to the math and mechanics of the model  More likely the issues are going to be around assumptions, so model users are usually tasked with resolving these • Risk management or internal audit are usually tasked with ensuring that resolutions are addressed timely, but are not responsible for implementing procedures for resolution. • Ensure that similar findings/issues are not outstanding across multiple iterations/years of the validation.

105

Exercise • Let’s go to Exercise #3: Case Study: Review a redacted model validation for a commercial bank’s risk model. • Exercises Pages 11-25

106

Model Risk Management

Regulatory Guidance and Best Practices

Session 1

Session 2

Model Development, Implementation, and Use

Session 3

Model Validation

Governance, Policies, and Controls Session 4

107

Session Objectives By the end of the session, participants will be able to: • Describe a best practice corporate model risk management governance organization • Three Lines of Defense (roles and responsibilities of model owner, validators, and auditors) • Model risk management staff • Evaluate reporting to Senior Management and the Board • Ensure model risk management policies and procedures are in place.

108

Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32 Page 33 Page 34 Page 35 Page 36 Page 37 Page 38 Page 39 Page 40 Page 41 Page 42 Page 43 Page 44 Page 45 Page 46 Page 47 Page 48 Page 49 Page 50 Page 51 Page 52 Page 53 Page 54 Page 55 Page 56 Page 57 Page 58 Page 59 Page 60 Page 61 Page 62 Page 63 Page 64 Page 65 Page 66 Page 67 Page 68 Page 69 Page 70 Page 71 Page 72 Page 73 Page 74 Page 75 Page 76 Page 77 Page 78 Page 79 Page 80-81 Page 82-83 Page 84-85 Page 86-87 Page 88-89 Page 90 Page 91 Page 92 Page 93 Page 94 Page 95 Page 96 Page 97 Page 98 Page 99 Page 100 Page 101 Page 102 Page 103 Page 104 Page 105 Page 106 Page 107 Page 108 Page 109 Page 110 Page 111 Page 112 Page 113 Page 114 Page 115 Page 116 Page 117 Page 118 Page 119 Page 120 Page 121 Page 122 Page 123 Page 124 Page 125 Page 126 Page 127 Page 128 Page 129 Page 130 Page 131 Page 132 Page 133 Page 134 Page 135 Page 136 Page 137 Page 138 Page 139 Page 140 Page 141 Page 142 Page 143 Page 144 Page 145 Page 146 Page 147 Page 148 Page 149 Page 150 Page 151 Page 152 Page 153 Page 154 Page 155 Page 156 Page 157 Page 158 Page 159 Page 160 Page 161 Page 162 Page 163 Page 164 Page 165 Page 166 Page 167 Page 168 Page 169 Page 170 Page 171 Page 172 Page 173 Page 174 Page 175 Page 176 Page 177 Page 178 Page 179 Page 180 Page 181 Page 182 Page 183 Page 184 Page 185 Page 186 Page 187 Page 188 Page 189 Page 190 Page 191 Page 192 Page 193 Page 194 Page 195 Page 196 Page 197 Page 198 Page 199 Page 200 Page 201 Page 202 Page 203 Page 204 Page 205

Made with FlippingBook - Online magazine maker