Preparing for Basel II modeling requirements; part 2: model validation

RMA Journal, The, June, 2003 by Jeffrey S. Morrison

This article, the second in a four-part series, discusses some approaches to the validation of statistical models as required by the new Capital Accord.

As mentioned in the first article in this series, the current school of thought surrounding the probability of default (PD) and loss given default (LGD) models mentioned in Basel Consultative Papers is that banks should have separate models for the obligor and the facility. The obligor model should predict the PD--usually 90-plus days delinquent or in foreclosure, bankruptcy, charge-off, repossession, or restructuring. Models on the facility side should predict the loss given default (LGD) or 1 minus the recovery rate.

In the initial article, logistic regression was the approach recommended for building PD models. This statistical technique uses a set of explanatory variables whose values today would hopefully predict a loan's probability of default sometime over the next 12 months. On the LGD side, the approach recommended was to use either linear regression or to bit regression to estimate the model.

Directives from Basel II

Paramount to using the advanced approach as specified in the Basel II Capital Accord is a focus on model validation. The New Basel Capital Accord, published January 2001, includes the following:

302. Banks must have a robust system in place to validate the accuracy and consistency of rating systems, processes, and the estimation of PDs. A bank must demonstrate to its supervisor that the internal validation process enables it to assess the performance of internal rating and risk quantification systems consistently and meaningfully.

305. The process cycle of model validation must also include:

* ongoing periodic monitoring of model performance, including evaluation and rigorous statistical testing of the dynamic stability of the model and its key coefficients;

* identifying and documenting individual fixed relationships in the model that are no longer appropriate;

* periodic testing of model outputs against outcomes on an annual basis, at a minimum; and

* a rigorous change control process, which stipulates the procedures that must be followed prior to making changes in the model in response to validation outcomes.

As of yet, The New Basel Capital Accord does not give specifics or standards related to the validation process.

Introduction to Validations

Validation includes issues of data quality, documentation, sensitivity analysis, model specification, sample design, the performance of statistical tests, and the development of measures for model accuracy. Although not minimizing the importance of these other areas, for brevity's sake the remainder of this article will focus on quantifying accuracy measures. In this light, model validation simply refers to checking the accuracy of your model over some specific period of time. How many loans actually went into default during the year and what did their predicted default probabilities look like? If most of your defaults had a predicted probability of default near 10%, then your model may be doing a poor job.

Not only is the validation process part of Basel requirements, it is central to any model development process, regardless of its application. Even econometric forecasting models--models developed using aggregated data with economic time series--are validated for accuracy. Credit-scoring models are also validated for accuracy. Because modeling is indeed an art, statistical algorithms are developed and redeveloped until a formulation is found that reflects the most accurate results and makes the most business sense.

Validations can be done in a variety of ways, ranging from the simple to the complex:

1. Performing the validation only on your model development sample.

2. Performing the validation on a sample of accounts that were not used to develop the model, but were taken from the same period of time.

3. Performing the validation. on a single holdout sample from time periods outside your model development window.

4. Performing a step-through simulation process across multiple time periods while recalibrating the model.

If there are sufficient defaults available, the second method is preferred. A random sample of data is held out from the model estimation; the second method runs the holdout data against the model to compute its predicted values for validation purposes. This method is widely used for validating a variety of different models and serves as an aid to the statistician in selecting the best model.

The first approach is the most straightforward and is typically performed as the model is developed. Here, the same data that was used for estimating the model is used for validation. Although this type of validation tends to overstate the model's predictive ability, it may be necessary if there are a limited number of defaults available for model building purposes.

The remaining methods are more advanced--not because their techniques are necessarily more complicated but because they require a greater depth of default history. The third approach holds out data for validation from prior periods to see if the level of accuracy remains the same from year to year. This is an indication of how stable your model may be over time. The fourth approach is a combination of validations and model recalibrations. The idea is to simulate model development and its predictiveness over time given that model revisions are done annually as new defaults are accumulated and added to the process.