Preparing for Basel II common problems, practical solutions part 5: artificial neural networks

RMA Journal, The, Sept, 2004 by Jeffrey S. Morrison

This article continues the series dealing with modeling requirements for Basel II. Previous articles have focused on missing data, model-building strategies, special challenges in model validation, and time to default. This article looks at a completely different approach to modeling PD (probability of default) and LGD (loss given default)--artificial neural networks.

When credit and repayment relationships are extraordinarily complex, it becomes far more difficult to arrive at the probability of default and loss given default. Previous articles in this series offered a variety of strategies, such as transforming the predictor variables, discretizing and binning the data, constructing dummy variables including interaction terms, and developing segmentation schemes using CART. (1)

However, it is possible that even these procedures do not go far enough. Developing interaction variables, for example, usually requires prior knowledge for the specification to be useful. Techniques such as logistic regression may prove less accurate than methods designed to handle nonlinear data.

An artificial neural network (ANN), often just referred to as a neural net, uses a mathematical approach to represent the processing of information in a fashion similar to the human brain. (2) Originally designed for pattern recognition and classification, ANN also can be used for clustering and prediction applications.

The terminology associated with designing ANNs can be so different from anything we're used to that even professional model-building practitioners can get lost in the nomenclature. In a 1997 paper (3), J. Stuart McMenamin attempts to bridge the gap between traditional econometric methods, such as regression analysis, and the concepts surrounding neural networks. He explains that neural networks are essentially nonlinear models that can approximate a wide variety of data-generating processes. Furthermore, at the heart of most ANNs is a recommended procedure for modeling probability of default--the logistic function.

So let's define some ANN terms by comparing them with terms used earlier in this series:

* In regression terminology, we estimate parameter weights or coefficients. In ANN terminology, these are called connection strengths.

* In regression, the parameter estimate for a particular variable is called its slope coefficient. In neural network terms, we call it the tilt parameter.

* In regression, we have a term for constants--the y-intercept. In neural network terminology, these are called bias parameters.

The objective for both processes is to find a set of parameter weights (or connection strengths) that will make the errors as small as possible. In regression, this process tends to be rather direct. In neural networks, however, the solution can be much more iterative and sometimes quite a bit slower.

Back-propagation. Although a variety of parameter estimation techniques abound in the industry, back-propagation may be the most popular. Basically, this is a repetitive process of using the estimated parameters to compute the forecasting errors when an adjustment is made so the next pass of errors is smaller. Back-propagation is an example of supervised learning because it requires knowledge of the "correct answers" as it learns the data and finds acceptable parameter estimates.

The Neural Network Structure

Figure 1 shows the most basic type of structure you will find in neural network software packages. This is called a feed-forward architecture. In this type of network, information is passed from one layer to the next in a one-directional fashion. No information is channeled laterally or backwards in the network. At the bottom of Figure 1, we see the process begins with the predictive variables at the input layer. These variables would be data such as LTV, loan type, debt to income, credit score, and so forth.

[FIGURE 1 OMITTED]

Next, we see this information is passed to the hidden layer containing the primary workhorses of the neural network--the activation functions. This is where the logistic functions or other nonlinear mechanisms reside. For a logistic function, the value is bounded between 0 and 1. Under some conditions, a node will take on a value close to 1, implying a firing or activation condition in a human neuron. Under other conditions, the firing or activation condition is not implied, as the node takes on a value closer to 0.

The more nodes you specify in the hidden layer, the more complex the network. It's typically up to the user to decide how many nodes to include in the hidden layer--that is, how much complexity the network will need to produce the best forecast possible. Usually, two to five nodes in the hidden layer will provide the best results. If too many nodes are included in this layer, the network can "overfit" the data, making it look predictive on training data but result in poor performance when new information is presented.

Finally, we see the process ends at the output layer, where the information from the previous layer is put together often in a linear fashion--not unlike what we might see in a regression equation.