It's a long way to Monte Carlo: simulated solutions of commonly used financial models do not converge very quickly

Business Economics, July, 2002 by Michele Gambera

This paper presents practical examples on how Monte Carlo simulation software must be evaluated for precision. The examples are from financial planning cases, but the relevance of the methodologies displayed here for other probabilistic problems, such as VaR and stress testing, is clear. Users must be comfortable not only with the assumptions of the models they use, but also with the reliability of percentiles, means, variances, and other moments of the final distributions that are produced by the software solving such models. Legal and reputation risks are the most obvious consequences of overlooking this issue.

**********

Monte Carlo simulation is one of the best-known solution approaches for probabilistic financial problems. Since its adoption by financial practitioners for stress testing and other problems has occurred rapidly, not everyone has had a chance to weigh pros and cons of the approach. This paper discusses the implications that the choice of the batch size (sometimes called the number of experiments, of replications, or of "runs") has on the reliability of a simulation. In the technical literature, this is called the convergence speed issue. This paper analyzes the impact of batch size on the solutions of two common financial planning problems. The results unfortunately show that, in practice, simulation accuracy can be easily overlooked. Thus, placing blind faith in simulation results could cause financial tragedy for clients or employers and, quite possibly, lawsuits and liabilities.

The practical implication is that, when running simulations, one should make sure that the batch size is sufficient to yield an estimate within an acceptable range of the true solution. When writing software, this can be achieved by using statistical software or a programming language offering double precision rather than a spreadsheet, and by obtaining error estimates using the methods reviewed in this paper. When obtaining the software from a third party, the producer should provide error estimates and indications of the batch size.

Solutions to probabilistic problems are generally moments of random variables (such as means and variances), which can be expressed as integrals. Integrals, in turn, may be solved either analytically (that is, by algebraically finding either an exact or an approximated "quadrature" solution) or numerically (that is, by computing an approximated solution). Since the solutions generally represent estimates, numerical solutions should be used when the exact analytical solutions are either very difficult or impossible to derive. Monte Carlo is one of several reasonable approaches to numerical integration, as explained by numerical integration manuals such as Davis and Rabinowitz (1984) or Zwillinger (1992).

The speed of convergence affects a very practical aspect of a Monte Carlo simulation, namely, its precision. As explained in the examples of this paper, if the batch size is too small, the simulation may yield a result that is substantially far from the truth. Press et al. (1992) point out in their famous book (p. 26) that Monte Carlo is appealing because it is relatively easy to program but has precision problems. Of course, for a planner, an incorrect answer implies incorrect financial advice. For a risk practitioner, a wrong estimate of VaR may imply shareholder lawsuits or the employer's financial distress.

In this paper we test models of retirement spending, that is, of the probability of outliving one's money given a wealth amount, an asset allocation, and an annual withdrawal amount. The problem is written as an example, and some of its assumptions are unrealistic, because the point of the paper is testing convergence speed and not replicating the results of studies such as Cooley et al. (1998) (also known as the "Trinity Study"), Jarrett and Stringfellow (2000), or Milevsky (2001). However, the results of the paper show that the authors of similar studies should provide detailed information about sample sizes and error estimates. This paper borrows examples from the financial planning literature, but its results can be clearly applied to VaR problems.

Models and Results

The models used in this paper are similar to those used in the papers listed above. There is a single person of age sixty, who can invest in a mix of three assets whose random returns in the same year are correlated (no serial correlation). The returns are lognormally distributed with constant moments (mean, variance, covariance). The person chooses a fixed allocation and performs annual rebalancing. There is no inflation, and the person withdraws a fixed dollar amount per year, starting at the end of this year, until death.

For simplicity, we test only one such allocation (i.e., sixty percent stocks, thirty percent bonds, and ten percent cash). It is important to remember that this paper is not concerned with the validity of an asset allocation or of a spending amount, but with the precision of the estimated outcomes.