New Here? Get the Free Newsletter

Oblivious Investor offers a free newsletter providing tips on low-maintenance investing, tax planning, and retirement planning. Join over 9,000 email subscribers:

Articles are published Monday and Friday. You can unsubscribe at any time.

Financial Simulations: Should You Trust Them?

Financial simulators–broadly grouped into a) historical return calculators and b) Monte Carlo simulators–are popular tools for financial planning. But it’s important to recognize their limitations.

Historical Return Calculators

Historical return simulators (e.g. FireCalc) allow you to test a given strategy against historical returns to see how often it would have worked. For example, you can check how often a 4% starting withdrawal rate would have been successful over a 30-year retirement given various stock/bond allocations.

Such calculators are useful for showing what has not worked in the past. Showing that a strategy has worked only occasionally tells us that we should have little confidence that it will work in the future. That’s why, for example, we know that it’s unwise to plan to withdraw 7% of your portfolio every year during retirement.

Monte Carlo Simulations

Monte Carlo simulators (this one, for example) allow you to perform similar tests. But instead of testing a proposed strategy using actual historical sequences of returns, they ask you to provide statistical descriptors of investment returns (average return, standard deviation of returns, correlation to other investments, etc.), then they test the proposed strategy against numerous return sequences generated using those descriptors.

Monte Carlo simulations are especially useful for testing how much a plan’s probability of success will change as a result of changing assumptions. (For example, if stocks end up being 10% more volatile over annual periods than they’ve been historically, will that be a major problem?)

Are Historical Returns Meaningful?

Consider this analogy: You’re trying to determine the average height of a group of people (as well as other facts such as the standard deviation of heights among the group). With every additional person from the group that you measure, your data set grows and you can be more confident in your conclusions.

We try to do the same thing with historical returns–collect an ever-growing pile of data and use it to determine things like average annual stock market return.

But there’s a problem here: As our sample size grows, our population could be changing. For example, I’d assert that the financial markets and world economies are meaningfully different from, say, 50 years ago in several ways (examples: instantaneous information on stock, bond, and commodity prices; automated trading in very large amounts by institutional investors).

What effect will those changes have on investment returns in the future? I don’t know. But I don’t think we can simply assume that such changes will have no effect.

As such, any data older than 50 years is of limited value. As we continue to collect more data, we have to keep throwing our old data out as it becomes less and less relevant. Even today’s data may not be particularly relevant if you’re concerned with returns several decades into the future.

Conclusion: The predictive value of any simulations based purely on historical data must be taken with a healthy dose of skepticism.

New to Investing? See My Related Book:

Book6FrontCoverTiltedBlue

Investing Made Simple: Investing in Index Funds Explained in 100 Pages or Less

Topics Covered in the Book:
  • Asset Allocation: Why it's so important, and how to determine your own,
  • How to to pick winning mutual funds,
  • Roth IRA vs. traditional IRA vs. 401(k),
  • Click here to see the full list.

A Testimonial:

"A wonderful book that tells its readers, with simple logical explanations, our Boglehead Philosophy for successful investing." - Taylor Larimore, author of The Bogleheads' Guide to Investing

Comments

  1. Good post. You are right on regarding the limitations of financial simulation tools. As a financial advisor I use these types of tools as ONE input. Nobody should rely on the output of a model to make financial decisions. They are a tool, nothing more, nothing less. The input and assumptions are a critical piece and the process of formulating these inputs is valuable in itself. At the end of the day financial decisions should be made by individuals based upon various pieces of information. The output of these models is but one piece of the puzzle.

  2. Healthy doses of skepticsm are good, but remember that the idea of a 3%-4% inflation adjusted SWR is entirely based on historical data. If you don’t put any faith in the historical data, then you can’t put any faith in that SWR, either.

    It should also be noted that while you’re focusing on the failures in the predictive models, one should also look at the successes. Using the 3%-4% inflation adjusted SWR on historical data, a portfolio has a good chance of becoming a runaway success.

  3. “If you don’t put any faith in the historical data, then you can’t put any faith in that [3-4%] SWR, either.”

    I disagree. The way I see it, for those comfortable assuming retirement will last 30 years or less, a 3.33% (100% ÷ 30 years) SWR would be possible if the portfolio precisely kept up with inflation each year–no better, no worse.

    Granted, portfolios can do significantly worse than keeping up with inflation in any given year (especially depending upon asset allocation). But if you’re comfortable assuming that the years above inflation will be sufficient to make up for the years below inflation (including impact of potentially bad sequence of returns), then a 3.33% starting withdrawal rate could be assumed to be safe.

    To me, such a back-of-the-envelope calculation gives approximately as much confidence as historical studies.

  4. Murnal Abate says:

    “But there’s a problem here: As our sample size grows, our population could be changing.”

    Doesn’t that mean that this new information is then built into the resulting measure of mean, STD, etc of the new distribution? In other words, historical data might very well be useful in estimating the trend of movement in stock, bond and commodity prices. Not perfectly, but perhaps usefully enough to serve our predictive needs?

    Otherwise, another in a long line of interesting blog posts…I’ve become a regular reader and generally enjoy your analyses.

  5. Hi Murnal.

    Yes, new data will be built-into the calculations. But the problem is that (if old data really is taken from a meaningfully-different population than new data), we have to throw old data out well before we reach a point of statistical significance.

    For example, if we want to know how the market typically performs (as measured by annual rate of return) over 10-year periods, we only have ~9 distinct data points in the first place. If some of those are no longer meaningful, we’ve got even less. And if markets continue to change in meaningful ways (so as to continually make old data irrelevant), we’ll never reach a point of statistical significance.

  6. I love old data. If we looked only at the last 50 years, we wouldn’t see the Great Depression. Nor the recovery from it. Yes, things were quite, quite different back then, but the market was still made of people investing in what was available with the same sorts of emotions, etc. Some of the changes, if they work as planned, have made another Depression less likely. Other changes may have new and unexpected consequences.

    So while I agree that past data is all selected from a time period different from the time period that we are (usually) wondering about, and therefore must be taken with several large grains of salt, what I like about it is that it is data from stuff that really actually happened in the real world full of infinite variables, etc. Looking at the past shows us just how big those swings can be and just how long they can last. For example, I now know that just because things are greatly overprices does not mean that they’re probably about to plummet. It could be four more dizzying years before that happens. Maybe even decades.

  7. I’ll toss in my 2 cents on this one. I don’t think there is any predictive value in simulations at all. I think that is a common misuse of Monte Carlo.

    I do see value in measuring probabilities, but it is more so for the purpose of answering the question of whether or not current efforts are reasonable. And if not, what ought to be done to address that. It can serve as a repeatable test as to whether savings rates, withdrawal rates, and asset allocations are reasonable for your financial plan.

    It’s almost impossible to nail down the perfect financial plan (i.e. no over- or under-doing it at all). But since both overplanning and underplanning have costs we’d rather avoid if possible, we can approximate and maintain a comfortable balance between the likelihood of each. This application, especially if repeated periodically, is a lot more trustworthy than a prediction.

  8. Skeptisicm? I’ll say.

    Might as well use the fortune telling black 8-ball thing that you shake.

    How does any tool that crunches numbers based yesterday take into account the big moves of future technology….like Facebook, Google, etc.

  9. Debbie: Agreed. Historical data is certainly useful for showing what can happen.

    Dylan: Thanks for sharing your thoughts. As always, your input is much appreciated. You’re nearly the only person I’ve heard who talks about the risk of “overplanning” (i.e., underspending/oversaving), but I think it’s a great point.

    Here’s my question: If MCS has no predictive value, how is it useful for determining whether you’re over/under -planning? (Perhaps you mean something different than I initially assume when you say simulations have no predictive value at all.)

  10. People often refer to MCS results as a “probability of success;” however, the result counts every trial that ends greater than or equal to the pass/fail point. It’s really more of a “probability of excess.”

    If MCS yields 80% successful trials, that’s really an 80% chance you’re over-planning and a 20% chance you are under-planning (the odds that you’ll nail it exactly are close to nill).

    It does not predict how much you can spend or how much money you will have in X years. It’s kind of like me predicting the New England Patriots will win two games next year. As long as they win at least two games, my “prediction” is correct, even if they win 16.

    The MCS result is also not predictive because if you are (as the odds suggest) over-planning, you will likely change what you are doing as that fact reveals itself simply because you can. This is often the reason why people can afford to move to a more conservative allocation later in life.

    An 80% MSC result, although not predictive, is still useful information because such a division (4:1) between the likelihood you are over-planning vs. under-planning can be comfortably maintained over time without drastic corrections (most minor corrections will be for the better). It yields a high degree of confidence without an unnecessary/unreasonable amount of sacrifice. As life presses on the range of outcomes narrows, minimizing the degree to which you are either over- or under-planning eventually resulting in your desired outcome in the end.

    This idea of measuring your progress at regular intervals and making minor course corrections as needed is a very similar concept to how the sailing ships of centuries ago successfully navigated to tiny islands in the middle of oceans without GPS. MCS simply measures your progress.

If you want to discuss this article, I recommend starting a conversation over at the Bogleheads investing forum.
Disclaimer: By using this site, you explicitly agree to its Terms of Use and agree not to hold Simple Subjects, LLC or any of its members liable in any way for damages arising from decisions you make based on the information made available on this site. I am not a financial or investment advisor, and the information on this site is for informational and entertainment purposes only and does not constitute financial advice.

Copyright 2013 Simple Subjects, LLC - All rights reserved. To be clear: This means that, aside from small quotations, the material on this site may not be republished elsewhere without my express permission. Terms of Use and Privacy Policy