Dr Robert Brown: The pitfalls of extrapolating trend function fits to data

Posted: November 8, 2012 by Rog Tallbloke in Analysis, data, Forecasting, Incompetence, methodology

I picked up this comment from Robert Brown on a WUWT discussion of the latest UAH anomaly from Roy Spencer, who no longer adds his ‘for entertainment only’ polynomial trend fit to the data. Michele Casati’s friends would be wise to take heed.

rgbatduke says:

Fitting data with meaningful functions is a basic statistical technique. Fitting data with arbitrary smooth functions is indeed meaningless — literally — and produces nothing more than “a guide to the eye” even if the fit works. In particular, it has little predictive power — many functions will interpolate or approximate a finite data set decently but have very different behavior outside of the fit interval, and the actual data (as it continues outside of the fit) cannot agree with all of them and may not agree with any of them.

Roy [Spencer] understands this perfectly well, because Roy actually understands mathematics and statistical methods pretty well. I personally am rather an expert on predictive modeling methodology, and wrestle with these problems all of the time. There are a few modeling techniques that effectively fit with a large basis to obtain some extrapolative success — neural networks, for example — but methods based on harmonic series e.g. Fourier analysis are notorious for their problems.

That isn’t even an issue here. Roy is (was) fitting a trivial smooth nearly harmonic curve that we all know would not extrapolate on the far side to fit the existing past data, let alone future data. I’m guessing he left it out because its inclusion was too encouraging to folks like Henry P who think that because they can achieve a crude approximate fit with a few hand-selected fourier components that they have discovered the secret of the ages and can thereby predict the future progression of global temperature, to the point where he is “certain” that his model is correct and therefore UAH’s actual data must be wrong. It also distracts the observer from doing the obvious — letting the data speak for itself.

In a noisy, chaotic series like this where there is little reason to select any given analytic basis as being “meaningful”, even linear fits are pretty meaningless, especially when performed on a remarkably short data series. You can see this lack of meaning by observing the rather large variation in the “best fit” slope of a linear trend resulting from “cherrypicking” the end points within the series. Shift the end point a few years, or even months, at either end and you can make the slope vary by a large factor, implying that even the linear trend in the data is highly uncertain.

The Koutsoyiannis paper on stationarity vs nonstationarity hydrology — the one that caught my eye several years ago as a “playah” in this game — has one of the best discussions of this point that I’ve ever seen. It should be required reading not only for all climate scientists (although especially for them) but for all scientists, period. He presents a series of fits to successive windows on the same set of data to show how utterly misleading and meaningless they all are as to the actual (rather simple) functional behavior of a given data set generated from an analytic function plus noise. I commend it to you:

http://itia.ntua.gr/en/docinfo/673/

(click on the preprint PDF or download the possibly paywalled actual publication). Note also that this page has a number of his other papers on this general subject — which broadly speaking is the introduction of what he calls Hurst-Kolmogorov statistical analysis into climate science, a sort of punctuated equilibrium model that describes one particular aspect of global climate almost astonishingly well (I reiterate, Bob Tisdale needs to apply it to SSTs as they obviously are a “perfect fit”).

Henry P would also benefit tremendously from reading the introduction to this paper. Note well that Koutsoyiannis is keeping it simple and only illustrating three distinct windows onto the data. He is also keeping it functionally simple as none of the functions that appear to fit in a window are unique even in that window — a linear fit or exponential fit would clearly work nearly as well as his parabolic fit, a fifth order polynomial would fit the entire sequence pretty well (and if not fifth, sixth or seventh — Wierstrauss’ theorem after all). Koutsoyiannis himself points out that if the series continued, his beautiful cosine fit could turn out to be fitting nothing but noise on an even longer timescale meaningful trend!

The point of this — in case it eludes you — is that merely fitting a finite segment of data to a functional representation of any sort is right up there with Tarot and Tea Leaves as far as having predictive value is concerned. A fit that worked remarkably well — “convincingly” well to the uninitiated — in the first two windows proves to be completely and damningly wrong in the third, not only wrong but literally irrelevant — even as the third window seduces you to conclude that the cosine law is itself meaningful just because it works across this finite sample.

Fits like this have some degree of reliability and extrapolability under only two circumstances. One is when there is a sound physical argument to support the use of some particular fit function, one where the parameters of the fit themselves provide actual information about the physics or other dynamics of the system. The other is where, over time, empirical experience is that some particular fit scheme just works, at least so far, in a robust way over a very long series and works — so far (!) — to extrapolate the series as time continues to evolve. The latter is enormously dangerous — so much so that Nassim Nicholas Taleb wrote an entire book (The Black Swan) criticizing its widespread use in pseudoscientific modeling of essentially unpredictable series wherein “black swan events” are known to occur with some unknowable frequency. All too often they support some sort of Martingale system — doubling down to ride a comfortable linear trend. Sometimes, however, the model fit scheme does have meaning and correspond to physics, we just don’t know how (yet), or is so general that when built in a certain way the model itself can replace the human brain and make information-theoretic compressions that correspond to unknown but real internal dynamics that are empirically safely extrapolable.

The latter is one of my primary businesses — literally, as fits/predictive models of this sort are enormously valuable when you can build them — but they are not for tyros to undertake. I’ve often thought about building a really clever neural network to model the entire planet, but this is a ten million dollar five plus year sort of project even for me — the computer required to build and run and refine it would all by itself be pretty expensive — but if done very cleverly I think it could manage the integrodifferential evaluation back to timescales in the remote past, robustly, allowing for noise and missing information to give us perhaps the best possible model for global climate ever built (and one that by its nature would be utterly unbiased, as AFAIK it is literally impossible to introduce a meaningful and deliberate bias into a standard neural network build process).

rgb

Comments
  1. Robert Brown is, as usual, annoyingly right!

    In my opinion, all curve fitting is just a subset of astrology – at least if it is used, as it often is, implicitly or explicitly, to imply a future trend.

    That is why I believe that the only rational way to graph the world mean temperature series is to display it raw and then overlay an end-to-end linear regression trend line (no subsets of the full time span allowed). The slope you get with the HadCRUT data is sobering – a rise of 0.41degC/century: http://www.thetruthaboutclimatechange.org/tempsworld.html

    This number should not be regarded as predictive beyond the upper end of the measured time span. However it is wholly appropriate to use it as a simple sanity check. It provides a counterpoint to alarmism – but without dabbling in complex, usually meaningless (and certainly to the lay public incomprehensible) statistical techniques.

    Most intelligent people take an end-to-end linear regression line not as a statistically valid ‘curve fit’ but simply as a mental reference value against which they can make sensible judgements about the wilder claims of future warming – for example, the predictions of a 3 to 5degC rise by the end of this century – which immediately look kind of foolish when judged against the 0.41deg/century linear slope.

    It is part of human nature that some people will make extreme predictions about any societal issue. So it is vitally important to have sober references against which to judge such claims. The general lack of reference values in climate science goes a long way to explaining why the world is now in the grips of a climate-alarmist political class.

  2. Stephen Richards says:

    Thanks Robert

    You clearly spend your days explaining your subject to us lesser mortals. Oh that I was one of your students when I took my degrees.

  3. Michael Hart says:

    Thanks, Robert.
    As I’m sure you’re aware, there will always be fresh young minds (and not-so-fresh older ones) who need these things to be pointed out.

    I had always assumed that Roy Spencer’s “fit” was a running joke, probably mocking a person who’s name begins with K- and ends with -evinTrenberth. Even the best jokes need replacing or updating from time to time.

  4. Doug Proctor says:

    Robert Brown says:

    “Fits like this have some degree of reliability and extrapolability under only two circumstances. One is when there is a sound physical argument to support the use of some particular fit function, one where the parameters of the fit themselves provide actual information about the physics or other dynamics of the system. The other is where, over time, empirical experience is that some fit see just works, at least so far, in a robust way over a very long series and works — so far (!) — to extrapolate the series as time continues to evolve. The latter is enormously dangerous …”

    As an exploration geologist, I understand only too well the dangers of predicting behaviour based on imperfect data, imperfect data distribution and inadequate models. We call these dangers “dry holes” and they are an excellent source of bankruptcy. They are, however, what we have to use. And while we recognize what we do is less than satisfactory, we struggle ahead with the advice that Patton gave: “A good plan today is better than a perfect plan tomorrow”.

    The climate wars are characterized by agenda science more than critical science. Schmidt, Hansen and others use linear relationships to their temperature profiles for two reasons. The first, the most important of the two, is that they hold CO2 to be dominant, progressive and in the shorter term, linear in its effect. What are clearly cycles in the record are just background noise, even though “signficicant” in a visual sense. The other is that use of linear relationships as a defining trend allows you to plough through questionable times like the 1997 – 2013 period and project into the future without concern.The times are not a-changing, the boservations are just messy. As much as I disagree with their agenda, they are trying to do what I have to do: figure out where we are going on the basis of understanding where we have been.

    The reason that geologists keep jobs is that there are multiple ways to interpret data. Likewise CAGW climatologists. Schmidt, Hansen et al are victims of what I have seen in numerous young geologists and most engineers: a blindness to what I call the Unique Solution Syndrome. They think that there is only one solution to any problem, and since they have found one/it, then all others must be false. It is a very comforting position until the well has nothing but salt water in it, or the temperatures drop. (Interestingly, this error does not mean that the position they take is rethought, but that they look and find where someone else gave them inadequate data on which they based their prediction. Such is man and men.)

    Brown is absolutely right in what he writes. We do not have the luxury of waiting until the story is done to determine where the story is going, however. What we can do is recognize that what we are doing is often falsely based. But not always without value.

    I was once told that we don’t need to understand geology in order to find oil. All we have to do is drill where the oil is. If noting the density of birch trees gives us the right drilling place 4 out of 10 times, then the guy who knows a birch from an elm is the brilliant one. So to with temperatures: if you can find a pattern that works (Brown’s second point noted above), then you are on to something valuable.

    The other reason that geologists continue to have jobs is that the technique that they used in this township successfully leads only to dry holes in that township. This would be the birch tree problem, as birch trees may not be a predictive item elsewhere. The “proxy” doesn’t reflect the conditions that lead to the oil deposits everywhere, but is highly correlatable in a local setting. The next geologist uses his imagination to find a reason to drill and if successful, becomes the One to Watch.

    No proxy is a perfect proxy unless the causitive forces that lead to its character are the same causitive forces that lead to the character of that which we wish to understand. The Unique Solution Syndrome means that those, like Mann, misunderstood a useful but limited connection to be a necessary connection, in other words. CAGW is based on a requirement for necessity in a world of sufficiency.

    The world will out. Academically we can say “we don’t know”, and that is true. But what we really need to know is not what is happening in one hundred years, but what is happening in the next ten. If global temperatures do not rise again (and exceed the last 30 years of trendsf), then the CAGW premise is false. There are other reasons a 1.4C/century temperature rise can occur, and the power of cannot be near that of the IPCC narrative. If the temperatures go down (as multiple skeptic theories suggest) then CAGW is again based on a false premise.

    So I look to the patterns that one sees and recognize that in the near-time the future is likely to be like the past. What was yesterday and is today is likely to be a good indication of tomorrow. How far can this simplistic thinking work? There are obvious 5 or 6 year cycles of up and down that look good, and if you lose the linear-function block, a fairly obvious curve to the last 30 years. Extending into the future like this is not mathematically “correct”, but what else are we to do?

    Pragmatically, we need to move forward intelligently but aware of the limits of our intelligence (in a military sense). Hansen et al want to jump forward because they have found The Answer. If, as skeptics on CAGW, we don’t believe that the science of today is different from the science that came before, we have the advantage of looking for the patterns of before and projecting them into the future. Remember that the near future is all the future we need to understand. Beyond the near-future, the further future will be informed by its recent past, our soon-to-be time.

    The climate wars have taken us beyond science. They have taken us into the industrial world of practical matters. Hansen, Gore, Suzuki and Erhlich like to think they are acting in a practical matter but they are not. They are thinking like academics who have gotten hold of The Solution. We, on the other side, cannot hobble ourselves by recognizing the limits of our knowledge. But we can see the probabilities inherent in limited projections as high, and that may be all we need.

    I think the politicians are aware of the problem the Hansens have given them. While they may be philosophically supportive of anti-consumerism (in other people!), the next couple of years could easily show the purported disaster to be in the minds of the doomsayers only. They are looking at the trends also, using the eyes their mothers gave them. This is the story we should be giving: these are the non0CAGW options and the time-frame for them.

    Pragmatism says you can’t get to the truth, but you can approach it. Sometimes that is enough. Like now.

  5. Thanks very much Robert–for the post and for the kind words about my paper. Hope to meet somewhere some time.
    Demetris

  6. Doug, you say: If global temperatures do not rise again (and exceed the last 30 years of trends), then the CAGW premise is false. There are other reasons a 1.4C/century temperature rise can occur, and the power of cannot be near that of the IPCC narrative.

    It’s 0.4degC per century not 1.4degC. Quite a difference and a wholly different order of embarrassment for the Hansen’s of this world.

  7. Ray Tomes says:

    Much of even accepted physics is based on curve fitting. Curve fitting can be useful for interpolation, but is next to useless for extrapolation. The test of whether you have a really meaningful model of behaviour or mere curve fitting is whether it can reliably predict the future. As many sunspot would be predictors have discovered, this can be a tricky process. Interestingly, cycles analysis has been more successful than other methods for sunspot prediction.

    To go back to my opening sentence, modern cosmology has not made useful predictions much beyond the range of its measurements and should not be considered a viable theory. Particle physics similarly does not have a great track record. I do not consider predicting a particle mass with 30% uncertainty limits to be a useful feat. I have done much better myself (2.5% error only).

  8. Brian H says:

    Tieing curve functions and parameters to physics is crucial, both for physics and the curve. Mutual falsification discipline must be imposed, however, once they are married. If the curve faithfully represents the physics, observational deviations from it invalidates the physics. And if the curve misrepresents the physics, it cannot be extrapolated. The Climate Science (IPCC-style) trick of painting huge swathes of allowable error (90% ‘certainty’) just doesn’t wash.