The co-moderator is raising an important topic for the Talkshop. Real world data and dimensionality, H, which might look normal but is not in time, with scaling, self similarity, perhaps maximum entropy at work.
The Hurst exponent.
And yet figure 2 looks normal enough. [as in statistical bell curve etc, given real data]
If you are familiar with this subject check I haven’t misled anyone and help out, add detail.
I would be stupid to attempt a proper explanation of this subject so I am going to write some snippets and give some pointers for elsewhere.
The story of Edwin Hurst has been expounded many times, a web search will return a vast array of information.
Try this, StatProb on Harold Edwin Hurst — “This is an instance of when a result is truly unexpected, is hard to comprehend, even by those best disposed to listen. New tools are not accepted without struggle.”
The head plot shows one of the most venerable datasets anywhere, the river Nile flood levels and it was work on this in Egypt which led Hurst to throw a spanner in the nice precise world of statistics, which to this day is widely disputed or ignored. And yet do you hear a great deal about flood or drought in Egypt, the Aswan High Dam works.
Figure 2 is a quick binning plot of the same data, frequency. I have put that there to demonstrate that looks are misleading. This could be taken as a plot of Gaussian noise… at your peril. Put it this way, tomorrow will not be the opposite of today, we get persistence of state, periods of dry, wet, hot, cold, wind, calm and so on. When is not encoded in plain noise, once again time is an omitted dimension.
River flow, origin of Hurst and rainfall are tightly related.
Rainfall and weather are tightly related, H spreads all across natural systems.
When Sligo of the Met Office amongst others recently declared on UK rainfall I somehow doubt they were speaking in skill but rather from a viewpoint of classical statistics. Critically probability is different with quantities which deviate from classical random, H=0.5, variance is wider. Has it really been so exceptionally wet, has weather gone mad, humans getting blamed of course?
I doubt it.
What I think is a good entrance point to the work of Koutsoyiannis and the superb ITIA web site is his 2003 paper
Climate change, the Hurst phenomenon, and hydrological statistics, Hydrological Sciences Journal, 48 (1), 3–24, 2003.
This includes showing statistical math which estimates probability taking into account H but also produces the usual answer if H=0.5
Head page for the copy of paper here http://itia.ntua.gr/en/docinfo/537/
There are many papers there and open discussions, some well known names appear. Link to publication list here
As a quick start here is a tool which computes H, a value which is an estimate. Given 10 different methods you will get 10 different answers.
A local usage Java tool (download and execute the jar file), works with V7 JRE, can estimate H using a variety of methods is here
Asks for details with email, seems straight “The SELFIS project is supported by the NSF CAREER grant ANIR 9985195, and DARPA award NMS N660001-00-1-8936, and NSF grant IDM IIS-0208950″
How about CET (Central England Temperature), use monthly, I’ve removed annual and centred mean on zero.
A good result, all estimators agree. H=0.7 or so, is typical for semi persistent data, such as Nile floods, a variety of things. (want the datafile, ask) Looks as follows
CET as used
Result using what was removed from the monthly set, annual cycle, showing patchy estimator performance but H is near zero, which means a simple cyclic data pattern. (CET data has a fixed offset so that might mess up the estimation)
Certain people don’t like this stuff one bit, wouldn’t if it says natural variation is much wider than the classic view, becomes difficult to justify humans doing much to earth systems.
Many blogs have touched on this subject, chiefio, WUWT, etc. providing many interesting takes. A list here would be nice (Koutsoyiannis links some where he takes part)
I’ve not touched Mandelbrot’s work on this one. (I have some about this here on paper, long forgotten where this detail didn’t register with me at the time, I think it needs real context before the penny drops)
Mistakes are mine. Please help improve this article and the understanding hereabouts.
Post by Tim Channon