Signal processing and normal meteorological maths

Posted: October 22, 2011 by tchannon in climate

Some time ago I experimentally processing some temperature data to see the real effect using signal processing instead of the usual stone age common math. Did it matter?

reynolds-creek-1

Figure 1

Figure 1 shows the effect.

Reynold’s Creek is a 10 year plus experimental environment monitoring experiment carried out in north west America where many parameters were measured. The data is published where it can be trusted and very unusually hourly data is provided. Discovering necessary time resolution data is rare, usually withheld.

The work is incomplete but I’d plotted that so it will do given there is talk on the ‘net about malpractices over Nyquist and I add, Shannon.

I simulated a min/max thermometer in software and processed using common human time period averaging. This is the most common method but if you research you will find many other methods, rarely used today.

The actual process is decimation, which is the trade between time and amplitude resolution. With signal processing low pass filtering is used extensively to meet Nyquist. I do not though throw away resolution gained.

I do not need to point out it does matter, you can see that.

A surprise might be the reduction in noise amplitude for daily data and if you looked at the spectra the gravel effect of introduced artefacts from the common math is missing.

Also notice that monthly is very different with an excellent example on the second winter. Correctly signal processing ignores human constains of month and sees spikes which do not fit with month. The normal method will fail to show say a 10 day cold patch around the turn of a month the same as the same patch within a month.

That will mess up things like month and year rankings.

There are major problems. The data generally is finite and therefore the data end problem is present. This is tough with no maths pure solution.

Strictly, defining a graph shape demands say three datapoints per month or year.

I had lost the data link but search found it. Here is the experiment home page.

Tim Channon (Not Tallbloke)

Comments
  1. Brian H says:

    02/02/85 and ~28/08/85 are very interesting; the real averages are near the min & max, respectively, while the median/reported averages are of course much higher/lower respectively. Cases, I gather, where the day spent most of the time near one extreme with brief excursions.

  2. Eilert says:

    Eye balling: There seems to be a 27-28 day rythm in the daily signal processed data.

  3. malagaview says:

    Eye balling: There seems to be a 27-28 day rythm in the daily signal processed data.

    Eye balling: There seems to be a stronger 365 day rhythm in the daily signal processed data.

    Conclusion: The annual cycle in temperatures is primarily driven by the sun and then modulated by the moon….

    Now where have I heard of the Solar-Lunar weather forecasting technique? 🙂

  4. malagaview says:

    Eye balling: There seems to be a 27-28 day rythm in the daily signal processed data.

    / RANT ON
    This is a prima facie example of where the Settled Science of Climatology is complete flawed (although I could think of another word beginning with “F” that is more appropriate).

    Lets take a look at the published Daily Average Temperature for any weather station… lets looks at what the published number actually represents.

    The published number is calculated from the Daily High Temperature reading and the Daily Low Temperature reading for the day in question… these two numbers are added together and this intermediate value is then divided by two… so the mathematics are clear… the result of this calculation is the mid-point between the Highest and Lowest temperatures for the day… this (therefore) is NOT a Daily Average Temperature… it IS the Daily Mid-Point between Extremes… it is the mid-point between the two most extreme outlying temperature points of the day.

    In my experience of statistical analysis outlier values are always suspect… sometimes these outlier data points are simply rejected as errors… sometimes the data points are tested for reasonableness before inclusion… either way: the statistical analysis of extreme outlier temperature values is not related to the statistical analysis of meaningful temperature averages.

    The other statistical problem with this published number is false precision… the accuracy of the high and low data points is subject to the precision of the thermometer… and the error margin is not usually linear with temperature… so we have an error margin. Additionally, there is a precision limit: are the thermometers measuring in whole degrees or tenth of a degree or hundredths of a degree or what? So the two outliers daily data points always have a plus or minus error… then we calculate the mid-point to as many decimal places are necessary… this precision of this result can far exceed the accuracy of the underlying data points… and this result may need to be truncated or rounded so it can be published. Basically, we have to take the published figure with a grain of salt, unfortunately we don’t know how big this grain of salt really is.

    Digger deeper into a published Daily Average Temperature we have to ask ourselves:

    Are the values Daily?
    If the daily data points are read (an reset) at midnight then the data points are from the current day (that has just ended)… if they are read at any other time of the day then they are NOT daily but (by definition) taken from two different calendar days. The data points must always be taken exactly 24 hours apart… any reading that is late, early or missed has the potential to destroy the integrity of the data series (if any exists).

    Whenever the data points are captured during the day… which is when humans are typically active… then the readings may be taken before the highest (or lowest) temperature is actually encountered for that day… so basically the whole daily notion underlying theses calculated results is totally suspect… and this is just for one station on one “day”.

    What Temperatures are they Measuring?
    Depending upon the sitting of the thermometer the temperature data points should have some relationship with the ambient air temperature in the location… but in many instances this is not the case… sometimes they are measuring the temperature over tarmac, cement and dark roofs… sometimes the temperature of aircraft exhaust fumes… sometimes the temperature of air conditioner outlets… sometimes the temperature around a barbeque…

    What do theses published Average Temperatures actually Represent?
    This is the key question… the answer is: nobody knows… they are numbers with some vague relationship with ambient air temperatures… but it is impossible to say exactly what they represent…. or whether they are representative of the general geographic location… or wider area… or variable sized data grids… let along Monthly Averages.. or Annual Averages… or as a meaningful constituent of any other data analysis.

    Proof of the Pudding
    The evident obscuration of the Lunar effect in the published Daily Average Temperatures indicates how meaningless these values are.
    /RANT OFF

  5. tchannon says:

    I’ve never seen nor heard of signal processed vs. conventional, so it is possible you are looking at new.

    Obviously the day variation has vanished, far too much to show easily.

    I point out that a frequent standard today is sampling at 10 minutes but this data is very rarely available and only then for brief timespans.

    There is one instance of 10 minute data where I have GB of it. If you recall the fuss over Steve Mcintyre’s mystery man at ClimateAudit, all has never been revealed (and I am not saying). I had the AWS 10 minute data so I could cross check. (completely legal, Antarctica Treaty)

    A lot of tales untold.

    Eilert, I recall using the Reynolds data to try and find a lunar signal. Nothing significant appeared although there were slight signals. Today I might be able to do better.

  6. P.G. Sharrow says:

    malagaview says:
    October 22, 2011 at 11:06 am
    “Now where have I heard of the Solar-Lunar weather forecasting technique? 🙂 ”

    Our friend Richard Holle at Aerology.com Gives daily predictions years in advance. More accurate then the weather guessers 7 day forecast. I am following the discussions for California weather as it develops in the eastern Pacific. Depressions forming on Richard’s predicted schedual for time and place. 😎 pg

  7. adolfogiurfa says:

    @P.G. Sharrow , @malagaview :
    “Now where have I heard of the Solar-Lunar weather forecasting technique? ”

    Our friend Richard Holle at Aerology.com Gives daily predictions years in advance
    Let´s think it over a little:
    -Also our friend Vukcevic has shown the relation among temperatures and GMF: One of his many pages about this:http://www.vukcevic.talktalk.net/NFC1.htm
    -Nobody will have doubts about the relation Sun-temperatures, but…..the Moon?
    -Temperature it is but a consequence of energy exchange.
    -GMF is about Electricity/Magnetism.
    -Then, if we conceive a circuit, where the Sun gives off energy, like, say, an anode (as EU guys contend) then the earth is the cathode and the Moon is earth´s ground.
    BTW: The french mathematician and philosopher René Guenon wrote about, in his book: “Symbols of the Sacred Science” about two gates, the “Soltiscial Doors”: the Gate of the Gods, the Deva Loka , the Brahma Loka, the ´solar door´and the gate of the Devils, the Asura Loka, the Indra Loka, the ´lunar door´ and…. ..the being will depart from it by one of the two doors depending upon the spiritual degree it has reached.
    -It seems that the Moon sucks in energy, changing from perigee to apogee:
    November 1 is the Celtic feast of Samhain. Samhain, Gaelic for “summer’s end,” was the most important of the ancient Celtic feasts.
    -The Celts honored the opposing balance of intertwining forces of existence: darkness and light, night and day, cold and heat, death and life. The Celtic year was divided into two seasons: the light and the dark, celebrating the light at Beltane on May 1st and the dark at Samhain on November 1st.
    http://www.allsaintsbrookline.org/celtic/samhain.html
    -Last but not least: It seems we are rediscovering a long forgotten objective science. Interesting times, indeed.

  8. malagaview says:

    @ adolfogiurfa – It seems we are rediscovering a long forgotten objective science.

    Science and Semantics
    Over the last few years I have been introduced to the Science of Semantics while researching the mythology surrounding AGW… and in this context I use the word mythology to mean the Settled Science quoted by members of the AGW Cult… who are frequently referred to as The Team or more cynically as The Usual Suspects.

    I do appreciate that words like mythology and cult can be interpreted as prejudicial… just as the word denier is also seen as prejudicial in the AGW debate. However, in the AGW context, the debate has moved from the realm of Science into the realm of beliefs. Specifically, we have moved from the realm of Science where a theory is never settled…. although a theory can be explicitly disproved… and all that can be said about a promising theory is that it has not been disproved yet. More specifically, we have arrived where we have Science by Press Release, Settled Science, Consensus Science and Pressure Group Politics… a place where government funding is channelled by the peer review process… a place where contrary academics are rarely funded… a place where contrary views are rarely published in peer review literature… as they say: follow the money when it comes to the peer review process. So be it.

    Myth Making
    Whenever I am reading a science textbook (or paper) I am aware very aware of the language being used… very aware when the author retreats to arguments of authority… very aware when the author retreats from the real world to hide behind computer models.

    However, the construction of a myth in modern day scientific literature is often far more subtle that these simple examples. It is far more subtle than the sin of omission, which is also referred to by many as cherry picking. There are been some very striking examples in the AGW literature where the authors have been cherry picking data to support erroneous arguments and theories. Cherry Picking also extends to the references quoted to support a particular paper or theory. The references may just be circular logic where a reference is to the author’s prior publications… and the author’s self-interest means he, or she, will not usually reference any papers that would contradict, or cast a shadow across, their own work. As they say: Caveat Emptor – Buyer Beware.

    Find The Lady
    The true art of modern day mythology is based, literally, upon confidence tricks. A simple confidence trick can be as easy as hiding the decline in a graph… but the true masters rely upon the ignorance of their readers… especially their ignorance of other fields of specialisation.

    These confidence tricks are usually quickly glossed over in the literature… perhaps described as an assumption… or more usually referenced as a known fact that requires no further explanation or investigation.

    A classic example is the Greenhouse Effect which most people remember being taught at school. The Greenhouse Effect phrase actually covers two distinct effects: 1) The Greenhouse Roof Effect which is real and 2) The Greenhouse Gas Effect which is not real. Overall, the conflated Greenhouse Effect is real because the roof of the greenhouse prevents convection.

    The AGW myth is based upon the fictitious Greenhouse Gas Effect… however, the Settled Science gains popular credibility because everyone know about the Greenhouse Effect… the argument is superficially plausible to most people… and it is only those curious enough that will actually look for the truth behind this Settled Science. As Anthony Watts has so clearly demonstrated this week: http://wattsupwiththat.com/2011/10/18/replicating-al-gores-climate-101-video-experiment-shows-that-his-high-school-physics-could-never-work-as-advertised/

    Key Words and Phrases
    Therefore, I am a developing a list of key words and phrases than can be used to trigger a myth detector process for fact checking… at the top of this list is Greenhouse Effect and Greenhouse Gas Effect… at the bottom of my list, written in pencil, are the words: Convection and Coriolis Effect.

    Convection
    Convection came up on my myth detector radar this week…. it is one of the know facts that is embedded into the science used to support Plate Tectonics and the Earth’s Dynamo Theory… both these theories initially lack credibility and have the hallmarks of a myth.

    Looking at Wikipedia it is apparent that convection is another of these composite, conflated terms which has many scientific meanings (natural, forced, gravity, buoyancy, granular, thermomagnetic, capillary action, Marangoni effect, Weissenberg effect) but is commonly understood to have one meaning by the majority of people who are not specialists in this area.

    The Wikipedia entry has a wonderful diagram of convection in the Earth’s mantle… it superficially looks convincing… until you read the small print on the diagram: it assumes constant viscosity in the mantle… which seems very un-realistic… especially as there are no mantle samples that have been measured… the small print also says that the diagram is a calculation which I assume means it was generated by a computer model.
    http://en.wikipedia.org/wiki/Convection

    Then I stumbled across an interesting paper by J. Marvin Herndon… and he has a lot to say about convection.

    “The simplest example of thermally induced convection arises when a horizontal layer of fluid is heated from below and an adverse temperature gradient is maintained. The adjective ‘adverse’ is used to qualify the prevailing temperature gradient, since, on account of thermal expansion, the fluid at the bottom becomes lighter than the fluid at the top; and this is a top-heavy arrangement which is potentially unstable. Under these circumstances the fluid will try to redistribute itself to redress this weakness in its arrangement. This is how thermal convection originates: It represents the efforts of the fluid to restore to itself some degree of stability.”

    **********************************

    Recently, Herndon pointed out that the Earth’s fluid core is wrapped in an insulating blanket, a rock shell, the mantle, that is about 2900 km thick, and which has a considerably lower heat capacity, lower thermal conductivity, and higher viscosity than the fluid core. Heat brought to the top of the core cannot be efficiently removed, so maintaining a significant difference in temperature between top and bottom of the core for extended periods of time, a requisite condition for long-term convection, is not possible.

    **********************************
    … because of the over-burden weight, the Earth’s core is about 23% more dense at the bottom than at the top. Thermal expansion at the bottom cannot overcome such a great difference in density, meaning the Earth’s core cannot become top-heavy, and thus cannot engage in convection. The implication is quite clear: Either the geomagnetic field is generated by a process other than the convection-driven dynamo-mechanism, or there exists another fluid region within the deep-interior of Earth which can sustain convection for extended periods of time.

    **********************************
    The basis presented here for the absence of Earth core convection can be generalized to the Earth’s mantle. Since the 1960’s, convection has been widely assumed to occur within the Earth’s mantle and has been incorporated as an absolutely crucial component of plate tectonics theory. The concept of convection in the Earth’s mantle is refuted because thermal expansion cannot overcome the 62% higher density at the mantle’s bottom than at its top.

    Uniqueness of Herndon’s Georeactor: Energy Source and Production Mechanism for Earth’s Magnetic Field
    J. Marvin Herndon – First Revision May 9, 2009

    Click to access Herndon%20Uniqueness.pdf

    So it looks like convection is going to being written in ink in my list of key words…

    Unless anyone knows differently??????

  9. malagaview says:

    PS
    One of the hallmarks of a myth is the invention of fictional terms and entities… they are really fudge factors…. and convection has a classic 🙂

    Convection is the movement of molecules within fluids (i.e. liquids, gases) and rheids. It cannot take place in solids, since neither bulk current flows nor significant diffusion can take place in solids.
    http://en.wikipedia.org/wiki/Convection

    So convection cannot take place in solids… but the mantle is know to be solid… so they invented Rheids… and suddenly Granite can support convection just like a fluid…

    Rheid
    Almost any type of rock can behave as a rheid under appropriate conditions of temperature and pressure. For example, the Earth’s mantle is believed to undergo convection over long time scales. Since the mantle is known to be solid (it supports the propagation of shear waves), it must be behaving as a rheid. Granite has a measured viscosity at standard temperature and pressure of about 4.5×1019 Pa·s so it should be considered a rheid.
    http://en.wikipedia.org/wiki/Rheid

    Kinda puts a new twist to the concepts of Whiskey on the Rocks. Cheers 🙂

  10. malagaview says:

    PPS

    This is a famous lump of Granite in Yosemite National Park called Half Dome….
    Perhaps it should be called Half Baked as it supports convection
    Just remember to wear your over gloves when climbing this hot lump of rock…

    Here are some terrace of houses built from granite in Aberdeen aka The Granite City…
    Guess they chose Granite so that convection could provide free geothermal central heating…