Roger Andrews: Sunshine and Temperature study following up Euan Mearns article

Posted: December 12, 2013 by tallbloke in Analysis, atmosphere, climate, Clouds, data, general circulation, Surfacestation, Tides, weather, wind

Guest post from Roger Andrews, who says: ” This is a review that extends Euan Mearns’ article on sunshine hours, cloud cover and SAT in the UK over mainland Europe and the North Atlantic. It reveals some interesting features that I make no attempt to explain – basically because I can’t – but someone else may have some ideas.” Apologies to Roger A for the delay in getting this article posted.

SUNSHINE, CLOUD COVER AND SURFACE AIR TEMPERATURES IN EUROPE
by Roger Andrews

The recent “UK temperatures since 1933” post discussed the relationships between sunshine hours, which were assumed to be an inverse cloud cover proxy, on surface air temperatures (hereafter SAT) at 23 UK stations. Here I summarize the relationships between sunshine hours, cloud cover and SAT over  Europe using observations from ~30 stations selected from the European Climate Assessment (ECA) data set (acknowledgement as requested to Klein Tank, A.M.G. and Coauthors, 2002. Daily dataset of 20th-century surface air temperature and precipitation series for the European Climate Assessment. Int. J. of Climatol., 22, 1441-1453.) Station locations are shown in Figure 1:

image1

The results are preliminary but indicate that the relationships between sunshine hours, cloud cover and  SAT become more complex when areas larger than the UK are considered, as illustrated in the following figures:

Figure 2. Mean annual SAT is well correlated with mean annual sunshine hours in Europe after 1985 but not before (the data were normalized by dividing by two standard deviations):
image2

Figure 3. Mean annual SAT is poorly correlated overall with mean annual cloud cover, with mean SAT increasing since 1950 but mean cloud cover showing no significant change:
image3

Figure 4. Mean annual sunshine hours and mean annual cloud cover show short term peak-trough matches with ~2 year periodicity but are also poorly correlated overall:
image4
Figure 5. Monthly sunshine hours means are generally well correlated with monthly SAT means at individual stations, but expressed to the nearest whole month sunshine hours lead SAT by two months at stations adjacent to the North Atlantic Ocean and by zero months in Southeastern Europe:
image5

Figure 6. Monthly cloud cover means are at best moderately correlated with monthly SAT means at stations in mainland Europe with no leads/lags except in the Central Baltic. At stations adjacent to the North Atlantic cloud cover and SAT are effectively uncorrelated (“U”):
image6
Figure 7. Monthly sunshine hour means are generally well correlated with monthly cloud cover means at stations in Central Europe but not well correlated at stations adjacent to the North Atlantic (there are no significant leads or lags):

image7

Figure 8. The correlations between SAT and sunshine hours /cloud cover decrease with increasing elevation in the Alps, with the SAT/cloud cover correlation turning positive (i.e. more clouds, higher temperatures) above ~2,500m. However, the match with increasing duration of snow cover is equally good. (The ten stations shown are 1=Wien, 2=Basel, 3=Geneva, 4=Hohenpeissenberg, 5=Feldberg, 6=Davos, 7=Wendelstein, 8=Saentis, 9=Zugspitze and 10=Sonnblick):

image8

Comments as to what these results may be telling us are solicited.

Some brief notes on data and data treatment:

The ECA data set is available at http://eca.knmi.nl/indicesextremes/customquerytimeseriesplots.php

It contains multiple variables for over 8,000 stations between Greenland and Kyrgyzstan and Svalbard and the Canary Islands, although the majority are in Germany. It stores monthly means for different variables in different files, so obtaining all the data for one station requires a number of separate downloads.

I downloaded monthly means of maximum temperature, minimum temperature, sunshine hours, cloud cover and snow depth (for the Alpine stations only). I calculated mean temperature by averaging Tmax and Tmin and also calculated Tmax-Tmin. This yielded six variables (sun, clouds, Tmax, Tmin, Tavg and Tmax-Tmin) that could be compared 15 different ways, and I limited the comparisons to sunshine hours and clouds versus Tavg partly for simplicity, partly because the (alleged) impacts of increasing CO2 are quantified in relation to average rather than maximum or minimum temperatures and partly because while Tmax tends to be most closely correlated with sunshine hours and clouds the correlation isn’t much stronger than the correlation with Tavg. The table below summarizes the means and standard deviations of correlation coefficients measured at 21 stations, with no allowance for leads or lags:

CORRELATION COEFFICIENTS, 21 EUROPEAN STATIONS

Variable

Versus Variable

Mean R Value

Standard Deviation

Sunshine Hours

Average Temperature

0.73

0.15

Maximum Temperature

0.76

0.15

Minimum Temperature

0.68

0.14

Max-Min Temperature

0.66

0.39

Cloud Cover

Average Temperature

-0.29

0.41

Maximum Temperature

-0.34

0.41

Minimum Temperature

-0.24

0.40

Max-Min Temperature

-0.59

0.23

Sunshine Hours

Cloud Cover

-0.62

0.26

The analyses are based on unadjusted monthly means except for the the annual means shown in Figures 2, 3 and 4, which are arithmetic averages of the monthly means. I discarded obviously suspect data, such as Liepaja  temperatures after 1999 and Bjornoya cloud cover. I downloaded data back to 1901 but there were too few stations to estimate reliable annual means before 1950.

Comments
  1. Doug Proctor says:

    The work I did on Central UK Max temps relating to sunshine hours on Tallblokes’ was, in the opinion of others, confusing (I felt I had to do the work from first principles, as I am not a climate scientist with a lot of background).

    In essence, what I found was that there was a cycle of temperature and sunshine hours that had a time element. The time element was a reflection of (whatever causes it) the AMO-PDO cycle. Removal of the AMO-PDO signal showed a clear delta sunshine hours to delta increase in max temperature relationship.

    The reason that Max temperature works better than av temperatures is that additional sunshine spikes the temperatures, while average temperatures smear the effect. But the effect of more ground insolation is still there, but gets lost within the natural variability and instrumental “noise”.

    Doug Proctor: Climate Change is caused by Clouds and Sunshine

    The following is the Abstract:

    Previously sourced and plotted data for averaged annual maximum temperature and hours of bright sunshine covering the period 1932 to 2010 for the Central United Kingdom were analyzed. Changes in the two relative to a stable period (1962 – 1973) amounted to increases of 0.98C and 108 hours in 2010.

    Three factors were found to be associated with all temperature changes:

    1) The duration of bright sunshine, such that C = 9.27E-3C X Sunshine hour – 0.10C. This factor was constant with time, but the changes in bright sunshine hours followed (with time) a quasi-sinusoidal pattern with indeterminate amplitude, but a peak-to-peak cycle of 62 years.

    2) A quasi-sinusoidal (with time) Pacific Decadal Oscillation-Atlantic Multidecadal Occillation-like variation, with a cycle length of 56 years and amplitude of 0.31C.

    3) A linear (with time), consistent increase of temperature, such that C = 9.53E-4 (Yr-1873) – 0.1425 C.

    The majority of temperature change was due to the sunshine duration factor. The PDO-AMO-like varying factor contributed the second most significant portion of the temperature change record, sometimes adding and sometimes subtracting from the temperature changes associated with increased/decreased bright sunshine. The third factor was tied to the PDO-AMO-like factor as a long-term warming, but added only a minor amount, 0.095C/century.

    The datum period 1962 – 1973 recorded a stable period of 1315.9 hours, i.e. a daytime cloudiness of 70.0%. From 1932 to 1948, and from 1980 to 2010, the Central United Kingdom experienced increased bright sunshine of about 42 and 108 hours, respectively. This is a bright sunshine increase of 3.2% gross and 0.96% net more sunshine for the earlier period, and 8.2% gross, and 2.5% net additional sunshine for the most recent period. Stated in the reverse, in the 1932-1948 periods when temperature rose 0.32C, there was 0.96% less cloudiness; in the 1980 – 2010 period, when the average maximum temperature rose 0.98C, 2.5% less cloudiness.

    The PDO-AMO –like temperature changes did not match perfectly either the timing or amount of temperature change associated with heat release and storage for either the PDO or the AMO events as individual events. The changes appear more of a non-equal combination of both, though the combination was not determined within this study.

    It is concluded that changes in the Central United Kingdom Maximum temperature history of the past 70 years is fundamentally a response to changes in the amount of sunshine (i.e., cloudiness) in association with rises and falls in temperature resulting from natural heat storage and release of the from the planet’s two largest oceans. The remaining, small portion of temperature rises seen in the Central UK may as well be attributed to land-use changes or inappropriate adjustments in the temperature records as it could to CO2-related changes in heat retention. Regardless of cause, this minor temperature rise, at 0.1C/century is of no consequence to the local biosphere.”

    As I stated, some found it too confusing to even discuss (Greg Goodman). So I dunno if it is all a crock. Maybe it is, maybe it isn’t (I need a consensus here before I make a decision with any certainty).

  2. tallbloke says:

    Thanks for the summary Doug. I’m packing for a trip and won’t be able to follow this, but I think the whole sunshine hours correlation to temperature is of high importance, so I hope people will put some thought into this article and also visit your older article.

  3. tchannon says:

    Anyone wanting data from the cited server in more portable CSV let me know the station index number (name might do) and the two letter data identifier. Have data here and I’ve thrown together extraction into a more suitable format. Two CSV, one for annual data, the other monthly. (reminder, as co-moderator I can make data available for download in a zip)

  4. tchannon says:

    I’ve extracted Valentia as a first look.

    I make that 48 days, which is for a maritime climate.

    Snag, the apparently strong correlation is primarily caused by the datasets having a strong annual component.

    Grabs oven glove, pops door

    Valentina flatbread
    In phase at 48 days, correlation about 0.8, a running correlation is pretty much flat


    And this ‘ere is Valentia doughnut
    Adjusted to quadrature, 90 degree phase shift, correlation zero, XY plot scribes a circle instead of a straight line.

    Normalisation was not required done like this.

    Hint, uses this, .

    Fractional dataset delay (subsample resolution) in a spreadsheet

  5. Roger Andrews says:

    I think the important questions here are:

    1. SAT in Europe has increased by about 1C since 1950 with no significant change in cloud cover. Does this mean that clouds didn’t have any impact on SAT or that cloud COVER didn’t? (Maybe the type of cloud is more important than the amount. The fact that cloud cover and SAT are correlated inland but not adjacent to the North Atlantic suggests that it probably is.)

    2. Cloud cover and sunshine hours are also well correlated inland but generally only weakly correlated adjacent to the North Atlantic, showing that sunshine hours are not always a good proxy for cloud cover. But sunshine hours correlate better with SAT than cloud cover anyway, so why not just use sunshine hours and forget about cloud cover?

    3. What are the relationships between elevation, sunshine hours, cloud cover and temperatures in the Alps telling us?

    4. Why do sunshine hours lead SAT, why do lead times get longer as we get closer to the Atlantic and how does this dovetail with the fact that SAT leads SST by about a month? (see graph below).

    Er, Tim: Who or what is Valentina? [mod: yes, now fixed, sorry –Tim]

  6. Roger Andrews says:

    Ah. I think maybe you mean Valentia. If so, could you please give me some insight as to what the graphs represent?

  7. tchannon says:

    Dang, sussed. I never make mitsakes.

    Sunshine hours vs, mean temperature, monthly. One with running correlation on infinitely variable time shift the other an XY plot. (correlation figure for the whole thing is correlation xxxx delayed

  8. Paul Vaughan says:

    Aggregation criteria are the spice of statistical exploration.

    Break the exploration down by time of year — or more specifically by temperature. I’ll give a local example: In summer lower temperatures and cloud cover go together (here) whereas in winter higher temperatures and cloud cover go together (but winter is also less settled as explained by John S. at Climate Etc. on the related thread awhile back). Low altitude/latitude have a predominance of months in a different state that tips the aggregate — basically you’re getting less of a blend of qualitatively differing states. Check by station to see if certain groupings of months show clearer patterns and review John S.’s notes. Mixing round-the-year is probably drowning out some clear signal. Visual diagnostics on sorted residuals will point to problems with model assumptions (of uniformity year-round) that can be pruned out to tune the aggregates. This sort of analysis isn’t going to lead to any breakthroughs, but if someone has extra time on their hands its a worthwhile exercise. I recommend keeping it simple. Just set a spreadsheet up to toggle months on/off and try narrowing & widening windows centered on January-February & July-August. For example, here’s something to think about:

    If you check the center of the opposite part of the year for CET, you’ll find the relationship upside down for the early 20th century. That points to a moved baseline and associated circulatory shifts — easy to misinterpret as something temporal rather than spatiotemporal without the right background knowledge.

    Best Regards

  9. wayne says:

    Roger Andrews, on your point 1. you might also add the altitude of the cloud tops (the thickness of the cloud layers) to cloud types as differences. I tend to think that is an overlooked parameter that varies greatly without being actually a different cloud type, which due to the lapse, dictates the general temperature that radiates further upward. Knowing the cloud top temperatures would be even better but I’ve never seen such a satellite dataset.

  10. tchannon says:

    RA, another day, I wasn’t being coherent last thing.
    The left hand plot shows the variation in correlation along the pair of data with a corr. length of about 18 months. Mean is about 0.8 the same as the corr. for corr. on all the data pair done together. This shows good consistency for the whole length of the data.

    Right hand is either as close to a straight line as it will get or when put into quadrature is tracing out a circle from the annual cycle.

    I might have a think later and have a go at removing annual. Maybe sunshine data is best done by factoring in the variation is daylight length for the station.

  11. Roger Andrews says:

    Tim: Good stuff. Could you calculate sunshine/temp leads/lags for some of the other stations too?

    To avoid confusion, I think what you are calling the “annual signal” is what I would call the “seasonal cycle”, correct?

  12. tchannon says:

    Maybe later.

    One year and maybe harmonics thereof.

  13. Peter Jackson says:

    In view of the results from Frank Lansner’s Ruti project covering this area, it may well be that this evaluation mixes “apples, oranges and pears” – in that his results show distinct differences in SAT according to site – coastal, inland and elevated – due to the influence of ocean heat on the former and elevation on the latter and the sources of weather systems for all. Breaking the analysis down by these criteria and and the areas as per Ruti may give more interesting results. These differences are seen in the comparison of the CET and Central European temperature records.

  14. tchannon says:

    Continental climate Zagreb, about 12 days lead

  15. Roger Andrews says:

    Tim: Keep it up. A couple of dozen more numbers and we’ll be able to publish 🙂

    Peter Jackson:

    I’m looking at the seasonal relationships between sunshine hours, clouds and SAT as a function of distance from the coast and elevation, and the graphs I presented above confirm that these relationships are influenced by distance from the coast and elevation, so I guess I don’t see where the apples, oranges and pears come in.

  16. Peter Jackson says:

    Roger Andrews

    What I was looking for in your study was evidence that the recent study showing that reduced cloud cover during the recent warming period at the end of the last century was associated with a 40 W/m2 increase in surface solar intensity was directly responsible for the major part of the warming shown in the SAT records but your lack of a good correlation prior to 1950 goes against this as the sole factor. Lamb found that wind pattern was a key factor for the UK temperature and this pattern differs between the UK and Central Europe – see the animation at Lubos Motl’s site. On this basis it is not just a relationship between cloud cover, sunshine hours and SAT that matters but the effects of the incoming weather that differs whether is comes over the ocean or land and in the former the direction from which it comes.

    I guess if you are using cloud cover as a proxy for weather systems then what you present is valid but as I probably did not make clear in my comment, so what is new in that, Lansner has done it already for SAT in his Ruti project albeit without linking it to sunshine hours and cloud cover but to prevailing weather patterns. Looking at the current weather maps it is clear that much of Central Europe is in a different weather system to the UK and countries along the Atlantic coast so when you are looking for correlations I do not see the point in mixing these areas up. It is a bee I have in my bonnet about SAT generally which I would expect to correlate with climate change by the distinct climate zones according to the Köppen climate classification system and not by regions or any other means – that is the climate creates these zones in the first place and they have distinctive features, especially factors such as soil types and water capacity that directly affect SAT as much as sunshine hours and cloud cover. But perhaps I am reading too much into your analysis which I find interesting in any case.

  17. Roger Andrews says:

    Peter Jackson:

    I’m encouraged that you find my “analysis” interesting, but I didn’t actually do any analysis; all I did was present data with the idea that others might be able to explain what was going on.

    This being the case, could you elaborate on your statements that the observed effects are caused by areas with different weather systems, soil type, water capacity etc? I’m not saying you’re wrong, but I can find no support for these statements in the data I presented.

  18. tchannon says:

    These are just demonstrations. Data is output by a prototype program, incomplete.

    These are deliberately over a huge lead/lag so it reveals the annual climatic cycle.

    Note the input data is monthly so you probably seeing what you have never seen before and quite likely didn’t know this could be done. Technically this underwhelming, everyday stuff for engineers..


    .

    When I get the basics right there is then a very messy stage, figuring out how to automatic data import where there are severe timescale mismatches and missing data. Somehow things have to be intelligently aligned.

    If i can do that, turn key and go eat, consuming disk space.

    What then, maybe a mountain of results. Writing code to automate plotting is a pain, all plotting tools are horrid to use. I don’t see that a few numbers are useful unless sanity check traces are eyeballed.

    So, what should I compute and output? (I have to do it)

    Incidentally I don’t buy the sunshine/cloud linkage. I’ve some things to say and show eventually, nothing to do with the datasets here. There is so much wrong I can’t think of a way of explaining, too alien.

    With the data here it is possible what is appearing is the surface (water or land) thermal characteristic. Let alone that the earth is dynamic, night was well as day.

  19. Roger Andrews says:

    Tim:

    Okay, I surrender. As a dumb non-engineer I admit to being overwhelmed by all this underwhelming stuff you smart engineers do every day 😉

    However, I can in fact duplicate your lead time estimates to within a few days from my own clunky monthly plots. Below is my plot of Valentia. From it I eyeball a lead of slightly more than 45 days compared to your 48. From my Zagreb plot (not shown) I eyeball slightly less than 15 days compared to your 11. Basically what you are doing is confirming my results while improving the precision of the estimates.

    What you probably could do that I can’t is sausage machine a whole bunch of comparisons at the touch of a button, but with six variables (sun, clouds, tmax, tmin, tavg, tmax-tmin, tavg) there are 15 different ways of comparing them at each stations and you would soon bury yourself in data if you ran all possible combinations. To simplify the exercise my suggestion would be to do three: sunshine vs. tavg, clouds vs. tavg and sunshine vs. clouds. The end product would be more accurate versions of Figures 5, 6 and 7 with the leads given in days rather than to the nearest month.

    It’s good to know that I’m not the only person who doesn’t buy the sunshine/cloud linkage. In fact all you have to do not to buy it – and not to buy the temperature/cloud linkage either – is spend a few minutes looking at my results. I was in fact hoping that simply presenting these results would have been enough to show that cloud cover isn’t much of a control on anything in Europe except maybe picnics, but clearly they weren’t, and if I’d known this beforehand the post would have had a much more provocative and eye-catching title, something like:

    CLOUD COVER/TEMPERATURE RELATIONSHIP DEBUNKED BY EUROPEAN DATA

    Over to you.

  20. tchannon says:

    “Basically what you are doing is confirming my results while improving the precision of the estimates.”

    Yes.

    Walk the walk, see what falls out.

  21. Doug Proctor says:

    I think I see a cross-purpose argument here: one in which the small area is compared to the large area, and comparisons being made that are unfair. Above, I see total sunshine hour and mean temperatures comparisons but what I, for for, did was focus in on “bright sunshine” and max temperature variations. True, the point that these are the indicators for a good picnic, one wag suggested. I agree, but as proof of the importance of bright sunshine (short-term could cover) to the temperature rises we appreciate while lolling about with pate and a chilled bottle of white.

    It is impossible not to see “bright sunshine” hours as indicative of low cloud cover – except at night, of course. It is impossible except for those who are housebound not to recognize that more sunshine leads to warmer ground-experienced temperatures. What is NOT impossible is to miss a connection to temperature when it is mixed in with other varying variables.

    The frog that hangs out in the boiling water doesn’t recognize the temperature rising if the temperature rises slowly at some rate that it recognizes as part of its natural experience. A 2% increase or decrease in cloud cover over decades would be a similar type of change, I would expect (those with statistical analytical skills should be able to run a program to show this if I can imagine such a program: it ain’t rocket science, no doubt, and some of you ARE rocket scientists).

    Regionality is, I believe, the key to the mathematical reality of “global” warming. Break the world into sensible blocks of land and sea, and look for the principal areas of temperature (and OCHC) and you will find a non-global “source” of the historical temperature rise. The location doesn’t have to be the same – a “proof” of global warming? – and in fact bolsters the case for regionality (or regionalism) dominating the records.

    In the same spirit as the cloud cover issue, the temperature rise issue, I propose the CO2 Keeler Curve should be re-investigated. The uni MODAL (to correct a previous brain-freeze type of mine) annual variation looks not like a global net sum signal but a regional, dominating signal. We are told, as is a base assumption of CAGW, that the Keeler Curve rise represents 100% human contributions.

    Somewhere in the last couple of weeks there was a CO2 paper or blog that showed CO2 rise in the pre-Keeler Curve days. Back to the 1880s. Fast forward to the 1940s: when the curve flattened, and note that that was when the temperatures DROPPED. Interesting, isn’t it? Almost as if somewhere in the world, as temperatures dropped, CO2 emissions dropped. Isn’t that counter-intuitive? You would think in cooler times CO2 emissions would INCREASE as biologic activity slowed the removed of A-CO2. But it didn’t.

    We need – IMnon-HO – to step back from large chunks of data and focus instead on the idea that small areas dominate the whole. It is a simple concept: keep 90% of the data in the same historic variable range, but spike the remaining 10%. You get the signal of the remaining 10%.

    Really, I’m shocked that we haven’t jumped on this idea already, because we have a world class example: Mr. Mann’s hockey stick graph, the result of a self-serving programme and ONE aberrant Yamal tree.

    So back to cloud cover, temperature hot spots and CO2 blurches. Is possible it is the anomalies that create the trend? Are we not to find the anomalies and explain them, rather than clutch, massage and spit out the entire mess and then say, behold! it is the great that changes?

    Who would have known that Al Gore would have a positive impact on the intellectual life of the disaffected citizen (for right or wrong, it hardly matters, as we are thinking for ourselves; it is the community who points out our errors or rightness of our ways. Which is not the same as the oppression of Consensus, by the way.)

  22. tchannon says:

    Inlined your plot.

    Also feasible to interpolate or use splines.

    Basic objective is getting an analogue signal of phase where some kind of spatial rule will appear.

  23. tchannon says:

    Grin. I was expecting the following to be difficult although the problem is trivial for this nearly 30 year old algorithm.

    Aside: so far Spearman gives the same answer, implying this is a linear domain.

  24. tchannon says:

    Given say 123 sg ss or 123 ss sg

    Data for station 123 is pulled in, date aligned, topped, tailed as necessary.

    Finds the best offset lead or lag.

    As expected there is a tricky issue to sort out, handling ends and bad data. right now it gives different answers for the above, hopefully caused by one end hanging differently off the end as the time offset varies.

    I have to figure out how to step around this and bad datapoints. Some degree of difference will remain. You might need to think about the reason.

    A question soon is what to output.

    Blog? I’ll probably post some new stuff soon. Tends to take time and energy.

  25. Roger Andrews says:

    Still here 🙂

  26. tchannon says:

    Summary:
    Three Key Conclusions
    1. Unimaginable anomalies infest real datasets
    Yogi Bera:
    If something has a 50% chance of happening, then 9 times out of 10 it will.
    Dasu and Johnson (2003, p. 186):
    Take NOTHING for granted. The data are never what they are supposed to be, even after they are “cleaned up.” The schemas, layout, content, and nature of content are never completely known or documented and continue to change dynamically.
    2. Different analysis methods exhibit different sensitivities to different data anomalies
    3. Comparison of what should be “equivalent” analyses across different scenarios can be extremely useful in uncovering anomalies

    Click to access pearson.pdf

    You are trying to fill up this disk. Swallowed 100M so far and 8900 files, starting to work. Made it to 159 results before barfing. No mechanism yet for rejecting ridiculous data, eg. 2 months of data.

  27. Roger Andrews says:

    I think the incisive observation of Yogi Berra expands the Heisenberg Principle into an area which up to this point has gone unresearched; specifically, not only can you change the behavior of a physical system by observing it, you can also change its behavior to the opposite of what you think is going to happen by announcing what you think is going to happen beforehand. Examples abound in televised sports events:

    “So-and-so has now played 45 consecutive holes without a bogey and he isn’t going to miss a putt this short.”

    Whoops.

    “This guy has made 39 consecutive free throws and this one is going to be a gimme”

    Clang.

    “No goals so far this season and it doesn’t look like tonight will be his night either.”

    Goal!!!

    I don’t know whether this has any bearing on what you’re doing but thought you should know about it 😉

  28. tchannon says:

    Something to get you going, log up until it aborts, no header, will import as comma delimited here

    https://tallbloke.files.wordpress.com/2013/12/log1.zip (10k)

    Just typed this in, not tested and subject to change
    local header = {‘stationID’, ‘lead_days’, ‘lead_samples’, ‘correlation’,’status’,
    ‘y_from’, ‘m_from’, ‘y_to’, ‘m_to’, ‘place’, ‘country’,’latitude’,’longitude’,’altitude’,”}

    A lot of station “files” have already been rejected, empty for example.
    Reason for the barf?

    (this is extracted data used as an intermediate or for import)
    “DATA IS FOR INDEX Sunshine duration (SS) with unit 0.01 Sunshine duration (h)”,
    2007,1,”nk”,
    2007,2,”nk”,
    2007,3,”nk”,
    2007,4,”nk”,
    2007,5,”nk”,
    2007,6,”nk”,
    2007,7,”nk”,
    2007,8,”nk”,
    2007,9,”nk”,
    2007,10,”nk”,
    2007,11,”nk”,
    2007,12,”nk”,
    end-of-file

    No big deal, just another issue to sort out.

  29. tchannon says:

    Trying a run with data length (I hope) minimum 3 years.

    Heard the shipping forecast, long time since I heard mention of possibly storm force 12, hurricane force (out at sea). Checking the US Navy forecast, a blow but not that extreme.

    Still going, log is already larger even with fewer reports.

  30. Roger Andrews says:

    Tim:

    Amazing! Thank you. No problem downloading the data to a spreadsheet.

    I have just a few questions.

    I’m looking at sunshine hours versus tavg?

    There are more stations to come?

    Do you have any plotting software? 🙂

  31. tchannon says:

    Yes I know, I forgot to encode what pair and sequence. Might just do the file name.

    SS ==> T where T lags SS if positive number

    Ran through to stn 5111… sure enough more crassness. An entire long SS file all no value.
    Double check, not me, entire Norwegian data is bollox.
    Now I suppose slow things more, count every darn data to check there are enough actual data points!

    Yes there is a lot more data.

  32. tchannon says:

    Try that, kludge, might work, was deleting unwanted data so a whole record of unwanted deleted back past nothing, error. Try, gets silly short, abort out.
    Might not finish before I turn in.

    Plot? Of course, rarely easy to do.
    What do you reckon is needed?

    Oh and does lattitude/long need changing to a more useful data format? i’m just copying the original.

    Got to 1681, numbers run to 8000+, maybe we get 1000 points, enough for a contour plot.
    2764

    Reckon a small whisky is in order. Occasionally and less pain when I try to sleep.
    4252 Hic.

    Code is a mess, hacks always are, if well, I am somewhat experienced. Key parts are the filter designer, neat, all of 13 lines where 3 are the keyword end, looks nothing, best computer science minimal. (basically nested loops with the parameters the loop index)
    4570, slower, either more data or garbage collection is kicking in, nope, seems solid on using 4.6M of memory. I like small, is fast.

    The other gem is the optimiser. If I’d have published that in the 1980s I would be well known in the field. Been used deadly seriously affecting things around the world. This too is tiny, and stupid. Key is lots of random numbers which break up patterns. Silly really works far better than most of the huge fancy methods, small whizzy.

    Oh, it’s finished. 8032
    Yippee but is it valid?

    178k log file, open, copy, paste. Got header row. Select, sort descending on column E
    About 900 stations with some kind of data.
    Disconnect good.
    Sort on B, delay, plot.
    Not bad, interesting curve.

    Yay, looks promising. Bundle and upload.

    https://tallbloke.files.wordpress.com/2013/12/log.zip (64k)

    ‘spose I’d better find some blog posts tomorrow.

  33. Roger Andrews says:

    What do I reckon is needed next? I would say a contour map of sunshine/temp lead times. You may need to remove the high-altitude stations in the Alps and some other outliers before you do it, however. Then we can look at it and decide what to do next (if you’re not already burned out, that is).

    Are your results valid? If they show lead times consistently decreasing with distance from the ocean, you betcha. It’s hard to manufacture something like that from bad data.

    Lat/long would probably be more useful in decimal numeric form, but no big deal.

    The scotch is on me.

  34. tchannon says:

    https://tallbloke.files.wordpress.com/2013/12/log2.zip (with decimal lat/long)

    Crude plot

    [UPDATED DATA AND PLOT, see following comment]

  35. tchannon says:

    Spotted a mistake, old trick of deg + (sec/60+min)/60
    Helps of seconds and minutes are the right way around. Never mind, plot is near enough.
    I’ll upload a correct file later and mark it as such. [Previous post updated data.]
    Stuff here is being done live in public.

  36. Roger Andrews says:

    No problem Tim

    I color-coded all the numbers I could read. Everything seems to fit except Spain.

    I don’t hear any applause yet.

  37. tchannon says:

    I like the plot. The real meaning of climate.

    Roughly speaking, maritime vs. continental climate.

  38. tchannon says:

    Contour plot doesn’t.
    In try this (see paper mentioned earlier), Spanish problem includes a bad figure. There are also duplicate stations, which must be duplicated in the dataset, fortunately the result is a very close match.

    A spot plot as you have done might be a good solution.

  39. Roger Andrews says:

    I’m going to sleep on this. Back tomorrow.

  40. Peter jackson says:

    Roger Andrews
    Apologies for linking you directly with this work – it comes from following too many blogs and hence being forced to skim for salient points.
    You ask: “This being the case, could you elaborate on your statements that the observed effects are caused by areas with different weather systems, soil type, water capacity etc? I’m not saying you’re wrong, but I can find no support for these statements in the data I presented.”
    My objection is to this and similar studies is the use of SAT, that is an intensive measure, to evaluate extensive measures – in this case radiation received at the surface (Solar SW and DLR from clouds) and precipitation, and especially over such a diverse geographic area where there are major differences in local climate, weather and soil types:
    http://www.metoffice.gov.uk/climate/uk/averages/regmapavge.html#nscotland
    http://www.britannica.com/EBchecked/topic/195686/Europe/34549/Climatic-regions
    https://maps.google.com/maps/ms?msa=0&msid=202977755949863934429.0004d971cf8c32fb044ff

    Click to access hessd-4-439-2007.pdf

    The Köppen climate maps have been linked directly with temperature and climate change:

    Click to access Chen_and_Chen_2013_envdev.pdf

    Click to access EMS2012-137.pdf

    If the sites where temperature is measured and the areas they represent had a constant heat capacity then the use of temperature in this way has validity with some limitations – this is the basis of the statement by Roger Pielke Sr that ocean surface MAST values be used to measure global energy flux and not land surface values – there is an interesting paper on this by Willis today in WUWT : http://wattsupwiththat.com/2013/12/18/the-fatal-lure-of-assumed-linearity/.
    The reason is that whilst the WMO might define the standard Met station in terms of equipment and site characteristics they do not do so in terms of the heat capacity of the soil type upon which they stand and in extrapolating the temperature from such sites to surrounding grids to the diversity of soil types and microclimates these regions have, never mind averaging them on a on a global scale. Soil type and carbon content are important because they define the heat capacity of the soil and its potential to fluctuate. A well drained sandy soil will have a relatively constant heat capacity that will be little influenced by precipitation whereas loam soils will have a higher heat capacity from their ability to retain water longer and this will fluctuate more with rainfall in both the short and longer
    This has two major effects – the first is that soil with a high water capacity will have significant changes in heat capacity with precipitation, hence the weather front and types of clouds. Secondly it will affect the radiation upwards from the surface such that depending on these soil factors there will be wide variation in minimum and maximum temperatures irrespective of the surface radiation remaining constant. Thus the temperature changes over time more reflect the response of the soils to the hydraulic cycle than surface radiation and the use of anomalies does not solve this as they must assume a constant heat capacity.
    Clive Best has shown a significant effect of differing soil moisture levels on MAST values and diurnal temperature range. In addition there is a significant effect of elevation such that homogenization between stations at differing elevations will suffer from both of these factors. This can be seen in the data from the Reynolds Range:

    Click to access climate.pdf

    This study is interesting because, unlike most Met sites, these have Class A Pan Evaporation units and measured wind speed, thus meeting the need for a measurement area with constant heat capacity and giving a direct measure of solar input and DLR from evaporative losses.
    Another relevant study was done at Armagh, one of the longest running temperature series on a single site that has shown a 0.6 oC increase in temperature over the 20th century with the cooling period in the 1940’s to 70’s.

    UHI study of the UK Armagh Observatory


    But a study by Coughlin and Butler of MAST values at three UHI evaluation sites meeting WMO standards only 1-2 km from the main site found that there was mean difference between the three sites and the official Observatory site of 0.11 oC for tmax and 0.41 oC for tmin. But also, the differences between the three “standard” sites had a range of 0.76 oC for tmax and 0.48 oC for Tmin.
    How then can the SAT values on sites as far apart as those in the Euan Mearns’article (or its extension to Europe) on widely different soil types, hence different heat capacities, and subject in wide difference in precipitation, wind speed from different weather fronts be considered to be part of the same climate system and averaged. They cannot, as the Ruti project has shown, with those near the coast reflecting ocean temperatures changes in AMO/NAO whilst those nearer to Central Europe are subject to the weather patterns of E Europe. So given that the temperatures in this work do not totally reflect the received surface radiation it follows that the rest of the analysis – examination, study, investigation, scrutiny or breakdown – take your pick! – is of little value. As I said in my first comment it is comparing apples to oranges and pears.
    There are several papers dealing with this apart from those of Pielke Sr:

    Click to access 319.pdf

    http://www.newton.dep.anl.gov/askasci/wea00/wea00105.htm
    http://biomet.ucdavis.edu/biomet/SoilHeatFlow/SoilHF.htm

  41. Roger Andrews says:

    Peter Jackson: You say: “My objection is to this and similar studies is the use of SAT, that is an intensive measure, to evaluate extensive measures – in this case radiation received at the surface (Solar SW and DLR from clouds) and precipitation, and especially over such a diverse geographic area where there are major differences in local climate, weather and soil types”

    We’re not even looking at radiation. We’re simply comparing seasonal variations in sunshine hours, cloud cover and temperature at approximately 900 European stations to see what, if anything, drops out. Results to date allow us to conclude that distance from the ocean and elevation (snow cover?) are the dominant controls on how the seasonal variations are interrelated, and they’re also consistent enough (see map above) to confirm that changes in local climate, weather and soil type don’t have much impact.

    Thanks for the links to all the Köppel etc. data. In this context you might be interested in the 64 climate zones I defined twelve years ago:

  42. tchannon says:

    Sigh, figured painfully how to do a particular kind of 3D plot, about right but

    Figure out what is needed from what passes as docs, rotate a b c
    I need c and “not currently implemented”

    And that is a demonstration of why I write my own tools and do it using tools I built from source here.

    A shame because it looked promising, a wall of coastline, all colour coded.

    Plan B? Try colour only. Got the encoding so that is a start.

  43. Roger Andrews says:

    Tim:

    “Roughly speaking, maritime vs. continental climate.”

    I put together monthly SST series for the Far North Atlantic, “UK Atlantic”, “Spain Atlantic”, “Canaries Atlantic”, the North Sea and the Mediterranean to see if they showed any lags/leads. The seasonal signals were all “in phase” except for the Canaries, where it lagged the other areas by maybe 15 days.

    I then compared the SSTs against eleven coastal stations for which I had SAT data and found that SST lagged SAT by ~20 days at all of them except Bordeaux (35 days), Vaernes (40 days) and Izana (40 days).

    Thoughts?

  44. tchannon says:

    Little work seems to have been done on phase yet this is crucial to cause and effect.

    As a subject, oh boy is this tricky. Simple whole signal delay is unlikely.

    Not long ago here the bomb spike stuff? I think few really get it, there is a hidden dataset. All we get is a characteristic for part of the whole. (rest has been removed)

    This characteristic is simple as far as phase is concerned but dependent on time, varies with frequency. In addition there will be another characteristic for fast stuff, not investigated, and another for the removed stuff.

    I think it may be necessary to split data into frequency bands.

    Lets carry on and see what falls out.

  45. tchannon says:

    Or PDF https://tallbloke.files.wordpress.com/2013/12/scatter.pdf

    Do for starters.

    Now ooz got a land outline to the right geometry and copyright suitable?

  46. Roger Andrews says:

    Best I can do at short notice

    Map (equirectangular projection, I had to expand your plot 38% N-S to fit it) from

    Do some of your stations still have minutes & seconds flipped?

  47. tchannon says:

    min/sec flipped? No, I just swapped two variable names,done. Effect is minor.

  48. Roger Andrews says:

    Correlation with wind speed?


  49. tchannon says:

    Cracked it, now I have to do it. Consumed 500M of disk.
    Need to figure out what export format is most suitable for plotting, have to write code to do this.

  50. tchannon says:

    This will come as a shock.

    You know how to drive a PDF reader? Vector graphics.

    https://tallbloke.files.wordpress.com/2013/12/scatter1.pdf (156k)

    Don’t assume this is easy, I’m exhausted.

  51. Roger Andrews says:

    Vector graphics? That’s what I’ve been saying all along. 😉

    Great work, Tim

    There are some stations that seem to be shifted east of their true position. If you like I can go through and check them for possible lat/long errors.

    Get some rest now.

  52. tchannon says:

    Information.

    Geographic information came from
    http://www.ngdc.noaa.gov/mgg/shorelines/

    I was able to export in a simple ascii format and write a program to translate that into a form within the very limited capability of the plot software at the same time as staying sanely sized (the data that is), in this case to a sequence of move line line etc.

    If anyone needs help with using the above contact me.

  53. Roger Andrews says:

    More observational data to add to the mix.

    Mean SST in the Atlantic, North Sea and the Med is roughly 5C higher than mean SAT at coastal stations (difficult to calculate an exact number).

    SAT at the coastal stations leads SST by about 20 days.

    SAT minus SST is at a minimum in June/July, when sunshine hours are at a maximum:

    If anyone is still around on this blog, what you are seeing here is the cutting edge of science in action.

    Question is, what to dice next? Tim, if it’s not too much trouble you might try plotting maps of R values for a) cloud cover vs. SAT and b) cloud cover vs. sunshine hours. Leads/lags aren’t going to tell us much in these comparisons and in a number of cases you won’t be able to calculate them anyway because of the lack of correlation.

  54. tchannon says:

    Much of what are seeing is ocean moderation of land climate.

    What is the objective of what we are doing here?

    R-values? This term has many meanings. Guessing you mean to do with Statistics, will Pearson correlation do?

    I can probably hack the same code to do this. Looks like it just needs some top level selection logic, same output will do, different meaning.

  55. Roger Andrews says:

    “Much of what are seeing is ocean moderation of land climate.”

    That’s what it looks like on the map. But how does “ocean moderation” cause sunshine hours to lead SAT and SAT to lead SST? And how come the effects we see as we approach the coast are the same as those we see with increasing altitude in the Alps? More here than meets the eye.

    “What is the objective of what we are doing here?”

    Good question. Three things. First to expand my maps to include all ECA stations with data so I can’t be accused of cherry-picking just a few of them. Second, to evaluate whether sunshine hours have an impact on temperature, whether cloud cover has an impact on temperature and whether sunshine hours are a good proxy for cloud cover. Third, to see if we can make any sense of the lead/lags. I don’t know what the final outcome will be but based on results to date I’m pretty sure that we will at least be able to show that cloud cover isn’t a major control on surface temperatures in Europe, which is something worth knowing.

    The values in the “correlation” column in the spreadsheet you sent me are what I meant by “R” values. Sorry, I guess I should have been more specific.

    I looked at some of the mislocated stations on your plot. Here’s what I found:

    The lat/longs in the data base are correct
    The coastlines are correctly located relative to the lat/longs
    Most stations plot in the right location.

    Exceptions are (there may be more, haven’t checked them all):

    * Stockholm, plots in the sea about a degree E of where the lat/long coords say it should be

    * Kirkwall, Leuchars, Stornoway, Malin Head & some other Irish stations, all of which plot in the sea 1-2 degrees E of where the lat/long coords say they should be.

    * Most if not all of the stations in mainland Spain, which also plot 1-2 degrees E of where they should be.

    * Ditto stations in the Canary Islands.

  56. tchannon says:

    Summit like this?
    https://tallbloke.files.wordpress.com/2013/12/scatter-ss_cc.pdf (152k)

    I’d hard coded the key, why it is dismal anyway.
    Since I can’t figure out the plot package colour system I’m using a crude method in a spreadsheet which is an intermediate (and silly, just that it takes time and effort to automate bespoke file handling)

    ‘spose you want this too
    https://tallbloke.files.wordpress.com/2013/12/log-ss-cc.zip (49k)

    Low will be low correlation (I hope) a higher number, negative correlations, if you see what I mean.

    Suspect some dodgy station data.

    RESULTS HAVE NOT BEEN EYEBALLED I’ve just fed different data in, no errors, right kind of answers. I’m assuming this aligns the datasets. Any volunteers to check 900 files?

  57. tchannon says:

    Could add rivers and lakes, gets larger and more confusing.

  58. Roger Andrews says:

    Thanks Tim. I’ll take a look tomorrow.

  59. tchannon says:

    Started cleaning up.
    Co-ord error sorted. (maths floor problem with signed deg:min:sec, min:sec act incorrectly, should be negated, humans don’t, so, convert using unsigned degrees and change the sign of the answer if needed)
    Value to colour now a code routine and uses full range, can fiddle with this later, never any perfect solution.

  60. Roger Andrews says:

    I wasted most of today averaging the numbers into 1 degree grid squares because I felt sure this would make them plot up better. It didn’t.

    Maybe larger dots?

    A spectral color code (red, yellow, green, blue) might work best.

  61. Roger Andrews says:

    As far as I can see all the stations are now in the right place except for a few in the sea off the east coast of Spain. Don’t know if they are worth worrying about.

    I’m beginning to wonder whether one-degree grid square averages may not be the best way of presenting the results after all. Here’s what I’ve produced so far, it’s not cleaned up or properly aligned but it does show which way the trend goes:

    If I were to send you a spreadsheet giving lat-longs and grid square averages could you plug it in to your whizz-bang software and plot it out, maybe with large circles? Just asking 🙂

  62. tchannon says:

    Straight plotting is simple enough.

    data for plot is

    lat long ignored red green blue
    lat long ignored red green blue

    Planting a filled shape is easy, adding a column for shape and size would be simple.

    Could sent a plot script which does the outline (add own data file) but you would need to install the plot software.

    Contour really needs better coverage.

    I’ve plenty more to sort out yet.

  63. Roger Andrews says:

    I just came across some monthly average temp – cloud – sun hours data that may allow us to put a few more dots on the map in countries where we don’t have any, like France and Poland. I’ll do some checking and see what I can come up with.

    http://www.france.climatemps.com/

    http://weatherspark.com/averages/32139/Paris-Ile-de-France

  64. tchannon says:

    There must be more out there.

    Didn’t directly say, send the data if you want.

  65. tchannon says:

    Much improved results

    Click to access scatter_delay_ss-tg.pdf

    Click to access scatter_corr_cc-tg.pdf

    Click to access scatter_corr_ss-cc.pdf

    Click to access scatter_corr_ss-tg.pdf

    Data results, header files will contain wrong description, go by filename.
    https://tallbloke.files.wordpress.com/2013/12/corr-delay-results1.zip
    [updated]

    I’ve deleted several of my earlier comments with older files.

    WordPress runs much faster when few people are active 🙂


    Fixed station location error
    Geometry changed
    File naming improved
    Automation improvement
    Delay computation brought in line with other changes
    Plot scripts now automatically computed

    To be done
    Add additional column to intermediate work file to carry both delay result and correlation result so that two plots can be done from delay result, delay and correlation after adjustment
    Header naming as above.

  66. Roger Andrews says:

    Merry Christmas Tim

    Hope Santa brought you lots of toys.

    Plots looking better all the time.

    When I download your data files I get no header and can’t convert the numbers into separate columns. I don’t insist you fix it today, however 🙂

  67. tchannon says:

    Indeed Happy Christmas.

    Had a decent lunch and that’s it. Might write more later, probably not what people want to read.

    Now why couldn’t you use the data? Huh?

    Damn, brain ‘art. I bundled the intermediate files, the actual data with colour fed into the plot software, space delimited (it doesn’t care what)
    What I forget is now the “log” files have real names.

    https://tallbloke.files.wordpress.com/2013/12/corr-delay-results1.zip
    (I’ll point the previous link at this too)

    Sites off the Netherlands in the North Sea are gas platforms.

  68. tchannon says:

    I may be able to fill some of the blanks, eg. France since CLIMAT data includes sunshine and temperature.

    So far all I see is either maritime / continental or the effect of the dominant north east flow past Europe from the Atlantic.

  69. Roger Andrews says:

    I never heard of CLIMAT, but if you can use it to fill in the blanks then go for it, I say. 🙂

    Re maritime influences. They’re rather complicated, at least as far as SST/cloud relationships are concerned (ICOADS data, by the way):

  70. Peter Jackson says:

    Roger Andrews
    Two papers for you to ponder that are relevant re my recent comments:
    http://hidethedecline.eu/pages/latest-news-hidethedecline/latest-page-changes.php
    https://archive.org/stream/climatenearthegr032657mbp#page/n7/mode/2up

    The latter is support for the view that in research you need to go back 50 years – the life cycle of current ‘leading’scientists – to avoid the consensus cycle.

  71. Roger Andrews says:

    Peter Jackson

    Thanks for the links. The second is an example of the clear and concise way in which people used to write scientific papers but don’t any more. Now that I’ve learned how to turn the pages I shall read it with interest.

    The first link claims that “(based on GHCN V2 raw) Non-coastal temperatures … were much more cold trended from around 1930 than the Coastal trends.” I performed a similar exercise about five years ago using 856 unadjusted GHCNv2 records divided into “land”, “ocean” and “coastal” subsets and found that the coastal stations actually showed slightly less overall warming than the land and ocean stations, although I doubt the difference would be statistically significant.

  72. Peter Jackson says:

    Roger Andrews

    Your evaluation of coastal versu non coastal stations used GHCNV2 data but Frank Landsner has found that the majority of stations away from the coast that are sheltered from the ocean air effect (OAS) show long term cooling or little trend. These stations have not been included in the major data sets used in the IPCC evaluations – BEST, CRU etc. His study, so far, is published today in Wattsupwiththat:

    The Original Temperatures Project

    I stand by my view that, because of the influence of soil water and heat capacities and other topographical features on surface temperature, especially the minimum, that varies with precipitation , wind speed, ground cover etc etc, the land surface data sets are useless as a means of monitoring climate change on a global scale but could be of some value if done, as Frank is doing, related to distinct climate zones with similar topographical and other features as per the Koppen classification that rightly emphasizes the importance of precipitation. This extends to attempts to link SAT values globally or over areas of diverse topography with cloud cover or surface solar radiation. It would be better to do this only for individual long term stations that are known to have a limited UHI effect and see how they compare over the area of interest.