Roger Andrews: Chunder down under – How GHCN V3.2 manufactures warming in the outback

Posted: October 11, 2012 by tallbloke in Analysis, Dataset, Incompetence, Measurement, methodology, Surfacestation

CHUNDER DOWN UNDER

HOW GHCN v3.2 MANUFACTURES WARMING IN THE OUTBACK
by Roger Andrews

I originally entitled this post “Blunder Down Under”, but then it occurred to me that all of what I’m about to describe wasn’t a blunder at all, but intentional. So I changed “Blunder” to “Chunder”, which as those of you familiar with the Strine language will know, means “throw up”.

NCDC recently released the latest version of the Global Historic Climate Network data set – GHCN version 3.2, which applies adjustments to remove the artificial man-made discontinuities and spurious gradients that allegedly plague the “raw” GHCN Version 2 data.

I started to go through the GHCN v3.2 data, and the first record I looked at was Alice Springs in the middle of Australia. I plotted the “raw” GHCN v2 data against the adjusted GHCN v3.2 data for the station, and here’s what I got:

http://oi49.tinypic.com/2e34y6d.jpg

GHCN v3.2 adds not one, not two, but three whole degrees of warming to the Alice record since 1880.

Well, I thought, not even NCDC would do that without some justification, which it would have, for example, if the GHCN v2 records from stations around Alice all showed strong warming. So I started to download the records from surrounding stations to see what they looked like. There aren’t too many records in this area, but I found Giles 580km southwest, Halls Creek 900km northwest, Tennant Creek 460 km north, Urandangie 510km northeast, Boulia 620 km east and Oodnadatta 440km south – all of them with readings covering a period of more than 50 years, although with some gaps, most notably at Oodnadatta. The Google Earth image below shows where these stations are relative to Alice. The seven stations combined cover approximately 1.5 million sq km, or 1% of the Earth’s total land surface area.

http://oi50.tinypic.com/2z6c6yu.jpg


http://oi50.tinypic.com/2d0az2u.jpg
And here’s how the records from the six surrounding station compare with Alice:


And the seven records together show little or no warming since 1880.The match isn’t perfect, but when seven unadjusted records compare this closely we can be reasonably confident that they’re recording real and not man-made temperature changes. (The plot excludes three 2-sigma outliers; Boulia in 1893 and 1894 and Halls Creek in 1949. The 1957-84 period is used as the baseline because all seven stations were operating in those years.)

But maybe GHCN v3.2 applied warming adjustments just to Alice Springs and left the other six records intact. Did it? No, except for Tennant Creek, which for some reason receives a net cooling adjustment, it applies large warming adjustments to the other records too:

http://oi49.tinypic.com/27xikh0.jpg


Where do these warming adjustments come from? They’re generated entirely by the algorithms NCDC applies to correct out supposed artificial discontinuities and spurious trends in the GHCN v2 data. These algorithms sound impressive when you read about them, involving as they do the use of non-parametric Kendall-Thiel paired function looping indexes and a Bayesian Information Criteria model among other things (more details here), but in many cases what they do in practice is simply adjust records that don’t show as much warming as they should to match records that do. And in this case they’ve matched the GHCN v3.2 records in the Alice Springs area to the GHCN v2 records from the much smaller Nullarbor Plain area to the south, which do show warming: (Note that “Alice Springs Area GHCN v2 Records” should read “Alice Springs Area GHCN v3.2 Records. I’ll correct this when I get time.)

http://oi49.tinypic.com/ou1p9c.jpg

There’s only one possible justification left for the GHCN v3.2 adjustments – that all the records in the Alice area except for Tennant Creek are cooling-biased by station shifts, as surmised by NCDC in its FAQ writeup here:

“Why are there more cold (negative) step changes than warm (positive) step changes in the historical land surface air temperature records represented in the GHCN v3 dataset? The reason for the larger number of cold step changes is not completely clear, but they may be due in part to systematic changes in station locations from city centers to cooler airport locations that occurred in many parts of the world from the 1930s through the 1960s.”

So I checked through the metadata for the seven stations to see whether they had all been moved from warmer downtown to cooler airport sites over the period of the GHCN v2 temperature record. I found that Alice Springs moved from downtown (?) to the airport in 1940 and Tennant Creek moved from downtown to the airport in 1969, but that was it, and neither move caused any significant shift in the temperature record, so that justification fails too.

We are now in a position to conclude that the GHCN v3.2 warming adjustments applied to the GHCN v2 records in and around Alice Springs have manufactured approximately 2C of nonexistent warming since 1880. Spread over an area of 1.5 sq km this adds only about 0.02C of non-existent warming to the global land surface air temperature series, but it adds about 0.2C of nonexistent warming to the series in the Southern Hemisphere, where there’s a lot less land.

And although I haven’t checked in detail it appears as if the GHCN v3.2 algorithms may have manufactured a lot of nonexistent warming over much of South Africa and South America too.

So while we ponder the question of whether UHI impacts, poor station quality, land use changes etc. might have introduced spurious warming into the surface temperature record, Team AGW is busily manufacturing it. And getting away with it too.

Comments
  1. Good. See also my short“US Temperatures Falsely Adjusted…”, which indicates just how simple-minded and fraudulent have been the adjustments in US temperatures that Steven Goddard has uncovered.

  2. Bloke down the pub says:

    Chunder is one of the long list of words that owes it’s origin to the Navy. When someone was about to puke over the side ( or throw gash overboard) the cry would go out to ”watch under”.

  3. tallbloke says:

    Why can’t GHCN give an extra 3C to somewhere cold and rainy like Britain?
    Alice springs is already hot and sunny.

    Not fair. :(

  4. HG says:

    Reblogged this on Howard Canitbe?.

  5. Roger Andrews says:

    TB:

    “Why can’t GHCN give an extra 3C to somewhere cold and rainy like Britain?”

    Because Britain already shows an AGW-compliant amount of warming. It’s those bludgers Down Under who are letting the side down. :-)

  6. Doug Proctor says:

    Has anyone ever seen a defense of such large “adjustments” that makes sense?

    Taking temperatures pre-1970 was a pretty simple operation. “Fixing” the observations by 3C must have a pretty simple reason, like the observations were made at 10 am and they are now made at 7am. But that should lead to step-function changes. The progressive changes boggle me.

    Anyone help? Not in the general, arm-wavy way, but in the specific.

  7. Paul Matthews says:

    Interesting post – the comparison with the other stations shows that for example the sharp cooling in the 1970s was genuine so the GHCN algorithm is wrong to put in an adjustment there. It’s the same story with Iceland.

    Paul Homewood writes about the new GHCN version here.

    In the GHCN FAQ document it says that the code that does the adjustment algorithm is available on the ftp site, http://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/software/ – but it isn’t, there’s just an empty folder.

  8. Zeke says:

    (That is an automatic message that comes up.)

    Looks like Roger Andrews has detected a historic Beynesian chill in the Outback of Australia, due to the use of non-parametric Kendall-Thiel paired function looping indexes by the Aboriginals no doubt! (:

  9. Roger Andrews says:

    Doug Proctor

    According to the metadata all of the stations I used had max/min thermometers, so changes in observation time shouldn’t enter into it here.

    A change in observation time from 7am to 10am, or from midday to 6pm, will cause a glaringly-obvious shift of several degrees in most records. I don’t remember seeing shifts of this magnitude in any GHCN records anywhere except at Jujuy, Argentina, and this was probably caused by a station move.

  10. J Martin says:

    It’s the same widespread surreptitious systematic delusional f—- that NOAA, GISS and others also practise.

    Eventually they will be publishing temperatures 6 degrees above reality. This could backfire on them when the low solar cycles increasingly makes it’s presence felt as we get past the drawn out (double ?) peak and descend to an extended period without sunspots.

    They will then have a bigger gap to explain between their fake warming and reality.

  11. So they suspect the empirical surface station data is not correctly reflecting the actual warming.

    So they form a ‘plausible’ hypothesis about why the data is under-recording the warming.

    So they write a computer program that faithfully and objectively reflects their hypothesis.

    So the program faithfully outputs the desired warming effect.

    So they congratulate themselves that they were right all along to suspect that the raw data didn’t show enough warming.

    So they call this proven science.

    Whereas we all know it is all unproven nonsense. Perpetrated by sad people who know no better.

  12. Streetcred says:

    From Jo Nova’s blog: “Threat of ANAO Audit means Australia’s BOM throws out temperature set, starts again, gets same results.”

    http://joannenova.com.au/2012/06/threat-of-anao-audit-means-australias-bom-throws-out-temperature-set-starts-again-gets-same-results/

  13. Streetcred says:

    From Jo Nova’s blog: “Australian temperature records shoddy, inaccurate, unreliable. Surprise!”

    http://joannenova.com.au/2012/03/australian-temperature-records-shoddy-inaccurate-unreliable-surprise/

  14. Streetcred says:

    From Jo Nova’s blog: “Announcing a formal request for the Auditor General to audit the Australian BOM”

    http://joannenova.com.au/2011/02/announcing-a-formal-request-for-the-auditor-general-to-audit-the-australian-bom/

  15. Exactly the same in Iceland, Roger. Every single station there has been adjusted upwards. The severe downturn in temps there in 1965 seems to have convinced the algorithm it was due to non-climatic bias. But the equally sharp uptick in temps later did not trigger the same assumption.

    http://notalotofpeopleknowthat.wordpress.com/2012/10/12/ghcn-adjustments-in-iceland/

  16. Roger Andrews says:

    Paul:

    Thanks for joining in. I’m aware of the work you’ve been doing on the question of “adjustments” to the surface temperature record and it seems that you’ve reached pretty much the same
    conclusions as me. In fact I don’t see how anyone who knows anything at all about data analysis and who isn’t drunk, stoned or insane could possibly conclude that these adjustments are valid.

    One difference between your Stykkisholmur example and the Australian records is that the warming at Stykkisholmur is manufactured by a single stepwise warming adjustment while the warming added to the Australian records is manufactured by a combination of stepwise and gradational warming adjustments. I think these gradational adjustments are added simply because the Southern Hemisphere doesn’t show as much warming as it should. Maybe back with more on this later.

  17. Doug Proctor says:

    I have worked with intellectual geologists and practical geologists. The intellectual ones can always justify their odd conclusions using bits of information they pull from left field. The practical ones talk about “sniff” tests that fail.

    I was looking at some core recently. I saw tidal couplets, very thin beds of sand capped with even thinner laminae of shale, repeated over and over again, same thicknesses. I saw fossilized, brackish water snails, oyster shells and carbonized plant fragments along minor erosive beds. So I went, “Estuary, like Cheasapeake bay.”

    The response was, “Well, we’d have to do a lot more study to say that!”

    There were oyster shells. Non-marine snails. Repeated deposits of tides. Bits of wood debris. A practical man would suggest that the world that has this attributes like Chesapeake Bay in this spot is pretty much going to be like the world of Chesapeake Bay only a couple of miles away, but I guess you could argue …

    That’s the point. If your life is based in arguing and studying things ad infinitum, “sniff” tests mean nothing. There is always an alternative that MIGHT be the answer. Or worth having them study before you do something.

    I guess a miracle of random fixes in multiple places could result in a global trend of progressively warmer temperatures today and cooler ones, yesterday. I suppose a tsunami from Japan could have brought my those oyster shells to Alberta where there was actually a displaced Humbolt current keeping everything cold. Maybe I’d better not get a drilling rig going before the geo-whiz guy can put another few months on the job.

  18. Eric Barnes says:

    Here’s noaa’s own quality control graphs for alice springs.

    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/products/stnplots/5/50194326000.gif

    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/products/stnplots/README

    Last Updated: 09/29/2010

    The following directory:

    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/products/stnplots/

    is comprised of sub-directories (that are named by the first digit of a station
    ID) that contain individual station plot files (in “gif” format).

    The plot files contain 9 individual graphs, arranged in a 3×3 matrix. The first
    column of graphs, contain 2-D colored symbol graphs of the actual monthly data
    for the entire period of record for A) the (Q)uality (C)ontrolled (U)nadjusted
    (QCU) data, B) the (Q)uality (C)ontrolled (A)djusted (QCA) data, and C) the
    differences between QCA and QCU monthly data. The second column of graphs
    contain histograms of the monthly data for QCU, QCA, and (QCA-QCU) respectively.
    Finally, the third column of graphs depict annual anomalies and their associated
    trend line for QCU and QCA, and the differences in the annual anomalies for QCA
    and QCU. Detailed axis titles and units are displayed in the title of each
    graph.

  19. Roger Andrews says:

    Eric:

    Thanks for the links. I can see they’re going to come in very useful.

    On your favorite US station, I think I forgot to mention that the NCDC algorithms also applied a Semi-hierarchical Segmentation function corner case. I’m sure that’s what did it. ;-)

  20. Brian H says:

    Busy, busy, busy
    -Vonnegut, Cat’s Cradle

  21. Roger

    There’s more confirmation here from the BOM that there have been no changes in location, time of observation or equipment , since they moved the station to the airport in 1941.

    And of course there must be more UHI effect at the airport now than then!

    http://notalotofpeopleknowthat.wordpress.com/2012/03/26/alice-springsa-closer-look-at-the-temperature-record/

  22. Roger Andrews says:

    I doubt there’s much of a UHI effect at Alice Springs Airport.

  23. Eric Barnes says:

    Thanks for your work Roger. :)

  24. tchannon says:

    I read this as no-one knows precisely where the equipment is, when and what things changed, which they certainly have between WWII and fly by wire.

  25. Roger Andrews says:

    The tinypic above shows precisely where the Alice Springs Airport station is. With one exception I also know precisely where all the other six stations are, or were, and when things changed.

    More to come.

  26. tchannon says:

    Ooo… so I am wrong, excellent.
    [occurred to me later this could be taken not as I intended, which is amusement over declaring a wrong opinion and hoping good stuff is about to appear]

  27. Good post Andrew. It seems that this type of adjustments are being made all over the world. The validity is highly questionable.

    However, it is only part of the story. The homogenization process also appears to have a significant impact on the resultant global series. I don’t want to detract from the thread topic, so will only post this link – maybe it will become a thread in its own right?

    http://itia.ntua.gr/getfile/1212/1/documents/2012EGU_homogenization_1.pdf

    The document needs no real introduction, but is a presentation from earlier this year. If anyone has a copy of the relevant paper, I’d appreciate a read.

  28. Roger Andrews says:

    Hi Jonathan:

    Thanks for the link, and the subject matter isn’t in the least O/T. But on skimming through the results I find that the authors’ homogenization process has somehow resulted in a COOLING adjustment at Alice Springs:

    It seems that the validity of homogeneity adjustments is indeed highly questionable. :-)

  29. Hi Roger,

    I must apologise for the inadvertent name swap. After I posted it I skimmed it again and noted the Alice Springs cooling and wondered if anyone would notice – you did :-)

    Anyway, I think it is pretty obvious that the homogenization process is able to carry trends over significant spatial areas but it is probably aided by breaks in the time series, and transforms to anomalies. The procedures presently used to create global temperature records seem to need fundamental re-evaluation.

  30. Roger Andrews says:

    Jonathan:

    Above I posted a graph showing a respectably close comparison between seven unadjusted temperature records in and around Alice Springs. Here it is again:

    http://tallbloke.files.wordpress.com/2012/10/ra3.png?w=921&h=525

    The match is close enough that we can accept the records as being substantially free of artificial distortions and therefore representative of actual temperature trends in the area. They don’t need adjustment. They’re OK as they stand.

    Now let me throw in another example from Iceland that I recently posted on Paul Homewood’s blog.

    Here we have another seven unadjusted records that also don’t need adjustment. They’re already representative of actual temperature trends in the area. Or at least they were until GHCN v3.2 trashed them.

    And hundreds of records in other parts of the world show equally good matches. I know because I’ve gone through them and matched them up.

    Bottom line? The homogenization procedures presently used to create global temperature records don’t need fundamental re-evaluation. They need to be junked in their entirety.

  31. […] trattati di una porzione del territorio australiano per il periodo 1880-2011 è pari a oltre 2°C. Se avete voglia di leggerla qui trovate tutta la spiegazione, altrimenti potete aspettare domani, perché pubblicheremo un’analisi un po’ più […]

  32. […] corner: cloud cond…Bernd Felsche on Photography corner: cloud cond…Climatemonitor on Roger Andrews: Chunder down un…Joachim Seifert on Joachim Seifert and Frank Lemk…tallbloke on Joachim Seifert and Frank […]

  33. Paul Matthews says:

    Jonathan / Roger, The Steirou & Koutsoyiannis paper (in fact just a conference report, I dont think there is a proper paper yet) was based on GHCN v2, a few versions before the current v3.2.
    As you say, in that version it looks like the adjustments cooled Alice Springs while the current version warms it.

    One further mad feature of the GHCN adjustments that I and Paul Homewood have been looking into is that the adjustments are constantly changing. See for example this figure which shows the Alice Springs data after adjustment as reported by v3.1 at several different times earlier this year. If you had downloaded the GHCN data in March you’d conclude that they created 2degrees of warming in Alice Springs, but if you’d done it in May you would have seen a 1 degree cooling!
    This is also on Paul Homewood’s blog <a href="http://notalotofpeopleknowthat.wordpress.com/2012/06/24/temperatures-altered-yet-again-in-alice-springs/"here.

  34. tallbloke says:

    Thanks for that Paul. It’s just crazy, the adjustments merry go round makes a mockery of the temperature histories so painstakingly gathered by the people in the field.

  35. Roger Andrews says:

    Paul:

    I did in fact download GHCN v3.1 for Alice Springs, and also a for lot of other records in Australia, Southern Africa and South America, and I was well into writing the results up when v3.1 suddenly got superseded by v3.2. But you’re right, GHCN v3.1 did add a cooling adjustment to Alice.

    I spent some time looking at BEST too. The BEST series added a +/-0.5C warming adjustment over the period of the Alice Springs record, and then it added another +/-0.5C of warming before that by extending the Alice series back to 1853, 25 years before the Alice record begins. BEST manufactured the extra 25 years of data using its whizz-bang kriging program, which projects data in from stations thousands of kilometers away.

  36. Roger,
    “They [homogenization procedures] need to be junked in their entirety.” Yes, agreed. I was being a little too diplomatic. There is far too much extrapolation within these processes both temporally and spatially. How suited is kriging (or their version) to data which is non-stationary and with potentially asynchronous variations?

    Paul,
    Thanks for the update on the S&K paper. Your observations on the ever changing nature of the adjustments makes me wonder whether there is a problem with the code or database, or more likely both. When I last looked at GHCN it was less than impressive [being generous]. I never got around to writing anything as the version changed and I couldn’t find time to go through the processing again.

  37. Tim Cullen says:

    The NCDC adjustments imply that their “Short Term Memory Loss” kicks in after 18 months.

    So I guess we will have to wait a little longer before July 2012 is erased from the NCDC data bank…

    Plumbers flat out during cold snap
    By Emma Sleath (Cross Media Reporter)
    9 July, 2012 12:16PM ACST

    Burst water pipes all over town are keeping local plumbers busy as a new cold weather record is set in Alice Springs.

    Local plumbers are currently in high demand as the cold weather wreaks havoc on water pipes all over town.

    By Saturday (when we hit a chilly -5.2 degrees), Alice Springs had endured five nights in a row of minimum temperatures below -4, a new record in the region.

    http://www.abc.net.au/local/stories/2012/07/09/3541747.htm?site=alicesprings

    On the other hand:
    Perhaps ingesting CO2 causes “Short Term Memory Loss”…
    Perhaps ingesting CO2 is associated with withdrawals from the dope data bank.

  38. Paul Matthews says:

    I asked S&K – they have not written a paper for publication yet, but they hope to.

  39. Roger Andrews says:

    Jonathan:

    You ask “How suited is kriging (or their version) to data which is non-stationary and with potentially asynchronous variations?”

    In real life I’m a mining consultant specializing in ore reserve estimation and I therefore get to do a lot of kriging. In fact I did a couple of kriging runs yesterday. So while I can’t claim to be an expert in kriging theory – and therefore don’t have a ready answer to your question about non-stationarity – I do know a bit about how it works in practice.

    Kriging has been around for a long time (it was developed by Danny Krige, a South African mining engineer, back in the 1950s) but there’s still a tendency for people who discover it for the first time, as BEST did a short while ago, to raise their arms to Heaven and say “Wow! This is the answer to a maiden’s prayer”. But of course it isn’t. It’s just another way of averaging data, and a complicated one too. For example, I have to input the following parameters just to carry out a simple kriging run:

    Variogram nugget
    Variogram range
    Variogram sill
    Length of primary search axis
    Azimuth of primary search axis
    Plunge of primary search axis
    Length of secondary search axis
    Length of tertiary search axis
    Maximum number of samples
    Minimum number of samples

    And that’s just ordinary kriging (OK). It gets more complicated if I want to do indicator kriging (IK), probability-constrained indicator kriging (PCIK), multiple indicator kriging (MIK) or outlier-restricted kriging (ORK).

    BEST does have one advantage in that they need only krige temperatures over the earth’s surface, i.e. in two dimensions, but this advantage is offset by the need to perform a separate run for each month (which is the only way of getting around the “asynchronous” problem you mention) and this would presumably have involved over 3,000 separate kriging runs.

    BEST’s kriging procedures, however, are a recipe for disaster. BEST uses its BerkeleyAverage program to create “an estimated temperature field for the entire land surface of the globe using Kriging to interpolate the available temperature data.” But how do you create a temperature field for the entire land surface of the globe when all you have is a few stations in the Northern Hemisphere? You don’t. Neither kriging nor any other statistical method will give you anything better than garbage results. That’s why I think the BEST series, arguably before 1900 and certainly before 1850, should, like homogeneity adjustments, be junked in its entirety.

  40. Roger,
    Thank you for the explanation. Its good to know you have a working knowledge of kriging and thus can comment from experience. You have confirmed my thoughts. I think you are probably being generous with the 1900 figure, let alone 1850!

    Having spent hours checking & processing GHCN, I don’t believe such spatially non-uniform, sometimes sparse, noisy, error (& adjustment) ridden data, irrespective of the processing, can give an accurate reflection of global temperatures. That said, there might be alternative methods – ones that do not rely on ‘smearing’ processes – which ought to be considered and which may be more appropriate. Such a method/s would either confirm the presently accepted global records or become a challenge to the status quo. It would be a lot of work.

  41. Roger Andrews says:

    Jonathan:

    “I don’t believe such spatially non-uniform, sometimes sparse, noisy, error (& adjustment) ridden data, irrespective of the processing, can give an accurate reflection of global temperatures.” Take out the adjustment bit, substitute gold (or silver, copper, lead, zinc or whatever) grades for temperatures, and we miners handle that sort of data all the time. And we usually get it right. :-)

  42. Brian H says:

    ” Jonathan Drake says:
    October 17, 2012 at 8:46 pm

    Having spent hours checking & processing GHCN, I don’t believe such spatially non-uniform, sometimes sparse, noisy, error (& adjustment) ridden data, irrespective of the processing, can give an accurate reflection of global temperatures. That said, there might be alternative methods”

    Give it up. Lost cause. 35°C is not the same thing in Perth, Sydney, Borneo, Tripoli, and Berlin. RH changes everything. TEMPERATURE CANNOT BE VALIDLY AVERAGED ACROSS DIFFERENT RELATIVE HUMIDITY REGIMES. The enthalpy forbids.

  43. Roger,
    “…we miners handle that sort of data all the time. And we usually get it right.” I know, but you do the processing correctly :-) I work in related industry and could potentially have been indirectly involved with the data you use.

    Brain H,
    I have given up – for the time being at least – my PC struggles with the workload. As you rightly imply, global temperature is a pretty meaningless figure, but it is the basis of the AGW tenet and therefore it must be challenged.

  44. […] siti che trattano la modifica delle serie di temperatura osservata, mi è stato segnalato da GG Tallbloke, con un post di Roger Andrews che si riferisce ad una stazione australiana, Alice Springs, dove i […]