Data errata caper: CRU methodology “inexplicable”

Posted: November 25, 2011 by tallbloke in Incompetence

Back in 2009 at the height of the climategate I revelations, we were all highly amused when we found a file called ‘Harry ReadMe’. This was the running log of Ian ‘Harry’ Harris, who had been tasked with sorting out the mess which was (is?) the University of East Anglia’s Climate Research Unit database system. It appears this runs on homespun Fortran written by a self taught ‘programmer’. For those not old enough to know what Fortran is, don’t worry, you’ll never need to know, unless you get a job in a computer museum.

In the climategate II email folder, I just came across this exchange, which gave me a chuckle. I’ve reversed the order of the responses, so it works as a narrative.

#0545

>>> From: “Wladimir J. Alonso” <REDACTED>
>>> Date: 23 October 2007 19:13:20 BDT
>>> Cc: REDACTED
>>> Subject: Vapor pressure in CRU TS 2.10
>>>
>>> Dear Ian,
>>>
>>> I am using the dataset CRU TS 2.10 on an ongoing study on worldwide
>>> diversity of palms, and we noticed that there is something a bit
>>> strange
>>> about the vapor pressure. As you can see in the attached image
>>> generated
>>> from one month (Jan/1981) picked up at random from the TS 2.10
>>> dataset, the vapor pressure present high values near the polar
>>> regions. This is in contrast with the data you present at http://
>>> http://www.cru.uea.ac.uk/~markn/jpg/glo_vap.gif
>>>
>>> I would much appreciate if you could help us on this.
>>>
>>> Thank you very much,
>>>
>>> Wladimir

> At 08:08 24/10/2007, Ian Harris wrote:
>> Phil, Tim
>>
>> Any ideas? I think he means ‘Northern Eurasia’ rather than ‘polar’.
>>
>> Harry
>>

On 6 Nov 2007, at 16:34, Tim Osborn wrote:

> Hi Harry — still catching up on my emails… is the query below
> now resolved because the wet-day counts and vapour pressures had
> been interchanged (or some explanation like that)???? Cheers, Tim
>

Hi Tim,

Yup – all the decadal 2.10 files for WET and VAP had been swapped.
And rescaled to look like they ought to. Inexplicable.

Harry

Comments
  1. hro001 says:

    Yup – all the decadal 2.10 files for WET and VAP had been swapped.
    And rescaled to look like they ought to. Inexplicable.

    Inexplicable?! Nah … must be another of one of those ‘well-documented in the literature’ “tricks” of the CRUsaders!

  2. tchannon says:

    Fortran has nothing to do with this.

    What you are looking at is the usual consequences of a group activity whereas the gems are under the control of a single will where they really are competent.

    I’ve seen plenty in the software world (but applies everywhere), lot of tales I could tell but ultimately the following is the truth in the real world. Those who know will know or chuckle, those who don’t, here is a towel, dry and darn well learn.

    http://www.laputan.org/mud/

    Fortran is old. And?
    It is highly examined and known and long sorted out for particular usages. It happens be be extremely good at what it does.

    Have a chuckle.

    I was working on the Met Office release of GHCN data, related to a CRU database. Wanted to look at some far Siberian stations where I learnt a surprising lesson about Siberia, USA. Why the Alaskan stations?

    Go look, I must have made a coding mistake, imported wrongly.

    Nope. Met Office don’t know their left from their right, Moscow is to the west of London.

    They use negative longitude for east, reversed from everyone else. This is part of the minefield of every Tom, Dick and Harry do their own different way, often I suspect just to be cussed. Take the RSS and UAH gridded data, RSS runs longitude zero to 360, UAH -180 to 180

    There again, assume nothing. Okay read about it. Umm… where and docs are also notoriously wrong.

    If bad things make it out of the door start worrying about what else.

  3. pouncer says:

    Upsidedown Tijander joined by topsy-turvey WetVap?

  4. tallbloke says:

    Pouncer: Indeed.

    Tim: Fortran still runs steel rolling mills in realtime. I t’s fine, if arcane and inefficient for purposes where agility and repurposing are commonly needed.

    The question mark here is who ‘rescaled’ the indices so they ‘looked right’. And why.