Met Office station data release

Posted: December 30, 2015 by tchannon in climate, Dataset, Surfacestation, weather

I am making available all the data collected from Met Office Datapoint for UK land stations.

Image

Example of data processed to show deviations.

This is hourly from 22nd July 2014 through 28th 31st Dec 2015, missing, etc. excepted. The data has been processed into time series with missing data filled with not available marks and also the verbatim datapoint XML as received.

A Talkshop page has been added, can get to it via top menu Portal, direct link here.

This ought to be a gold mine for those able to work on data. Millions of datapoints. The Met Office do not archive this immediate data for public access so whilst there are defects, you’ll have trouble finding this elsewhere.

Image

Example showing extraction of climatology estimate from short data. Details on linked page. Not perfect but not too bad either for < 2 years of data.

Station locations

The map as the first page of a day thumbnail plot is intended to help. Shows location and the squares are clickable, invokes Bing maps. (not all exact locations are known, Not all have good images)

File created 28th Dec 2015 (1MB PDF)

If you use PDF search on the station ID it will take you to the plot for that station, which includes name, location, etc.

Foot
Exactly one person has said they took a copy of the Synth software and that it works.

Post by Tim

Comments
  1. tallbloke says:

    A geat resouce, well done Tim. Are you continuing to collect and update? I think we should try to get a volunteer team to take this on and get other computers prepared.

    I’ll do a post about the synth software to engage more interest.

  2. E.M.Smith says:

    It looks like the graph labels might be wrong?

    MAX on the bottom with MIN near the middle temp range?

  3. tchannon says:

    Automatic data collection has been put in place, not as simple to do as it might seem.

    An old bitsa PC here is on a mains timeswitch. (keep it simple stupid)
    Mains power outages here will time slip, not going to worry about it. We might lose a few hours once in a while.

    XP Pro runs the script(s) on computer start and if after a sensible pause if no-one logs in will shut the PC down. The timeswitch goes cluck off an hour later, long enough for Met Office server retries as necessary, has to be done without upsetting sysops.
    Collected data is verbatim received XML as a timestamped disk file. Can be copied over the network into the main archive if needed.

    Emails an account here after logging.

    A manually run script named regen looks for any unprocessed data and does things. Will spot any aliens. Duplicates are benign.

    Work? Yes, spot the xml file, is in the zip archive.
    Network is always on, firewalled.

  4. tchannon says:

    Thank you ems, fixed.

  5. Tim – bizarre as it seems, the NOAA data is not stable month to month, even though you might think the importance of climate change would have inspired a definitive clean-up of data and data collection 15 years ago. What is the case with this data? Is it “terminally” clean and new data so smartly recorded it doesn’t need adjustments anymore?

    At some point science should not have wiggle room (outside of error bands). The Yanks haven’t got this concept yet. Have the Brits?

  6. tchannon says:

    Doug, the data I make available is pre-quality control and has known faults. (which I think are programming cockups by Met Office staff). Broadly this is station data, not going to get changed as such post hoc. Too many cross checks could be done. (simple case, archived METAR data)

    I think I checked a few cases against published means, seems okay.

    If I get around to it a hell of a story appears over the July 1st 2015 Met Office heat spike. I’ve done the work others didn’t or couldn’t. Bottom line is questioning the validity of all maxima in recent years, equipment matters. Might wake some people up to the subtlety of issues.

  7. tchannon says:

    Looks like I for the first time have a full year of data, 2015.

    There are many holes, the nature of Datapoint. I know one day is missing, otherwise this is about the erratic nature of station communications, some probably Met Office bugs, been like that for years. (see web)

    It will be possible to script up computing daily, monthly, and annual. The easiest way is use meteorological mathematics, common average. Handling missing data is the hard call, what to do. This can probably be ignored for annual and monthly, no material effect.

    Another call is which parameters to compute.

    Might do this later. (and update the data archives on-line here)

  8. tchannon says:

    Updated data, 2015 is now complete.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s