Introducing Berkeley Earth’s Data Pipeline 2.0

A rebuilt, expanded and more resilient foundation for the Berkeley Earth temperature record.

For fifteen years stakeholders from across the climate, resilience, and research ecosystems have come to Berkeley Earth for a clear, transparent, and trustworthy read on how our climate is changing. Much of that trust rests on work that has, until very recently, remained relatively invisible: the data pipeline that gathers, merges, and quality-checks millions of observations from more than 50,000 global weather stations every month.

Following a number of upstream data disruptions last year, Berkeley Earth made a commitment to invest in building a more resilient and robust data pipeline, one less dependent on any one single upstream source of observations, and one that could maintain crucial data continuity in the case that any single ingest product was disrupted or lost.

Today we’re excited to announce the launch of Berkeley Earth’s newly rebuilt pipeline 2.0, not only reinforced but expanded to provide not only the most comprehensive but also the most resilient observational temperature record available. Below is an overview of what we improved, what hasn’t changed, and why this matters for the greater climate and resilience ecosystem.

What’s New

• Nearly 3x more data ingest sources and 15,000 new weather stations. We’ve substantially expanded the number of independent ingest sources feeding our analysis, building in redundancy and resilience into the foundational layer of our pipeline.
• Observations straight from the national agencies. Rather than waiting for observations to reach us through a single intermediary, we now pull observations directly from the national weather agency itself, including: Météo-France, Germany’s Deutscher Wetterdienst, the UK Met Office, Environment and Climate Change Canada, Australia’s Bureau of Meteorology, the Japan Meteorological Agency, and Brazil’s INMET, with more on the way.
• Additional international observational collections were added, both current and historic, including: the WMO’s WIS2, Southern Africa’s SASSCAL, the DWD CLIMAT monthly summary, and the HARD historic arctic archive. Pulling directly from these collections significantly expands our coverage across the African continent, adds texture to our historic arctic archive, and adds significant redundancy to our existing data sources, ensuring additional pipeline resilience.
• Faster, and more resilient. Going straight to the source gets measurements into our hands sooner and builds real redundancy into the system. Our data pipeline is no longer dependent on any one source or intermediary for raw observations, leaving us in a far more resilient and robust position.
• More depth in high-demand markets. Thousands of new additional stations across grid cells new to our analysis strengthen an already world-leading network in the regions our work is used most, including the UK, France, and Germany.
• Full expanded station archive available on Synthesis. The full library of Berkeley Earth’s station data archive, including the newly added stations, is currently available via Berkeley Earth’s climate risk platform Synthesis. In the coming weeks and months we’ll be working to build near-real-time updates to the data availability on Synthesis, and we’ll update you once that’s available.

What Hasn’t Changed

• The warming trend remains unchanged – Many of the additional stations new to our analysis sit in regions that were already well monitored, so the fundamental historic trend remains unchanged…which is consistent with our expectations. What these additional stations do is create crucial redundancy in the case that a portion of reporting stations is lost, and ensures we’re able to continue delivering our climate monitoring program to our users and clients – no matter what happens upstream. 
• Our methodology does not change –  the upgrades to our data ingest pipeline described above sit upstream to the trusted bias correction and downscaling methodologies that define the Berkeley Earth Surface Temperature and BEST-HR high-resolution data products; our fundamental methodology remains unchanged following these upgrades.

What’s Next

The rebuilt pipeline is faster by design, which means earlier monthly updates going forward, and we expect to have near-real-time data flowing into Synthesis, our high-resolution climate platform, available within the next few months.

NOW AVAILABLE ON SYNTHESIS

Our complete station archive

Along with this rebuild, Berkeley Earth’s full station archive is now open on Synthesis, made available for researchers, analysts, and anyone who wants to get their hands on the data behind the record. A few key stats about our world-leading station data archive:

• Around 1.6 billion observations from tens of thousands of stations worldwide — one of the most complete land-station archives anywhere.
• Records stretching back to the mid-1700s, so you can trace change across centuries, not just decades.
• Observations from every continent, now with added depth across Western Europe, North America, and Brazil, drawn from national agencies and major international networks alike.
• We provide both raw and quality-controlled data side by side, with the open and transparent methodology that gave rise to our original surface temperature data product.
• Currently updating on a monthly cadence, with plans to build additional capacity to add near-real-time updates in the near future.

For any questions about the station data archive, please reach out to synthesis@berkeleyearth.org.