Disclaimer: The goal of this deep-dive is to do exactly what it says - “to demystify weather.” I have done my best to simplify (hopefully not oversimplify) weather for everyone, especially the space segment responsible for observations. While I am not a meteorologist or atmospheric scientist, I have spoken to dozens over the course of my career in Earth Observation (and my stint at Tomorrow.io). If something mentioned below is blatantly wrong or rather incomplete, please send me a note. All kinds of (hopefully) polite feedback is welcome!
Context
Growing up in a tropical country - India - I have had my fair share of extreme weather experiences, including a couple of times when our house was flooded during the major cyclones in Chennai (in 2005 and 2010). One experience that I would never forget was in 2015 (Cyclone Vardah) when I had already moved away to Europe, but during a period of 3 days I had lost contact with my entire family and most of my friends, back home - not knowing if they were displaced or evacuated. Thankfully, nothing serious happened, as thanks to the early warning systems from the national meteorological agency, my family managed to evacuate on time. I am sure many of you reading this have had similar experiences. And, if you are running a business or employed in one, you might have several such stories that possibly resulted in economic loss.
As we look ahead to COP27, when the world’s politicians will meet up to decide the fate of the planet and how we plan to respond to the changing climate, I think it is worth diving deep into the sector that makes all the science possible and more importantly, makes everyone on the planet “feel” climate change. Weather is a manifestation of climate - and as the planet warms, that’s how we will feel it more.
Weather used to be something we all used to ‘break the ice’ with, but today it is something that is melting the ice - in a way that it probably shouldn’t.
In this piece, I attempt to demystify the weather sector, with an emphasis on the space segment and the growing significance of the private weather sector.
Weather vs Climate
But first things first … While most of the observations used to monitor and predict the climate originate from pretty much the same sources (mentioned below in observations), and there are inherent similarities between weather and climate models, I will be only focusing on weather in this piece. Several satellites mentioned in the piece, in fact, contribute to observing the Global Climate System and acquire direct data from at least half of the 54 Essential Climate Variables.
Weather refers to short-term changes in the atmosphere impacting the conditions on a minute, hour and daily basis, while climate refers to average weather conditions over years, decades or more. Source: ESA
And, if you want something more fun:
Weather is your mood; climate is your personality. Weather is what you are wearing today; climate is what is in your closet!
Source: Dr. Marshall Shepherd on the TerraWatch Space podcast
Agenda
I have thought about this piece for a long-time and made multiple edits and revisions to the overall content - but throughout, I always stuck to my initial goal of demystifying weather from space, for everyone, while sticking to the science and not oversimplifying things. There might be a follow-up, but here is what we’ve got in store:
- An Overview of the “Weather Operating Stack”
- A Summary of the “Observations” layer
- Deep-dive on “Weather from Space”
- My “Pesky Points of View”
- Some Final Thoughts
Note: The history of weather forecasting and how it came to be what it is today is an incredibly fascinating story, but I am not going into any of those details in this piece. But, if interested, I would suggest you check out The Weather Machine.
The Weather Operating Stack
The following figure shows an overview of the weather operating stack of the value chain. As you can see, this is a pretty complex ecosystem - I have tried my best to simplify it for everyone while keeping the essence of the science. Let’s dive in!
Note: Not included in this are subcontractors of NASA/NOAA such as Ball Aerospace, L3Harris, Raytheon Technologies BAE Systems, GeoMetWatch, Lockheed Martin etc. - although have built major satellites or instruments, they don’t operate in the weather industry. Nevertheless, their contributions are vital for the space segment.
If you are enjoying this, please share!
Observations
This layer has to do with acquiring data from multiple sources (see below), calibrating & validating it and disseminating it for integration into the models. The most fascinating bit about the observations segment to keep in mind is that it is globally coordinated through the Integrated Global Observing System, led by the World Meteorological Organisations (WMO). So, whether it is deciding what type of data to collect, which instruments to launch, where they are launched to and when data is collected, all of which is coordinated and shared on an international level, at least from a governmental standpoint (I would argue, this is a larger example of global cooperation than the often cited International Space Station)
Modelling
The variety of observations acquired are assimilated into numerical weather prediction models (NWP) - the two most common ones are the Integrated Forecasting System (IFS) from the European Centre for Medium-Range Weather Forecasts (ECMWF) and the Global Forecast System (GFS) from the US National Weather Service (NWS). As technologies advance on the processing (artificial intelligence) and computing (cloud) ends, these two major NWP models are expected to rapidly advance in their performance, going up to kilometre resolution, especially as they become open-source.
Products
Products refers to the day-to-day forecasts of the various parameters we hear about on a daily basis: temperature, humidity, winds, precipitation and more, depending on the needs. While the products are largely the result of the NWP models developed by the meteorological agencies, those from some companies include additional advancements - on the observations end and/or the modelling end. Some products are more suitable for nowcasting (prediction for the next few hours), while others more for medium-range and long-range forecasting (for the next few days).
Insights
This layer has to do with contextualising the weather information and transforming it into actionable intelligence - whether it is for an individual or a business. As such, this is accomplished by simplifying all that complex weather information and representing them on mobile applications and desktop dashboards. Even though several meteorological agencies part of the value chain provide Insights, the role of the private sector has been historically significant - essentially repackaging the Products and making them more intuitive (as smileys for individuals and alerts for businesses).
Zooming In On Weather Observations
Although weather forecast models rely on satellite data more than any other weather observation (anywhere between 60 and 80 percent, depending on the Product), it is crucial to acknowledge the role of other sources of observations - the complementary and sometimes, substituting role they play in the observations. Whether it is the weather stations on land that not only acquire data, but validate data from satellites; the hurricane hunter aircrafts that fly into the eye of the storm to collect real-time observations; the weather balloons launched everyday to collect observations; the buoys (and more recently drones) deployed on oceans for marine observations.
Weather From Space
The WMO’s Space Program, leading global coordinated efforts for acquiring weather observations from space, consists of a constellation of weather satellites from a number of countries, operating together with a shared vision. The agency, which is the weather arm of the United Nations, is also responsible for drafting long-term plans and identifying key gaps that need to be filled to guarantee and continuously improve global forecasting.
An Overview of Spaceborne Weather Sensors
While there are various ways to summarise the “state of weather observations from space”, I chose the easiest option - to provide an overview of the sensors and examples of such sensors in orbit, from both public and private sector initiatives.
In reality, the more effective way is to look at this, is by variable, in order to identify the current and upcoming gaps for each of them, history of observations and to understand the evolving role of the private sector (see WMO’s OSCAR for more).
- Imagers are cameras that are traditionally mounted on satellites that are in geostationary orbits, allowing them to capture images of the same point on the planet - this is what we all see during our weather forecasts on TV. Data from several commercial Earth imaging satellites - that offer high-resolution and high-frequency data - are increasingly becoming prominent for monitoring weather events. Lightning imagers, integrated into some weather satellites capture the location and intensity of lightning flashes - whether it is cloud-to-cloud or cloud-to-ground.
- Sounders acquire data through soundings of the atmosphere and depending on the wavelength (infrared, ultraviolet, microwave or radio) provide various profiles of key variables such as temperature, pressure and water vapour. Microwave sounders, specifically, have been known to provide the highest impact per observation, due to their ability to be efficient in clear and cloudy conditions - while also not being as expensive as radars. Companies, such as Tomorrow.io, Spire and Orbital Micro Systems are pursuing plans to launch microwave sounders to complement observations from the governmental satellites. Radio occultation sounders acquire data from signals originating from navigation satellites that are refracted by the atmosphere, leading to key weather observations. This is a segment that has seen increasing private sector involvement (Spire, GeoOptics, PlanetiQ), and the most commercial growth.
- Radars are crucial for weather observations due to their ability to monitor all day in any weather conditions. However, radar satellites, whether they are cloud & precipitation radars to monitor rainfall, altimeters to map ocean surfaces or scatterometers to monitor sea surface winds, tend to be expensive (on average, costing close to a $1 billion per mission). However, the recent trend of miniaturisation of satellites seen in the imaging segment - that has led to an explosion of commercial SAR providers - could likely happen in weather companies, the first of which will be Tomorrow.io with precipitation radars.
- Lidars are used to monitor a number of variables depending on their configuration - ice sheets from lidar altimeters (ICESat-2)or wind profiles from doppler lidars (Aeolus). However, due to the complexity of the instruments and the associated risk, this has been a largely public sector endeavour. As seen from the Aeolus mission, the social, economic and environmental impact of lidars for weather forecasting can be huge.
- There are several other instruments have been launched across a variety of missions to acquire observations for many crucial variables - from the radiation emitted from the planet to the amount of solar energy reaching the planet.
The Gaps in Weather Observations
As I mentioned above, this is a tricky subject - the WMO maintains a database of gaps across missions and variables. While this is a scientific analysis, and I am no way qualified to deem something as an official gap in weather observation, two of the most obvious gaps I have come across are: in weather stations and radars.
- Weather Station Coverage: While the reality is more complex than showing a map of the the weather stations (state of the instruments and the quality of observation also need to be considered), the map below helps make a point. While satellites and other sources may help cover the gaps, on ground weather stations are crucial for calibrating and validating the data acquired from satellites.
- Weather Radars: Radars are crucial for continuous monitoring, specifically for monitoring rainfall and developing flood maps, but radars are not omnipresent, and even if they are their quality of observations impact the forecasts. From a precipitation standpoint, there is currently only one radar in space to collect information on rainfall and predict storm intensity (NASA GPM mission) - this is one of the reasons NOAA’s Hurricane Hunters are sent into the hurricanes to collect real-time data for tracking hurricane track and intensity, as most other sensors on orbit cannot see through the clouds.
Bottomline: We are increasingly heading towards weather inequality - with extreme weather affecting parts of the world that have contributed to the least global emissions, while also suffering from the highest gaps in weather observations.
The Role of the Private Sector
Although the role of the private sector was largely focused on improving the weather Products and delivering Insights, its involvement in the Observations segment is not recent - several companies have attempted to build commercial weather satellite systems, but failed due to reasons mentioned below. While the miniaturisation of satellites and reduction in launch costs have led to increasing sales of commercially acquired weather observations in the past few years (see NOAA’s and EUMETSAT’s contracts for radio occultation data), the future of the commercial (spaceborne) weather data industry lies in the future of a piece of regulation - the WMO Article 40.
WMO Resolution 40 for Data Sharing
Weather has always been a considered a ‘public good’. However, as the commercial weather sector grew into prominence in the last few decades, the WMO introduce Resolution 40 focused on open data sharing to continue the global sharing of weather information, given that majority of countries in the world did not have the means to launch their own satellites or setup their observation systems. Therefore, commercial weather data companies (including Spire and GeoOptics who received the contracts from NOAA and EUMETSAT) had to follow the resolution while selling data to public agencies i.e. any data acquired by commercial companies that is received by the agencies would be shared to the other meteorological agencies globally.
The total addressable market for commercial weather data is, thus, limited to the commercial data procurement budgets of only few (not all) meteorological agencies - therefore preventing the growth of the private weather sector.
My Pesky Points Of View: Public-Private Partnerships, Defense Customers and Vertically Integrated Strategies
Public-Private Partnerships
Commercial initiatives in weather are evolving rapidly across the operating stack - therefore, it is not unlikely that, sometime in the future, there will be a commercial company that will have a better product (i.e. forecast) compared to the public agencies. There will be agencies that can afford to purchase data or products from these companies, while others might not be able to. Therefore, I believe, weather should always remain a public good - in other words, I would never want anyone to pay to receive the best weather forecast (like some will pay $8 for Twitter Blue!).
However, it must be acknowledged that there are some gaps in observations that can be filled in by the commercial Earth observation companies - the growing role of the private sector can already be felt in the Earth imaging segment (see ESA EarthNet and NASA CSDA initiatives). The sweet spot, I believe, lies in the ability to form mutually exclusive public-private partnerships. This might potentially happen through:
- Coordinated planning of missions and technology advancements with the private sector, based on best practices from other segments of the space industry (launch, Earth imaging and communications).
- Anchor tenancy contracts for commercial weather satellite missions from national weather agencies that not only guarantee the viability of the business plans of the companies, but also the continuity of data acquisition for agencies.
What I am certain that we do not want is the public and the private sectors going in two different directions - that would, I believe, be disastrous - especially for a world seeing more extreme weather events, needing more weather information.
I specifically don’t want to have a repeat of the following situation - where one city/country paid a commercial entity for weather services and was not impacted, while the other relatively poor neighbouring city/country, which did not pay for it, was heavily impacted.
We told [Union Pacific] that a tornado was heading to a spot; two trains stopped two miles apart, they watched the tornado go in between them. Unfortunately, it went into a town that didn’t have our service and a couple dozen people were killed. But the railroads didn’t lose anything.” - An executive from a commercial weather company. Source: The Guardian / HBO
Defence Customers
The role of the defence customers is also not to be understated - given the importance and impact of weather on its daily activities, particularly in case of conflicts. With the US Defence Meteorological Satellite Program coming to an end, and the Electro-Optical Infrared Weather System being planned, the military is already funding innovation in weather through specific programs (see US Air Force contract to Tomorrow.io and TCOM subcontract to Spire).
It is highly likely that, in the short-term, the future of commercial weather data from space will be dependent on the defence industry - just like how Earth imaging companies had the defence and intelligence entities as their anchor customers. What works in the favour of the commercial weather companies is that unlike Earth imaging, the use cases and the demand for weather products and insights is already well laid out, understood and urgent (given the context of climate change).
Vertically Integrated Strategies
For commercial companies launching a variety of sensors on satellites and acquiring a diversity of useful weather observations, the total addressable market might be limited due to the data sharing policies between meteorological agencies, but where, I believe, there will be a growing demand is from the various industries on the planet - after all, weather impacts literally every single individual, business and government.
But given the lack of in-house expertise in meteorology in these companies, they are more likely to consume weather Products, and increasingly more weather Insights - essentially companies that can not only provide accurate weather information but contextualise that into actionable intelligence - applied to that specific use case. Some industries such as insurance have started to heavily invest in building weather products, relevant to their use cases.
Therefore, I believe the biggest market for commercial weather companies is not the acquisition and dissemination of data, but rather its transformation into intelligence. As hurricanes such as Ian impact the developed world (losses estimated in tens of billions of $), both governments and businesses are more likely to look for products and services that can help minimise their losses and help them prepare better.
Some Final Thoughts
This has been a long piece. But, I want to add some closing notes - if not, I might not go to sleep peacefully. While some of these points below can even be in conflict with whatever that has been said above, in terms of how the industry is set up or the overall trends, I believe, these are some important learnings I have gained over the years, discussing with several prominent experts.
- It is not all about acquiring more weather observations from space to fill the gaps, in fact, as an EUMETSAT official mentioned on my podcast (see below), more data might not mean better forecasts. Weather is still a scientific endeavour, and there are no shortcuts to achieving a higher quality forecast (i.e. learning to respect the nature while understanding than attempting to control it).
- The value of weather - just like any technology - lies in its ability to make a positive impact and change someone’s world for the better. So, if the focus is solely on acquiring the most amount of observations and achieving the best quality forecast, without having an impact (i.e. lives and economic losses minimised), what good does it achieve? Weather is not just an application on mobile phones that those of us who have smartphones use. It is an economic opportunity - a recent study put the conservative estimate of the benefits of weather at $162 billion across a number of industries such as agriculture, construction, transportation, energy and more (see figure below).
- Finally, weather is a multidisciplinary endeavour - as the following figure shows there is more to it than what meets the eye. As we look at developing new technologies to acquire data, develop models, create products and provide insights, we need to also focus on the impact and action (i.e. last-mile delivery), which might not be scientific or technological tasks at all - rather psychological.
Thanks for reading. Subscribe for free to receive new posts directly to your inbox!
Until next time,
Aravind
Founder, TerraWatch Space
