Inputs and algorithms

How we estimate and use satellite, aerosol and weather sources to make our data

Overview: Global Coverage, Rapid Update data

Solcast’s global live and forecast data are produced using high-resolution (1-2km) imagery from five geostationary weather satellites and provide Global Coverage (all major continents except Antarctica).

We gather the latest imagery every 5-15 minutes (depending on the sampling rate of the given satellite), and ingest it into our cloud-based processing system hosted on Amazon Web Services. From there, we crunch over 600 million forecasts an hour, using a probabilistic ensemble, updating all of our forecast data with each new satellite image. We refer to this process as “Rapid Update” forecasting and we use this approach for each of our live and forecast solar data products. We make all of this data available in real-time through our ServiceStack API framework, with a variety of APIs specifically tailored to each data product.

What follows here is a high-level walkthrough of our inputs and methods, to assist you in better understanding how our technology operates.

Input Data: Weather Satellites and Numerical Weather Models

For our live and forecast data, we ingest new satellite imagery from five geostationary weather satellites every 5-15 minutes. These satellites are Meteosat-08, Meteosat-11, GOES-16, GOES-17 and Himawari-8.

Solcast Satellites

Altogether, this provides us with near-global coverage of our “Rapid Update” live and forecast data, where we are refreshing the data available in our APIs with each new satellite image.

Solcast Rapid Update Coverage

Creating Solar Irradiance Estimates

Each of these images are a snapshot of the Earth from space, and can be as fine as 500 meters in spatial resolution. When these are transmitted every five minutes (e.g. GOES-16) this generates nearly two terabytes (TB) of raw image data each month.

Solar Radiation Data in Four Key Steps

Solcast first processes each satellite raw image (1) through geo-coding and quality control algorithms in a matter of a few seconds before passing them onto our albedo models.

Satellite Image

1. Satellite Image

Solcast first processes each satellite raw image (1) through geo-coding and quality control algorithms in a matter of a few seconds before passing them onto our albedo models.

Background (Albedo)

2. Background (Albedo)

In step (2), we use the latest satellite image in conjunction with a collection of archived images for the same location over the past month. Using sophisticated statistical models, we generate a representation of the ‘background’, more commonly referred to as the ‘albedo’, which allows us to separate cloud cover from snow, bright sand, ocean glare or other aspects of the imagery that could be wrongly interpreted as cloud.

Cloud Opacity

3. Cloud Opacity

The next step (3), is the decomposition of the current satellite image into cloudy and cloud-free regions. For any regions determined to contain cloud cover, we then apply a proprietary 3D cloud modelling approach that vertically decomposes cloud cover into layers and characterise the thickness of each layer to sunlight. We term the total aggregate thickness, ‘cloud opacity’.

Solar Radiation Estimate

4. Solar Radiation Estimate

In the fourth step, we apply a modified version of the REST2 clear-sky radiation model, that allows us to use the latest global aerosol (dust, salt, smoke, etc.) and water vapor content to generate precise estimates of the solar radiation available to cloud-free regions. For areas with cloud cover, we use the cloud opacity (3) to produce estimates of the total amount of solar irradiance reaching the Earth’s surface. Throughout Step 4, we use a combination of peer-reviewed methods, industry-standard models, and in-house Solcast algorithms, which are utilised according to the following flow diagram. Notably, our separation model was developed by our CTO Dr. Nick Engerer as a part of his PhD work, and was globally recognised as the top performing model for this purposes globally in 2015.

Methodology.PNG

Numerical Weather Model Data

Solcast utilises Numerical Weather Prediction (NWP) data from several models including GFS deterministic, GFS ensemble, CMC ensemble, ECMWF deterministic and ECMWF ensemble. We use several forecast parameters including temperature, humidity, and winds at multiple atmospheric levels; cloud cover and solar irradiance; snow depth; water vapour; and aerosol optical depth. Each of these inputs assist with the four above steps in processing the satellite imagery in order to aid our interpretation between clouds and background/albedo in static satellite imagery, to produce clear-sky (cloud-free) solar radiation estimates, and adjust them for the impacts of aerosols.

Forecasting Cloud Cover

Once cloud opacity fields have been determined and allocated to the appropriate vertical layers, Solcast then utilises a combination of NWP wind fields and computer vision algorithms to create predictions of where cloud opacity will move next, out to a horizon of 4-hours ahead. This is referred to as our ‘nowcasting’ horizon.

For the nowcasting horizon, cloud opacity field is advected forward in time by an 18 member ensemble including combinations of NWP wind forecasts and computer vision vector fields derived from the preceding hour of satellite imagery. All 18 ensemble members are combined to produce probabilistic forecast data, including a median (P50) forecast, and a corresponding 10th and 90th percentile forecast (available as P10 and P90 in our API forecast data). The resulting forecast covers a physically realistic range of scenarios, so that rapid fluctuations in cloud, are faithfully and sharply portrayed in the probabilistic forecast, rather than smoothed out as in some statistical post-processing methods.

For time horizons greater than 4-hours, an ensemble of five numerical weather models are used to extend the forecasts out to 7 days. On the day-ahead horizon, we include machine learning based bias corrections that are applied based on the observed cloud opacities from satellite imagery.

Our cloud opacity forecasts are then used to create the GHI, DNI and DHI (global, direct, diffuse, respectively) solar radiation predictions as discussed above. These are made available directly in our Solar Radiation Data products, but for our PV Power products (Utility Scale, Rooftop, Grid Aggregations), these data are passed onto our PV Power Modelling algorithms.

PV Power Modelling

Solcast employs two types of PV power models. For our Utility Scale product, we utilise the pvlib package, and enable users to add extensive detail to each of their solar farm sites to improve accuracy and account for local conditions (e.g. terrain shading). The World API and Rooftop Sites products employ our “Simple” PV power model, which is a proprietary approach based on a quadratic PV radiation to power conversion algorithm.

In the Utility Scale Solar Forecast and Rooftop Solar data products, power forecasts are generated through customer provided metadata (including capacity, orientation and tracking information). Once provided, our APIs will immediately begin generating live and forecast PV power estimates for the site.

Our Grid Aggregations data product produces behind the meter solar and utility scale solar power predictions at locality-level (i.e. suburb, township) clusters, before aggregating total power outputs according to the end user specified regional or network based groupings. Here, the PV power modelling utilises the same generalised quadratic PV power model, and can be locally tuned wherever PV measurement or estimated actuals data are available.

Test now, free

Access solar radiation and PV power live & forecast data via API (including CSV download) in our API Toolkit in just a few minutes, for free!