Inputs and algorithms

How we estimate and use satellite, aerosol and weather sources to make our data

Overview: Global Coverage, Rapid Update data

Solcast’s global live and forecast data are produced using high-resolution (1-2km) imagery from five geostationary weather satellites and provide Global Coverage (all major continents except Antarctica).

We gather the latest imagery every 5-15 minutes (depending on the sampling rate of the given satellite), and ingest it into our cloud-based processing system hosted on Amazon Web Services. From there, we crunch over 600 million forecasts an hour, using a probabilistic ensemble, updating all of our forecast data with each new satellite image. We refer to this process as “Rapid Update” forecasting and we use this approach for each of our live and forecast solar data products. We make all of this data available in real-time through our ServiceStack API framework, with a variety of APIs specifically tailored to each data product. Our API also makes it possible for our customers to share their PV measurement data via one-off uploads or regular POST commands to the relevant API to enable machine learning technologies that dynamically improve forecast accuracy for your specific site(s).

What follows here is a high-level walkthrough of our inputs and methods, to assist you in better understanding how our technology operates.

Input Data: Weather Satellites and Numerical Weather Models

For our live and forecast data, we ingest new satellite imagery from five geostationary weather satellites every 5-15 minutes. These satellites are Meteosat-08, Meteosat-11, GOES-16, GOES-17 and Himawari-8.

Methodology.PNG

Altogether, this provides us with near-global coverage of our “Rapid Update” live and forecast data, where we are refreshing the data available in our APIs with each new satellite image.

Methodology.PNG

Creating Solar Irradiance Estimates

Each of these images are a snapshot of the Earth from space, and can be as fine as 500 meters in spatial resolution. When these are transmitted every five minutes (e.g. GOES-16) this generates nearly two terabytes (TB) of raw image data each month.

Solar Radiation Data in Four Key Steps

Solcast first processes each satellite raw image (1) through geo-coding and quality control algorithms in a matter of a few seconds before passing them onto our albedo models.

Satellite Image

1. Satellite Image

Solcast first processes each satellite raw image (1) through geo-coding and quality control algorithms in a matter of a few seconds before passing them onto our albedo models.

Background (Albedo)

2. Background (Albedo)

In step (2), we use the latest satellite image in conjunction with a collection of archived images for the same location over the past month. Using sophisticated statistical models, we generate a representation of the ‘background’, more commonly referred to as the ‘albedo’, which allows us to separate cloud cover from snow, bright sand, ocean glare or other aspects of the imagery that could be wrongly interpreted as cloud.

Cloud Opacity

3. Cloud Opacity

The next step (3), is the decomposition of the current satellite image into cloudy and cloud-free regions. For any regions determined to contain cloud cover, we then apply a proprietary 3D cloud modelling approach that vertically decomposes cloud cover into layers and characterise the thickness of each layer to sunlight. We term the total aggregate thickness, ‘cloud opacity’.

Solar Radiation Estimate

4. Solar Radiation Estimate

In the fourth step, we apply a modified version of the REST2 clear-sky radiation model, that allows us to use the latest global aerosol (dust, salt, smoke, etc.) and water vapor content to generate precise estimates of the solar radiation available to cloud-free regions. For areas with cloud cover, we use the cloud opacity (3) to produce estimates of the total amount of solar irradiance reaching the Earth’s surface. Throughout Step 4, we use a combination of peer-reviewed methods, industry-standard models, and in-house Solcast algorithms, which are utilised according to the following flow diagram. Notably, our separation model was developed by our CTO Dr. Nick Engerer as a part of his PhD work, and was globally recognised as the top performing model for this purposes globally in 2015.

Methodology.PNG

Numerical Weather Model Data

Solcast utilises Numerical Weather Prediction (NWP) data from several models including GFS deterministic, GFS ensemble, CMC ensemble, ECMWF deterministic and ECMWF ensemble. We use several forecast parameters including temperature, humidity, and winds at multiple atmospheric levels; cloud cover and solar irradiance; snow depth; water vapour; and aerosol optical depth. Each of these inputs assist with the four above steps in processing the satellite imagery in order to aid our interpretation between clouds and background/albedo in static satellite imagery, to produce clear-sky (cloud-free) solar radiation estimates, and adjust them for the impacts of aerosols.

Forecasting Cloud Cover

Once cloud opacity fields have been determined and allocated to the appropriate vertical layers, Solcast then utilises a combination of NWP wind fields and computer vision algorithms to create predictions of where cloud opacity will move next, out to a horizon of 4-hours ahead. This is referred to as our ‘nowcasting’ horizon.

For the nowcasting horizon, cloud opacity field is advected forward in time by an 18 member ensemble including combinations of NWP wind forecasts and computer vision vector fields derived from the preceding hour of satellite imagery. All 18 ensemble members are combined to produce probabilistic forecast data, including a median (P50) forecast, and a corresponding 10th and 90th percentile forecast (available as P10 and P90 in our API forecast data). The resulting forecast covers a physically realistic range of scenarios, so that rapid fluctuations in cloud, are faithfully and sharply portrayed in the probabilistic forecast, rather than smoothed out as in some statistical post-processing methods.

For time horizons greater than 4-hours, an ensemble of five numerical weather models are used to extend the forecasts out to 7 days. On the day-ahead horizon, we include machine learning based bias corrections that are applied based on the observed cloud opacities from satellite imagery.

Our cloud opacity forecasts are then used to create the GHI, DNI and DHI (global, direct, diffuse, respectively) solar radiation predictions as discussed above. These are made available directly in our Solar Radiation Data products, but for our PV Power products (Utility Scale, Rooftop, Grid Aggregations), these data are passed onto our PV Power Modelling algorithms.

PV Power Modelling, Measurements and Tuning

Solcast employs a proprietary PV power model, based on a quadratic PV radiation to power conversion algorithm that is uniquely suited for machine learning applications. We have selected this approach, as our PV power products are designed to benefit from user PV power measurement data, which are uniquely specified for each data product.

In the Utility Scale Solar Forecast and Rooftop Solar data products, power forecasts are generated through customer provided metadata (including capacity, orientation and tracking information). Once provided, our APIs will immediately begin generating live and forecast PV power estimates for the site. Ideally, these estimates should be considered to be a preliminary best-guess of PV system performance. We recommend that all users provide PV Power measurement data in order to enable our machine learning based PV Tuning technology (Read on below) to further refine the relationship between satellite-derived solar radiation and power modelling. This allows us to generate a PV power model unique to your site(s), which captures degradation, shading and soiling impacts.

Our Grid Aggregations data product produces behind the meter solar and utility scale solar power predictions at locality-level (i.e. suburb, township) clusters, before aggregating total power outputs according to the end user specified regional or network based groupings. Here, the PV power modelling utilises the same generalised quadratic PV power model, and can be locally tuned wherever PV measurement or estimated actuals data are available.

PV Tuning Technology

Solar installations are complicated – they are often split amongst several arrays which are impacted by variations in topography, orientation, shade, dirt and degradation. Solcast’s tuning technology takes these factors, plus power output, solar irradiance and weather data, to learn the real characteristics of your PV plant.

Better forecasts are possible with tuning because it:

  • Captures the impacts of shading on your system, including vegetation, topography and surrounding infrastructure
  • Detects the orientation of your PV system (azimuth and tilt angles)
  • Senses the degradation of your PV system and assigns it a loss factor
  • Individualises the way your PV system responds to cloud cover
  • Uses these learned parameters to provide you with an improved solar forecast specific to that asset
  • Capture Shading Impacts
  • Detect Degradation, Estimate Tilt
  • Identify Azimuth Angles
  • Individualise Cloud Impacts
Actual
Untuned
Line chart

Using detailed PV plant specifications makes sense when you are modelling a system before commissioning or during early operations. However, Solcast’s experience in solar forecasting shows that better results are obtained when physical assumptions are reduced and actual real PV power measurements are used.

PV Tuning is offered as an option with Utility Scale forecast data (includes manual supervision of initial tuning and capacity to deal with availability and curtailment data). A basic version of PV Tuning comes included with Rooftop Solar forecasts too. Both products can be started without PV tuning, and added later if you decide to improve your forecasting accuracy.

Test now, free

Get free historical data credits when you sign up, so you can try our data for yourself. You can download data automatically within around 1 to 9 minutes of submitting your request.