Solcast’s forecast accuracy proven in major competitive trial

Solcast (a DNV company) has achieved an industry-leading result in a major competitive forecast accuracy trial run by EPRI.

EPRI trial composition and results

The trial, conducted by independent US nonprofit Electric Power Research Institute (EPRI), was the world’s most comprehensive open solar production forecasting trial of the past five years. The trial was run in partnership with a top-3 US utility, spanning 12 weeks, 5 large solar plants, and 5 different time horizons from hour ahead to day ahead. Entry to the trial was open to the industry, and 9 vendors participated. The trial, spanning July to September 2023, was operated using the DOE-funded open source evaluation software “Solar Forecast Arbiter” (SFA). Every hour, entrants submitted their forecasts of solar production for each plant to the SFA API. Using SFA, EPRI calculated forecast errors, and also forecast submission reliability.

Of the 9 vendors participating, Solcast had the lowest forecast error overall. Solcast also had the lowest error in 4 of the 5 forecast horizons, and was the only provider to achieve <11% normalised Mean Absolute Error (nMAE) in any time horizon. 

Read more about the trial methodology below.

Why this result matters

Prospective and existing customers perform accuracy evaluations regularly, as a means of testing the claims of data vendors prior to subscription. Successful evaluations are one of the reasons why hundreds of new customers choose Solcast each year. Typically, these evaluations are performed privately, using the smallest scope that can reliably inform a business decision.

This trial was rare because of the open invitation, the duration, the broad scope of forecast horizons assessed, and the fact that entrants could see their results against the (anonymised) competitors every day.

This result affirms why Solcast data is used in the management of over 350GW of assets globally, and its position as the world's largest solar data provider. 

Accuracy evaluations are a huge effort to perform well. Through 7 years of supporting customers through evaluations, the Solcast team has learnt many best practices and pitfalls to avoid. If you’re looking at conducting your own accuracy evaluation, check out our guides on evaluating forecast, historical and PV Power model accuracy.

One of the first steps we recommend is to review Solcast’s own published accuracy validation results for sites spread around the world. The Solcast forecasts and the measured actuals for most relevant nearby sites can be sent to customers to rapidly aid their analysis. 

Trial methodology

Interested in how this trial operated? We can share the following details here, and this will be followed by EPRI with forthcoming detailed papers for its members and the public.

The five sites selected by EPRI and the utility for the trial spanned a range of capacities and geometric configurations, including one site with a combination of fixed and tracking, giving a broad assessment of forecast performance. One of the sites was not used by EPRI in final aggregated metrics due unforeseen changes to the site configuration.

As part of the trial, changes in operational conditions including assets being temporarily offline were flagged, and in some cases were excluded from the analysis, because the forecasting objective was to forecast unconstrained generation. 

The trial parameters heavily penalised missed forecasts, so all participants were incentivised to ensure high reliability. Updated forecasts were allowed, so that early initial submission of forecasts, supplemented with frequent updates until the final deadline, could be used by entrants as a strategy to maintain reliability. These choices were made to replicate the real world requirements of an operational forecast - so that the results reflected what utilities and asset operators actually require from a forecast provider

The trial assessed five different combinations of forecast horizon and temporal resolution. 

• HA1-5min: 5 minute resolution, issued hourly 60 minutes before the start of the hour
• HA1-15min: Same as HA1-5min but 15 minute temporal resolution
• HA2: Hourly resolution forecasts, issued hourly 150 minutes before the start of the hour
• HA4: Hourly resolution forecasts, issued hourly 270 minutes before the start of the hour
• DA: Hourly resolution, issued once per day by 05:00am for the 24 hours of the next day

HA1 forecasts had higher error than HA2 due to their finer temporal resolutions.

‍