Introducing Super-Rapid Update Forecasting for Solar Farms

The edge of forecasting technology

Traditionally, weather forecasts have been made from big, heavy weather models that update infrequently, and are usually based on data that's at least 1 or 2 hours old by the time the model is finished running. These models are excellent for forecasting tomorrow's general weather, but for the next hour or two they are no better than looking out the window.

At Solcast, we helped solve this problem by updating our cloud, solar radiation and solar power forecasting every 5, 10 or 15 minutes, tracking individual clouds from satellite and forecasting their movement and development for the next 2 to 4 hours ahead. We do this satellite-based forecasting up to 18 times for each individual cloud larger than about 500-1000 metres across over every continent except Antarctica (that's a lot of clouds), but for some applications this still isn't enough!

The challenge is that some solar forecasting applications require precise details of individual cloud impacts for the present time to about 10 minutes ahead. Examples include the control of batteries and other firming or smoothing devices, and 5 minute power market dispatch. Renewable generation has become mainstream, which means growing up and acting more predictably. Without enabling technologies like better forecasting, renewables penetration starts to hit hard upper limits.

This is what solar power generation looks like on a day with cumulus clouds coming and going. The longer-lived of these ramps (about 5-15 minutes duration or longer) were already well predicted using our satellite methods, but the shorter jumps could cause surprises between satellite updates.

Latest generation satellites allowed us to master forecasts for say an hour or two ahead under most conditions, but we quickly realised (around mid 2017) that for these specialised short-range applications a totally different approach would be required. The problem is that these satellites only get us down to 5 or 10 minute updates, and even then there's some time to receive the data, check it and update the forecast. We got our processing and forecasting time down to around 1-2 minutes, which is great for most applications, but only where those first 10 minutes or so.

Realising we had a new problem to solve, we started looking closely at possible solutions. We knew that statistical forecasts (using live measured data to nudge the forecast up or down) were good at minimising average error statistics, but completely incapable of ever forecasting any ramp. This led us to start looking at sky camera technology, which made sense since sky cameras can take pictures in near real time - far more rapidly than a satellite that needs to scan huge areas.

Sky cameras can fix that problem, right?

Wrong, at least in a simplistic sense. Basically, what we found was that sky cameras are an important part of the solution, but they cannot be a whole solution on their own.

The first problem we found was a fundamental complexity and cost issue - many of those we spoke to were grappling with this complexity - they hadn't worked out how to make a decent forecast using the camera yet. Those who had seemed to be making heavy, pricey, complex camera units with attached computers to do the forecasting. An engineering PhD was generally required to install and configure the thing. Some brilliant research was being done in the university space (e.g. at UCSD), but there were no real products - only complex solutions. Unsurprisingly, we heard some horror stories from customers who had adopted these systems.

The next problem was the fragility issue - how do you make a good forecast when the camera is down due to a power or comms issue? And what do you do when the lens gets dirty? One of the more amusing issues is bird poo - the cameras need to be mounted high enough for a good view, and birds tend to land on them frequently. The combination of dirty lens and the visual flares around the sun can cause even well-designed algorithms to be shouting cloud even under completely clear skies.

Great view from up here

Looking up while looking down

Eventually, we came full-circle onto a solution. Cameras were fragile, and had complexity and cost issues - but there was clear potential for big gains since they could see even small clouds approaching. The satellite technology was slower but far more robust. It was clear that we would need to combine these two technologies to make an accurate and robust forecast. And we would need to make a product, not another overly complex hardware-centric solution. On the camera side, we partnered with an IOT company (announcement coming soon on this) to develop a low-cost, easy-to-deploy camera unit with modern cloud-based image processing, and forecasting technology that uses a combination of physical and machine learning approaches.

We call this new product Super Rapid Update, and it's available now as an extension to our Utility Scale Solar Forecasts, only via an Enterprise plan. Here's how we do it.

Only good for 10 minutes Aside from the sky camera picture, this product will only help you if accuracy and minute-by-minute cloud detail in the first 10 minutes (15 at the most) really matters. If not, our existing Utility Scale Solar Forecasts product will work fine for you.

Satellite tracking Included with the product is our standard Utility Scale Solar Forecasts, which map the solar farm (or microgrid) and start tracking its surrounding clouds via satellite. Forecasts are ready the same day, and the following day the data has pinpoint accuracy at the ~5 minute level of granularity. This means we're ready to start supervising and blending the SCADA and Camera signals.

Live SCADA feeds When we set up a new customer, this is the first thing that needs to be done. Our Solcast API has a simple, secure method for the sending of live SCADA information via the internet to our API. A good engineer or developer usually takes less than 2 hours to get this set up. This feed is a very important source of truth of what the solar farm is actually doing, and the history of the SCADA measurements allows us to perform our unique tuning technology - which uses physically-constrained machine learning to capture the unique local fingerprints of the site.

Low cost sky camera Every site gets a camera, multiple cameras on larger sites. The camera arrives in a box, and just needs to be mounted on a raised structure, like a shed or on a pole. The camera is installed with a smartphone app, and starts sending pictures up to the cloud, where the images are processed to make a forecast. Separating hardware from software allows the camera units to be smaller, cheaper and solar-powered - this means that algorithm development can iterate quickly, and broken units don't become expensive wrecks.

Camera-based cloud tracking

If the camera goes down, gets very dirty or a bird poops on it - no big deal. This is the kind of fragility we designed our system to handle. With the satellite data in place, we can detect the fault and reject the camera signal - while still producing a relatively accurate forecast that can still track larger individual clouds. We alert the customer to the issue - usually a wipe of the camera lens takes care of the problem.

If you are keen to learn more about Super Rapid Update, get in touch with us for a chat.

Note: The development and proving of this product has been supported by the ACT Government's REIF program, and ARENA.