Looks interesting!

**PowerWeather** · August 23, 2017, 2:28am

I’m getting better results now that I have written a routine to automatically download and combine the Solcast estimations for my two arrays at 5:30am. I then apply a shading profile, which changes daily for half of the year around the winter solstice, to account for known losses that my system experiences due to shading from a neighbours tree.

After smoothing my actual readings to last 30 minute averages to match the Solcast data set interval, I am now getting very high goodness of fit in mostly clear / clear conditions eg. 21 Aug 17

Cloudy / rainy days are still a bit hit and miss however, particularly in the early morning when the Solcast estimate is at its freshest eg. 15 Aug 17 and 20 Aug 17

Having said that, there are days when the opposite is true ie. afternoon forecasts prove less accurate than morning eg. 17 Aug 17

Here are the graphs of the subject dates:

My next step is to revise Solcast estimates hourly throughout the day and see whether that improves goodness of fit on cloudy / rainy days, particularly later in the day.

Thanks again for giving me the opportunity to evaluate your generation forecast data!

**darren** · August 28, 2017, 3:14am

Appreciate you sharing your graphs. Great to see

You will definitely see an improvement if you update the forecasts hourly. The satellite based forecasts that are updated for Australia/Asia/NZ every 10 minutes as new imagery comes in, however it’s only for the near future of ~4 hours. Beyond ~4 hours results are blended with our longer term forecasts based off numerical weather models and some machine learning which are only updated daily (a bit after 5am AEST usually). For the absolute best result, forecasting every 10 minutes will keep your forecasts up to date with the very latest we have available, but generally every hour will also give you much better results. I can see on your 15th/20th day examples the forecast starts to deviate at ~7:30am which makes sense based on a 5:30am forecast time in complex cloudy situations. The frequent short term forecasts (nowcasts) are much better situated to give you a more accurate forecast as our cloud tracking can much more accurately predict where the cloud will be and how they will effect PV output.

I’d love to see the results when your forecasts are more frequent (hourly or every 10 mins) given you are accounting for shading yourself.

**PowerWeather** · August 28, 2017, 4:02am

I’ve implemented updates to Solcast forecasts every 30 mins during generation hours in my App and am already seeing much greater accuracy of current versus predicted generation. eg. Screen shot just now of my app shows current instant generation amount (see red arrow) with font colour dependent on how accurate the Solcast estimate for this five minute window (interpolated) is. A font colour of MediumSeaGreen, which it currently is and has been since implementing this feature, indicates the estimate is within +/- 5%.

I’m saving the shading-compensated and combined N & E array 30 minute data and will graph up accuracy versus time out from estimate when we get a rainy day again. Sunday is looking promising.

**darren** · August 28, 2017, 4:09am

That’s fantastic! Within +/-5% matches our validation work closely for within the first hour after a satellite update.

Love seeing your visualization work that is incorporating our data. Feel free to post any interesting visuals here, it’s always interesting to see visualization where our data can add value!

Thanks again for sharing

**PowerWeather** · September 18, 2017, 8:55am

I have now integrated the solcast forecast (updated every 30 minutes and corrected for shading) into my existing 24 hr rotating weather forecast information bar such that it overlays daylight hours with forecast generation estimates (edit: now colour coded to indicate deviation from ideal insolation at each time interval). It is the yellow line on the following graphic and is scaled to show maximum possible generation (3096W) at the top of the bar down to 0W at the bottom. The graphic advances one pixel right every 6 minutes.

**PowerWeather** · September 20, 2017, 8:14am

I have a question regarding API call csv data format. In the last week, the PeriodEnd parameter has changed from being in one format, ie. 2017-09-20T11:30:00Z, to being in three potential formats, namely:

The previous format eg. 2017-09-20T11:30:00Z (most records)
A different format eg. 2017-09-20T11:00:00+00:00 (usually the first 8 records)
An abbreviated format with no time information at all eg. 2017-09-21 (all midnight records)

This caused some issues with my reading of the data until I adapted it to account for all three formats possible but it’s a bit ugly. I have checked the JSON version of the same data sets and there is only one format, as it should be.

I may end up switching to JSON format anyway but I was just wondering whether the multi-format date/time data field change to CSV was intentional

**darren** · October 4, 2017, 10:42pm

That’s no good regarding format change for the CSV endpoints. Thanks for letting us know about the issue and sorry for the delayed response.

I’ll look into the problem from our side and post back here when I understand what has happened.

I know about the missing timezone offset for midnight UTC, and it is on our todo list, but all records should be without timezine offset as all dates should be in UTC.

Could you give me an example of a request with all 3 different formats so I can reproduce?

**PowerWeather** · October 4, 2017, 11:33pm

Sure. Below is an extract of the latest 30 min Solcast forecast csv file I have for my North facing array showing each of the 3 data formats. Line 9 and onwards are in the standard format. Lines 1-2 and 4-8 show the second format (Z dropped and with the extra +00:00). Line 3 shows the date only format. The API call I used was: https://api.solcast.com.au/pv_power/forecasts?longitude=149.213&latitude=-35.384&capacity=3100&azimuth=7&install_date=20150318&api_key=XXXXXXXXXXXXXX&format=csv

PeriodEnd,Period,PvEstimate
2017-10-04T23:00:00+00:00,PT30M,1679.63711162445
2017-10-04T23:30:00+00:00,PT30M,1749.67692615641
2017-10-05,PT30M,1763.28031333703
2017-10-05T00:30:00+00:00,PT30M,1882.10264424809
2017-10-05T01:00:00+00:00,PT30M,1621.47695667698
2017-10-05T01:30:00+00:00,PT30M,1822.57680296399
2017-10-05T02:00:00+00:00,PT30M,2022.02755621531
2017-10-05T02:30:00+00:00,PT30M,2090.16718724436
2017-10-05T03:00:00Z,PT30M,2167.9624822652
2017-10-05T03:30:00Z,PT30M,2022.38465598681
2017-10-05T04:00:00Z,PT30M,1786.75722920438
2017-10-05T04:30:00Z,PT30M,1531.05067625976
2017-10-05T05:00:00Z,PT30M,1272.47597804145
2017-10-05T05:30:00Z,PT30M,1052.31213981049
2017-10-05T06:00:00Z,PT30M,846.543370782682
2017-10-05T06:30:00Z,PT30M,629.5515419379
2017-10-05T07:00:00Z,PT30M,406.505091365806
2017-10-05T07:30:00Z,PT30M,187.223734840439
2017-10-05T08:00:00Z,PT30M,38.4564473592096
…

**PowerWeather** · October 4, 2017, 11:57pm

BTW, it looks like we have an interesting day ahead in Canberra on the solar forecasting front. I now overlay both solcast and weatherzone-derived solar forecasts on my rolling 24 hour forecast timeline and you can see the slight differences below. I have noticed over the last few weeks that the solcast forecast only seems to get appropriately pessimistic within a few hours of forecast worse conditions whereas the forecast I derive from weatherzone data gets onto it a lot earlier. While the Solcast forecast generally ends up being more accurate when produced within a few hours of the conditions actually hitting, further out (6-24 hours) Solcast tends to be less accurate and overly optimistic than what I can forecast with weatherzone.

Solcast:

Weatherzone-derived:

**darren** · October 5, 2017, 12:21am

Thanks for the example, I can see the issue and it is indeed only with our CSV endpoint (and JSV). This seems to have been caused by an internal change in how we store and retrieve short term forecasts.

Fix should be deployed later today. Would a change to the following interrupt how you are parsing dates? This does truncate the timezone completely unlike the JSON endpoints but is still compatible with ISO8601.

2017-10-04T13:30:00Z
2017-10-04T14:00:00Z
2017-10-04T14:30:00Z
2017-10-04T15:00:00Z
2017-10-04T15:30:00Z
2017-10-04T16:00:00Z
2017-10-04T16:30:00Z
2017-10-05T04:00:00Z
2017-10-05T04:30:00Z
2017-10-05T05:00:00Z
2017-10-05T05:30:00Z
2017-10-05T06:00:00Z
2017-10-05T06:30:00Z
2017-10-05T07:00:00Z
2017-10-05T07:30:00Z

**darren** · October 5, 2017, 12:24am

Thanks for this, nice find showing differences. I’ll pass this onto James and Nick to dig into. Work to improve our forecasts, both longer term and short are always ongoing, good to have an example with your PV output data.

**PowerWeather** · October 5, 2017, 1:05am

A change like that will be fine. I use PHP’s strtotime function and it can easily handle this format. It is better to be parsing any single format rather than 3 though!

**PowerWeather** · October 5, 2017, 1:06am

I’ve had better examples, but just haven’t saved them. I’ll keep an eye out for more.

**darren** · October 5, 2017, 1:13am

Ok great. Thanks again for reporting the issue. I’m adding some tests to better catch CSV specific formatting problems to avoid this getting to our production systems in the future.

Currently JSON is primarily used for integration testing, however CSV is heavily used, more so than I initially thought but users so before the beta stage ends for the API, there will be more work on CSV on the way.

Just as a heads up, we are aware of the naming inconsistency of row headers like cloud_opacity vs Azimuth. There will be a fix to change this to match each other. There will probably be an email to recent CSV endpoint users with a few weeks warning as I known it will be a breaking change for some users. This won’t be for a while yet though.

**PowerWeather** · October 5, 2017, 1:50am

You can get a better idea of the longer term optimism of Solcast forecasts by looking at the latest forecast which shows forecast generation for 2 hours out, around the 3pm time mark period, (each tick mark represents 3 hours and 3pm is the next one) has now dropped to around about what the weatherzone data has been telling me for much longer.

Solcast:

**darren** · October 5, 2017, 1:56am

What data/components from weatherzone are you using to forecast PV?

**PowerWeather** · October 5, 2017, 2:09am

From their 48 hour forecast page for a given location, I use cloud cover for obvious reasons, rain chance to give a rough indication of how dense the cloud cover is, and temperature to alter panel output expectations due to temperature coefficient. This data is in 3 hour intervals, which is not ideal, but once weighted and interpolated to 5 minute intervals it is proving to be a reasonable estimate of generation behaviour over the next 24 hours.

**darren** · October 5, 2017, 2:51am

Interesting approach! If you’re able, it would be great to see some of the data to compare on an interesting/cloudy day. Being able to reproduce what you are seeing would enable us to dig further into it.

**darren** · October 5, 2017, 3:51am

Regarding this data change, I’ve removed the problematic +0000 entries at the start. The UTC midnight times are still not ideal but a change to fix these will be coming down in the near future. If you write your code in way to check if UTC midnight is missing hours info (even just a length check would work), when the fix comes through, nothing on the client side will change.

Eg, this

2017-10-06

Will become

2017-10-06T00:00:00Z

**PowerWeather** · October 5, 2017, 9:32am

My code has been checking to see what the 11th character is. If is a comma then it 00:00:00 Z. Otherwise I just take the next 8 characters, ignore the rest and tack on a Z for the strtotime function. As previously mentioned, its a kludge that has worked well and will still work when you make the fixes you have mentioned, so all is good.