Introduction

There are many examples of how to use forecasting to save time and cost in a data science project, especially measuring simple or complex relationships among variables to accurately hit your target metrics over time and future-proof business decisions.     For example, electric usage and solar power generation forecasting in the energy industry always helped businesses make the right calls in a competitive environment to save operating costs and, most importantly, the environment.  Now, machine learning-driven forecasting and the availability of localized weather data are improving the accuracy of these predictions compared to the limitations of earlier statistical methods.

Note: Oracle Cloud Infrastructure Forecasting is currently in limited availability. To access Forecasting, goto Oracle Beta Programs page and click Oracle Data & AI Cloud Services Umbrella Beta Program.

In this blog, I will use the OCI forecasting service to demonstrate how to forecast solar energy generation with additional weather features. However, before we start the project, we need to understand that the efficacy of solar energy generation can be affected by various factors, including:

• Amount of sunlight: The amount of sunlight received at a particular location can affect the amount of solar energy panels produce.
• Panel orientation and tilt: The orientation and tilt of the solar panels can impact the amount of sunlight they receive and, therefore, their efficiency.
• Temperature: High temperatures can reduce the efficiency of solar panels, as they can cause the cells to overheat and produce less energy.
• Shading: Even partial solar panel shading can significantly reduce energy production, as shading can block incoming sunlight.
• Panel age and maintenance: The age and maintenance of solar panels can impact their efficiency, as older or poorly maintained solar panels may not perform as well as new, well-maintained ones.
• Weather conditions: Cloudy or rainy weather can significantly reduce the amount of sunlight reaching the solar panels, leading to decreased energy production.

Prepare your data

I collected the solar production data sample from a 10kw inverter with a data logger and 32 solar panels. The data logger collects data every 5min. Below is an example of the data collected:

I collected the localized weather from the local weather station, and you can find your local weather station using the longitude and latitude of your solar panel location from weather.com. Once you have your location weather station id, you can retrieve hourly or daily weather data using your local station id using the weather.com API. Below is an example of the hourly weather data collected near my solar panels.

The above samples are the raw data that has to be processed and formatted to the data set that the OCI Forecasting service can process.  I used a python script to consolidate the time series solar and weather data to the same time interval, in this case, hourly, and ensure the solar data has corresponding weather data using data/time.  Below is an example:

Process weather data

data = pd.read_csv('hourly-weather-jan2022.csv',index_col=3,parse_dates=True)
temp = data.loc[:,['metric.tempHigh']]
groupByHour= temp.groupby([temp.index.month,temp.index.day, temp.index.hour]).mean()
groupByHour.index.set_names(["month", "day", "hour"], inplace=True)
groupByHour['ds'] = groupByHour.index.to_series().apply(lambda x: '2021-{0}-{1} {2}:00:00'.format(*x))
groupByHour['ds']= pd.to_datetime(groupByHour['ds'],format='%Y-%m-%d %H:%M:%S')
weatherData = pd.DataFrame(groupByHour.to_records())
weatherData.drop(['month', 'day', 'hour'], axis=1, inplace=True)
weatherData['metric.tempHigh']= weatherData['metric.tempHigh'].map('{:.2f}'.format)
weatherData.rename(columns = {'metric.tempHigh':'average_temp'}, inplace = True)
weatherData.rename(columns = {'ds':'solar_log_date'}, inplace = True)
weatherData['uid']="1"
weatherData.to_csv('processed-weather.csv', index=False)

Process solar production data

path =r'solarlogs'
all_files = glob.glob("../" + path + "/*.csv")
li = []
dateparse = lambda x: datetime.strptime(x, '%d.%m.%Y %H:%M')
date_cols = [0]
for filename in all_files:
    df = pd.read_csv(filename, parse_dates=date_cols, date_parser = dateparse, header=[0,1])
    df['DateTime']= pd.to_datetime(df.iloc[:, 0], yearfirst=True)
    li.append(df)
df = pd.concat(li)
n = df.shape[1]
print("Number of columns is:", n)
df.sort_values(by='DateTime', inplace=True)
columns_to_keep = [x for x in range(df.shape[1]) if x in [33,34]]
df = df.iloc[:, columns_to_keep]
df = df.fillna(0)
df.set_axis(['y', 'ds'], axis=1, inplace=True)
df.index =  df['ds']
df = df.groupby([df['ds'].dt.month, df['ds'].dt.day, df['ds'].dt.hour]).sum()
df.index.set_names(["month", "day", "hour"], inplace=True)
df['solar_log_date']= df.index.to_series().apply(lambda x: '2021-{0}-{1} {2}:00:00'.format(*x))
df['solar_log_date'] = pd.to_datetime(df['solar_log_date'],format='%Y-%m-%d %H:%M:%S')
df.rename(columns = {'y':'solar_production'}, inplace = True)
solar_data = df[(df['solar_log_date'] >= '2021-02-04') & (df['solar_log_date'] < '2021-02-11')]
solar_data=pd.DataFrame(solar_data.to_records())
solar_data.drop(['month', 'day', 'hour'], axis=1, inplace=True)
solar_data['solar_production']= solar_data['solar_production'].map('{:.2f}'.format)
solar_data['uid']="1"
solar_data.to_csv('processed-solar-log.csv', index=False)

Once the data is processed, you will have the primary and additional datasets to upload to the OCI Object Storage for the OCI Forecasting project.  Primary and Additional Datasource columns should contain to 3 variables: a date, a time series Id, and a target variable. Below are the sample datasets:

Primary Dataset (Solar)

Additional Dataset (Weather)

Using OCI Forecasting

Now, you can create a project and forecast using the datasets. Below is the configuration of my forecast project:

Error metrics compare a forecasting model’s data points with the actual historical data points and use them to measure errors for forecasting models.  The forecasting service uses error metrics to compare the error in different models and chooses the best model that fits the actual values to predict future values.  A list of supported error measures are:

• RMSE: RMSE stands for Root Mean Squared Error, a commonly used metric to evaluate the performance of a predictive model.  It measures the square root of the average of the squared differences between the predicted and actual values.  RMSE is often used in regression analysis to compare the accuracy of different models or assess a model’s performance on new data.
• MAPE: MAPE stands for Mean Absolute Percentage Error, a commonly used metric to evaluate the accuracy of a predictive model.  It measures the average percentage difference between the predicted and actual values, dividing the absolute difference by the actual value.  MAPE is often used in forecasting and demand planning to assess the accuracy of a model’s predictions, especially when dealing with data that has different scales or units.
• MSE: MSE stands for Mean Squared Error, a commonly used metric to evaluate the performance of a predictive model.  It measures the average of the squared differences between the predicted and actual values.  MSE is often used in regression analysis to compare the accuracy of different models or assess a model’s performance on new data.  The smaller the MSE, the better the model makes accurate predictions.
• SMAPE: SMAPE stands for Symmetric Mean Absolute Percentage Error, a commonly used metric to evaluate the accuracy of a predictive model.  It measures the average percentage difference between the predicted and actual values, where the absolute difference is divided by the sum of the predicted and actual values.  Unlike MAPE, SMAPE is symmetric, which means it is not affected by the direction of the forecast error.  SMAPE is often used in time series forecasting to compare the accuracy of different models or to assess the performance of a model on new data.

OCI Forecasting provides several algorithms.  You can select more than one desired algorithm or let the service choose the algorithms automatically.  Below is a list of supported algorithms:
 

Statistical Algorithms:

• NAIVE: Naïve 
• SNAIVE: Seasonal Naïve 
• SMA: Single Moving Average 
• DMA: Double Moving Average 
• HWSA: Holt – Winter’s Seasonal Additive 
• HWSADAMPED: Holt – Winter’s Seasonal Additive method with a damped trend 
• HWSM: Holt – Winter’s Seasonal Multiplicative 
• HWSMDAMPED: Holt – Winter’s Seasonal Multiplicative method with a damped trend 
• SES: Simple Exponential Smoothing 
• DES: Double Exponential Smoothing 
• DESDAMPED: Double Exponential Smoothing method with a damped trend 
• SA: Seasonal Additive 
• SM: Seasonal Multiplicative 
• UAM: Ensemble Arithmetic Mean 
• UHM: Ensemble Harmonic Mean 
• ARIMA: Autoregressive Integrated Moving Average (ARIMA) Algorithm 

Machine Learning Algorithms:

• PROPHET: Local Bayesian structural time series model from Prophet open source 
• EFE: Endogenous Feature Engineering model

Deep Learning Algorithms:

• APOLLONET: A proprietary deep learning model
• PROBRNN: Probabilistic Recurrent Neural Network model 

The calculate explainability option helps you understand the factors that influence the forecast, for each time series, for each time step of the forecast horizon, and remember to select the Generate CSV files for download option.  

Forecast Results

Once the forecast is completed, it will create a zip file with the result to upload to another system or display on your dashboard. Below is the forecast result:

Below is the csv output:

Forecast report

Forecast output

Conclusion

A few takeaways from my experience in forecasting solar energy production using OCI Forecasting Service:

• The accuracy of weather forecasts can vary depending on several factors, such as the lead time of the forecast, the location, the weather conditions, and the forecasting methods used.  Generally, short-term weather forecasts (up to 3 days) are more accurate than long-term forecasts (beyond 7 days).  The accuracy of weather forecasts can also depend on the specific location, as some areas may have more complex weather patterns or fewer monitoring stations.  For this use case, I only use the average hourly temperature.
• Add other features/factors as additional data from your solar inverter data  that might affect solar energy production, like the age of the solar panel and inverter.
• For this use case, only forecast using a maximum of 7 days of weather data.
• When creating the forecast project using the data asset, remember the timestamp or date column’s data type shall be a date, the time-series name data type shall be string, and the target column data type shall be double or int.
• Let the service choose the algorithms automatically but experiment on different error metrics to compare a forecasting model.
• Create an architecture to automatically ingest the data from the Object Storage using OCI Events and process in OCI Functions or OCI Data Integration before running the OCI Forecasting Service.  Here is the link to the reference architecture: Use OCI Forecasting to quickly create forecasts for your business
• Remember to name your data asset when you create a forecast in OCI Forecasting; otherwise, it is hard to find the data asset to use for other forecasts or projects.  If you forgot to name it when you created it, you could always go back to rename it later.
• Forecasting has an upper limit for each file: 100,000 time-series items with 1,500 data points per series.  Forecasting has no lower limit for the number of time-series items.  For this use case, I have only one time-series item and use the 7 days hourly data, but if you use a 5min interval or lower data point, you exceed the allowed data point limit.
• Ensure the length of the additional data series is greater than or equal to the size of the primary series plus the forecast horizon.  That means you must provide future values for the prediction timestamps.  For example, to forecast the next two weeks’ solar production, provide values for the next two weeks.  Also, Missing values are not allowed in the additional data, and the dates must match the primary dataset.
• As you cannot set the floor and cap forecast values, you might need to process the forecast result data before you display it on your dashboard. In this use case, the forecast value for solar generation cannot be negative. Also, it should be at most 10kw as the inverter’s maximum capacity is 10kw.