Optimeering Forecast Overview - How Do We Create Our Forecasts?
Optimeering’s forecasts are produced using a standard generalised infrastructure and forecast methodology, irrespective of the market being forecast. Our general approach is summarised as follows.
Step 1: Publicly available data
All of our forecasts use publicly available data (that is, data that is available to market actors, 3rd party participants, and indeed “members of the public”). This data is either freely available or is made available via payment of reasonable data feed subscription fees.
Example data sources used by Optimeering are weather data (published by met.no), TSO data (ENTSOE transparency platform, as well as on TSO’s own publicly facing data platforms), and market data (exchanges such as Nord Pool).
This data is typically obtained by Optimeering via API feeds, processed, cleaned and stored by us in our cloud-based data storages in databases, flat files, and similar.
Step 2: Feature engineering
The data is automatically analysed, processed and combined into specific model features in-code. These features have been developed by Optimeering based on our insight and understanding of how the power markets work, how they are cleared, and what drives the outcomes (prices, volumes etc). The features we develop are a big part of our “secret sauce” and are important in ensuring the quality of our forecasts.
Our features are prepared both on-the-fly and pre-calculated and stored, depending on the computational and data requirements of each feature, before being made available to our prediction models.
Step 3: AI/ML model ensembles
All our forecasts are generated using AI and ML (machine learning) modelling. We typically use “model ensembles” - that is, collections of models in series and parallel. Each model takes in features to generate an output, before feeding these to a subsequent model in the ensemble.
An example of this might be a model that processes wind-based features to produce a wind imbalance forecast for a given price area, which is then fed into a subsequent imbalance price forecast model.
Ultimately, each model ensemble will produce a price or volume forecast for the secured market and time interval. Some of our models produce forecasts for multiple time intervals and price areas simultaneously, others produce forecasts for a single time interval and price area.
As our models are AI/ML models, specific model instances are trained (or “fitted”) using historical data from the past several years. We undertake a comprehensive model parameter tuning and pruning process to ensure that we do not overfit, and that our model architecture and structure is designed to be as accurate as possible.
Step 4: Forecasts
The model ensembles are run on live data (more specifically, on feature data) either at regular time intervals (e.g. every 15 minutes) or when new data (and hence updated features) are made available to the models.
The outputs from our model ensembles are combined to produce the specific market forecast provided by Optimeering. Some forecasts use a set of rules to perform this, for others we use an AI/ ML “meta model” to learn how to best combine the ensemble forecasts based on past data and to then perform the combination process.