SMART4RES
Next Generation Modelling and Forecasting of Variable Renewable Generation for Large-scale Integration in Energy Systems and Markets Artificial IntelligencePower and Energy Systems
Description
The development of data science and increasing quantities of data collected opens new possibilities for renewable energy (RES) forecasting. The Smart4RES project aimed to bring substantial performance improvements to the whole model and value chain in renewable energy sources (RES) forecasting, focusing on optimising synergies with storage and supporting power system operation and participation in electricity markets. INESC TEC developed a patented vertical federated learning algorithm for distributed learning and forecasting with data owned by different entities, and developed algorithmic solutions and a software prototype for a data market. Moreover, it developed a stochastic tool for dispatching synchronous and virtual inertia under forecast uncertainty in isolated power systems and a risk-aware methodology for ranking flexibility options and balancing risk/stakes to solve voltage and congestion problems in electrical grids. The INESC TEC team published six papers in Q1 journals.
Scientific Advances
This project aimED to cover future scenarios with extreme RES integration and produced the following advances:
1) Explored high temporal resolution RES power forecasts to generate probabilistic forecasts of synthetic inertia/FCR from RES plants and integrate this information in a data-driven economic dispatch with additional security constraints to guarantee minimum inertia or FCR aiming to securely operate isolated power systems with near-100% RES integration.
2) Past research from the 80s in expert systems and distributed computing for voltage and congestion management will be revisited by exploring recent advances in deep learning and reinforcement learning to produce improvements in the numerical and computational performance of these classical techniques and implement a predictive framework.
Moreover, it aimed to fully explore synergies between new distributed and privacy-preserving analytics and smart contracts technology to create a new marketplace for forecasting services and redefined classical business models for data providers. In particular, the following advances were produced:
1) Distributed learning algorithms that enable different agents (e.g., RES power plant owners, market players) to explore geographically distributed time-series data and improve forecasting skill while being privacy-preserving by using algebraic cryptography techniques and different distributed optimization techniques (i.e., variants of ADMM, stochastic mirror descent). This means that instead of sharing their data, learning problems will be solved in a distributed manner. Different communication schemes were explored: centralized and peer-to-peer, synchronous and asynchronous, and their implications in terms of convergence time and privacy.
2) Algorithmic solutions for data markets, which allows different agents to sell and buy data of relevance for RES forecasting, with pricing being a function of their value for other agents. Besides the algorithmic considerations, adequate pricing of data products will be proposed (e.g., establish a link with the continuous intraday markets in Europe) to assess the value of data for the buyers, as well as a blockchain-based platform with smart contracts.
Results
1) By exploring concepts from extreme value theory (EVT), which is dedicated to characterizing the stochastic behavior of extreme values, it proposed a novel wind power forecasting methodology, focused on improving the forecasting skill of the distribution’s tails, which combines spatiotemporal information (obtained via feature engineering), gradient boosting trees (GBT) as a nonparametric method for quantiles between 0.05 and 0.95 and the truncated generalized Pareto distribution (GPD) for the tails.
2) With the focus on privacy-preserving protocols for very short-term forecasting with the vector autoregressive (VAR) model, one research outcome was a novel combination of data transformation and decomposition-based methods so that the VAR model is fitted in another feature space without decreasing the forecast skill. The main advantage of this combination is that the ADMM algorithm is not affected and therefore: (a) asynchronous communication between peers can be addressed while fitting the model; (b) a flexible privacy-preserving collaborative model can be implemented using two different schemes, centralized communication with a neutral node and peer-to-peer communication, and in a way the central node or peers cannot recover that original data.
3) Marketplace where data owners purchase forecasts and pay according to resulting forecasting accuracy. This avoids the confidentiality problem of sharing raw data directly. Cooperation between sellers is done through a market operator who receives all agents' data and prepares forecasts: (i) sellers with similar information receive similar revenue, (ii) the market price is a function of the buyer’s benefit, and so the buyer does not pay if there is no improvement in the forecasting skill, (iii) buyers pay according to incremental gain, and (iv) buyers purchase forecasts, instead of features, and have no knowledge about which datasets were used to produce these forecasts. To the best of our knowledge, this is the first work to describe an algorithmic solution for data markets that enables different RES agents to sell data (historical power production, NWP, etc.) and buy forecasts of their power production, and where the economic value of this data is fundamentally related to imbalance cost reduction in electricity markets.
4) A predictive grid management framework that uses sensitivity indices and risk metrics to rank flexibility options and summarize human operators in multi-criteria problems, taking risk vs cost curves. This approach contrasts OPF-based methods that, despite their mathematical interpretability, do not enable human operators to analyze action-cause relations extracted from sensitivity indices. Furthermore, it facilitates enhanced interaction between operators and the decision-aid tool, specifically by allowing them to rank and explore various flexibility solutions and adjust the risk level based on the decision stakes. This relationship between risk level and stakes has not been previously studied in prior works about stochastic and robust optimization.
Scientific Advances
This project aimED to cover future scenarios with extreme RES integration and produced the following advances:
1) Explored high temporal resolution RES power forecasts to generate probabilistic forecasts of synthetic inertia/FCR from RES plants and integrate this information in a data-driven economic dispatch with additional security constraints to guarantee minimum inertia or FCR aiming to securely operate isolated power systems with near-100% RES integration.
2) Past research from the 80s in expert systems and distributed computing for voltage and congestion management will be revisited by exploring recent advances in deep learning and reinforcement learning to produce improvements in the numerical and computational performance of these classical techniques and implement a predictive framework.
Moreover, it aimed to fully explore synergies between new distributed and privacy-preserving analytics and smart contracts technology to create a new marketplace for forecasting services and redefined classical business models for data providers. In particular, the following advances were produced:
1) Distributed learning algorithms that enable different agents (e.g., RES power plant owners, market players) to explore geographically distributed time-series data and improve forecasting skill while being privacy-preserving by using algebraic cryptography techniques and different distributed optimization techniques (i.e., variants of ADMM, stochastic mirror descent). This means that instead of sharing their data, learning problems will be solved in a distributed manner. Different communication schemes were explored: centralized and peer-to-peer, synchronous and asynchronous, and their implications in terms of convergence time and privacy.
2) Algorithmic solutions for data markets, which allows different agents to sell and buy data of relevance for RES forecasting, with pricing being a function of their value for other agents. Besides the algorithmic considerations, adequate pricing of data products will be proposed (e.g., establish a link with the continuous intraday markets in Europe) to assess the value of data for the buyers, as well as a blockchain-based platform with smart contracts.
Results
1) By exploring concepts from extreme value theory (EVT), which is dedicated to characterizing the stochastic behavior of extreme values, it proposed a novel wind power forecasting methodology, focused on improving the forecasting skill of the distribution’s tails, which combines spatiotemporal information (obtained via feature engineering), gradient boosting trees (GBT) as a nonparametric method for quantiles between 0.05 and 0.95 and the truncated generalized Pareto distribution (GPD) for the tails.
2) With the focus on privacy-preserving protocols for very short-term forecasting with the vector autoregressive (VAR) model, one research outcome was a novel combination of data transformation and decomposition-based methods so that the VAR model is fitted in another feature space without decreasing the forecast skill. The main advantage of this combination is that the ADMM algorithm is not affected and therefore: (a) asynchronous communication between peers can be addressed while fitting the model; (b) a flexible privacy-preserving collaborative model can be implemented using two different schemes, centralized communication with a neutral node and peer-to-peer communication, and in a way the central node or peers cannot recover that original data.
3) Marketplace where data owners purchase forecasts and pay according to resulting forecasting accuracy. This avoids the confidentiality problem of sharing raw data directly. Cooperation between sellers is done through a market operator who receives all agents' data and prepares forecasts: (i) sellers with similar information receive similar revenue, (ii) the market price is a function of the buyer’s benefit, and so the buyer does not pay if there is no improvement in the forecasting skill, (iii) buyers pay according to incremental gain, and (iv) buyers purchase forecasts, instead of features, and have no knowledge about which datasets were used to produce these forecasts. To the best of our knowledge, this is the first work to describe an algorithmic solution for data markets that enables different RES agents to sell data (historical power production, NWP, etc.) and buy forecasts of their power production, and where the economic value of this data is fundamentally related to imbalance cost reduction in electricity markets.
4) A predictive grid management framework that uses sensitivity indices and risk metrics to rank flexibility options and summarize human operators in multi-criteria problems, taking risk vs cost curves. This approach contrasts OPF-based methods that, despite their mathematical interpretability, do not enable human operators to analyze action-cause relations extracted from sensitivity indices. Furthermore, it facilitates enhanced interaction between operators and the decision-aid tool, specifically by allowing them to rank and explore various flexibility solutions and adjust the risk level based on the decision stakes. This relationship between risk level and stakes has not been previously studied in prior works about stochastic and robust optimization.
