Microgrid control for renewable energy sources based on deep reinforcement learning and numerical optimization approaches

A. Yu. Zhadan ; H. Wu ; P. S. Kudin ; Y. Zhang ; O. L. Petrosian

Vestnik Sankt-Peterburgskogo universiteta. Prikladnaâ matematika, informatika, processy upravleniâ, Tome 19 (2023) no. 3, pp. 391-402

Résumé

Optimal scheduling of battery energy storage system plays crucial part in distributed energy system. As a data driven method, deep reinforcement learning does not require system knowledge of dynamic system, present optimal solution for nonlinear optimization problem. In this research, financial cost of energy consumption reduced by scheduling battery energy using deep reinforcement learning method (RL). Reinforcement learning can adapt to equipment parameter changes and noise in the data, while mixed-integer linear programming (MILP) requires high accuracy in forecasting power generation and demand, accurate equipment parameters to achieve good performance, and high computational cost for large-scale industrial applications. Based on this, it can be assumed that deep RL based solution is capable of outperform classic deterministic optimization model MILP. This study compares four state-of-the-art RL algorithms for the battery power plant control problem: PPO, A2C, SAC, TD3. According to the simulation results, TD3 shows the best results, outperforming MILP by 5 % in cost savings, and the time to solve the problem is reduced by about a factor of three.

Détail

Keywords: reinforcement learning, energy management system, distributed energy system, numerical optimization.

Bibliographie

[1] Markelova A., Petrosian O., Laistseva M., “Microgrid management with energy storage system and renewable energy generation”, Control Processes and Stability, 8:1 (2021), 430–434

[2] Hatziargyriou N. D., “Special issue on microgrids and energy management”, European transactions on electrical power, 21:2 (2011), 1139–1141 | DOI

[3] Reddy P. P., Veloso M. M., “Strategy learning for autonomous agents in smart grid markets”, Twenty-Second International Joint Conference on Artificial Intelligence (Barcelona, 2011), 1446–1451

[4] Chen C., Duan S., Cai T., Liu B., Hu G., “Smart energy management system for optimal microgrid economic operation”, IET Renewable Power Generation, 5:3 (2011), 258–267 | DOI | MR

[5] Mohamed F. A., Koivo H. N., “System modelling and online optimal management of microgrid with battery storage”, 6$^{th}$ International Conference on renewable energies and power quality (ICREPQ'07) (Sevilla, 2007), 26–28

[6] Colson C., Nehrir M., Pourmousavi S., “Towards real-time microgrid power management using computational intelligence methods”, IEEE PES general meeting, IEEE, 2010, 1–8

[7] Abdilahi A. M., Mustafa M., Aliyu G., Usman J., “Autonomous Integrated Microgrid (AIMG) system: Review of Potential”, International Journal of Education and Research, 2:1 (2014), 1–18

[8] Mnih V., Kavukcuoglu K., Silver D., Graves A., Antonogłou I., Wierstra D., Riedmiller M., Playing atari with deep reinforcement learning, 2013, arXiv: 1312.5602

[9] Perera A., Kamalaruban P., “Applications of reinforcement learning in energy systems”, Renewable and Sustainable Energy Reviews, 137 (2021), 1–22 | DOI | MR

[10] Muriithi G., Chowdhury S., “Optimal energy management of a grid-tied solar pv-battery microgrid: A reinforcement learning approach”, Energies, 14:9 (2021), 1–24 | DOI

[11] Wei C., Zhang Z., Qiao W., Qu L., “Reinforcement-learning-based intelligent maximum power point tracking control for wind energy conversion systems”, IEEE Transactions on Industrial Electronics, 62:10 (2015), 6360–6370 | DOI

[12] Xi L., Yu L., Fu Y., Huang Y., “Automatic generation control based on deep reinforcement learning with exploration awareness”, Proceedings of the CSEE, 39:14 (2019), 4150–4162

[13] Wang B., Zhou M., Xin B., Zhao X., Watada J., “Analysis of operation cost and wind curtailment using multi-objective unit commitment with battery energy storage”, Energy, 178 (2019), 101–114 | DOI

[14] Gao Y., Yang J., Yang M., Li Z., “Deep reinforcement learning based optimal schedule for a battery swapping station considering uncertainties”, IEEE Transactions on Industry Applications, 56:5 (2020), 5775–5784 | DOI

[15] Kell A. J., McGough A. S., Forshaw M., Optimizing a domestic battery and solar photovoltaic system with deep reinforcement learning, 2021, arXiv: 2109.05024

[16] Wu Y., Tan H., Peng J., Zhang H., He H., “Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus”, Applied Energy, 247 (2019), 454–466 | DOI

[17] Duan Y., Chen X., Houthooft R., Schulman J., Abbeel P., “Benchmarking deep reinforcement learning for continuous control”, International Conference on Machine Learning (New York, 2016), 1329–1338

[18] Official website Schneider Electric, (accessed: March 1, 2022) https://www.se.com/ww/en/

[19] Power Laws: Optimizing Demand-side Strategies (accessed: March 1, 2022) www.drivendata.org/competitions/53/optimize-photovoltaic-battery/

[20] Sedighizadeh M., Esmaili M., Mohammadkhani N., “Stochastic multi-objective energy management in residential microgrids with combined cooling, heating, and power units considering battery energy storage systems and plug-in hybrid electric vehicles”, Journal of Cleaner Production, 195 (2018), 301–317 | DOI

[21] Sutton R. S., Barto A. G., Reinforcement learning: An introduction, MIT Press, London, 2018, 590 pp. | MR | Zbl

[22] Babaeizadeh M., Frosio I., Tyree S., Clemons J., Kautz J., Reinforcement learning through asynchronous advantage actor-critic on a GPU, 2016, arXiv: 1611.06256

[23] Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O., Proximal policy optimization algorithms, 2017, arXiv: 1707.06347

[24] Fujimoto S., Hoof H., Meger D., “Addressing function approximation error in actor-critic methods”, International Conference on Machine Learning, PMLR, 2018, 1587–1596

[25] Haarnoja T., Zhou A., Abbeel P., Levine S., “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor”, International Conference on Machine Learning, PMLR, 2018, 1861–1870

[26] Mittelmann H., “Latest benchmark results”, Proceedings of the INFORMS Annual Conference (Phoenix, 2018), 4–7 | Zbl

[27] Brockman G., Cheung V., Pettersson L., Schneider J., Schulman J., Tang J., Zaremba W., OpenAI Gym, 2016, arXiv: 1606.01540