Deep Q-Network-Based Energy Management of Battery–Supercapacitor Hybrid Energy Storage in a Renewable Microgrid

MinhPhong Lê

doi:10.55927/fjmr.v5i10.213

Authors

MinhPhong Lê Thu Duc College of Technology

DOI:

https://doi.org/10.55927/fjmr.v5i10.213

Keywords:

Deep Q-Network, Energy Management, Hybrid Energy Storage System, Battery–Supercapacitor, Renewable Microgrid

Abstract

The increasing variability of renewable generation creates challenges for energy management in microgrids equipped with hybrid energy storage systems. This study develops a Deep Q-Network (DQN)-based strategy for coordinated battery–supercapacitor operation. The controller uses a seven-dimensional state representation and 25 discrete control actions within a reinforcement-learning environment. A rule-based controller (RBC) serves as the benchmark. During 50 training episodes, the cumulative reward improves, although the training loss remains variable. The evaluation results show distinct battery and supercapacitor utilisation patterns, but intermittent grid imports and negative cumulative reward indicate that the learned policy requires further improvement. The findings demonstrate the feasibility of DQN-based hybrid storage control, rather than overall performance superiority over RBC.

References

Asadi, Y., M. Eskandari, and M. Mansouri. 2026. “Revolutionising Battery Energy Storage Systems Energy Management: Dynamic-Aware Solutions with Integrated Quantum Particle Swarm Optimisation and Deep Reinforcement Learning.” IET Energy Systems Integration 8 (1): 1–15. https://doi.org/10.1049/esi2.70038.

Atef, M., S. Alahakoon, U. Mumtahina, P. Wolfs, T. Khatib, and M. Uddin. 2026. “A Review of Microgrid Energy Management Systems: Methods, Challenges, and Future Directions.” International Journal of Ambient Energy 47 (1): 1–28. https://doi.org/10.1080/01430750.2025.2595117.

Banka, S., and D. V. A. Kumar. 2026. “Hierarchical Data-Driven and PSO-Based Energy Management of Hybrid Energy Storage Systems in DC Microgrids.” Automation 7 (2): 1–21. https://doi.org/10.3390/automation7020050.

Dawn, S., S. Manideep, S. S. Rekha, C. R. Rao, K. D. Rao, A. Al Mansur, and T. S. Ustun. 2025. “Advancing Renewable Energy Integration in Deregulated Markets: The Role of Energy Storage, EVs, and Policy Frameworks.” Energy Exploration and Exploitation 43 (6): 2745–2797. https://doi.org/10.1177/01445987251348052.

Eslami, F., M. Gangineni, A. Ebrahimi, M. Rathnayake, M. Patel, and O. Lavrova. 2026. “A Review on Protection and Cybersecurity in Hybrid AC/DC Microgrids: Conventional Challenges and AI/ML Approaches.” Energies 19 (3): 1–38. https://doi.org/10.3390/en19030744.

Ghoudelbourk, S., R. Yousfi, and H. Benbouhenni. 2025. “Energy Management of an Autonomous Hybrid Renewable Power System.” In 2025 IEEE 13th International Conference on Smart Energy Grid Engineering, 190–194. IEEE. https://doi.org/10.1109/SEGE65970.2025.11203682.

Gowtham, M., and S. Suresh. 2025. “Supercapacitor Integration in Hybrid Energy Storage Systems for Electric Vehicles: A Comprehensive Review.” In Proceedings of the 2025 International Conference on Intelligent Computing, Information and Control Systems, 170–176. IEEE. https://doi.org/10.1109/ICOIICS67115.2025.11390385.

Guo, Z., E. Grano, H. de Carvalho Pinheiro, and M. Carello. 2026. “A Comparative Evaluation of Rule-Based Strategies, ECMSs, and MPC Strategies for Fuel Cell Hybrid LCV Energy Management.” World Electric Vehicle Journal 17 (3): 1–25. https://doi.org/10.3390/wevj17030163.

Ioannou, I., S. Javaid, Y. Tan, and V. Vassiliou. 2026. “A Hierarchical Predictive-Adaptive Control Framework for State-of-Charge Balancing in Mini-Grids Using Deep Reinforcement Learning.” Electronics 15 (1): 1–47. https://doi.org/10.3390/electronics15010061.

Kavaliauskas, Ž., M. Milieška, G. Blažiūnas, G. Gecevičius, and H. Zhairabany. 2026a. “Optimisation of Hybrid Energy System Control Using MPC and MILP.” Applied Sciences 16 (8): 1–22. https://doi.org/10.3390/app16083690.

Kavaliauskas, Ž., M. Milieška, G. Blažiūnas, G. Gecevičius, and H. Zhairabany. 2026b. “Optimisation of Control for a Hybrid Renewable Energy System with Energy Storage Using Deep Reinforcement Learning Methods.” Sustainability 18 (11): 1–43. https://doi.org/10.3390/su18115443.

Khan, W., F. Renhai, A. Aziz, M. Z. Yousaf, Z. Cai, M. U. Iqbal, and M. S. Geremew. 2026. “Deep Reinforcement Learning-Based Energy Management for Design and Control of Off-Grid Renewable Microgrids with Dual-Battery Storage.” Energy Exploration and Exploitation 44 (2): 821–869. https://doi.org/10.1177/01445987251387368.

Kırant-Mitić, T., and K. Voss. 2026. “A Rule-Based Predictive Control Framework for Market and Grid-Oriented Operation in Thermally Activated Buildings.” Journal of Building Performance Simulation 19 (2): 209–238. https://doi.org/10.1080/19401493.2025.2565652.

Li, Z., W. Long, and H. Tian. 2026. “Current Applications and Future Prospects of Deep Reinforcement Learning in Energy Management for Hybrid Power Systems.” Energies 19 (9): 1–50. https://doi.org/10.3390/en19092216.

Mulaudzi, S. T., T. Rasimphi, and D. Tinarwo. 2026. “Assessing the Contribution of Microgrids to Sustainable Agriculture and Community Development at the Masia Development Centre.” Clean Energy Technologies 2 (1): 1–16. https://doi.org/10.54963/cet.v2i1.1945.

Sargentis, G., N. Papadodimas, I. Benekos, N. M. Katsoulakos, P. Dimitriadis, N. Tepetidis, and D. Koutsoyiannis. 2026. “Stochastic Assessment of Renewable Energy Reliability: A Case Study of North Euboea, Greece.” Journal of Energy and Power Technology 8 (2): 1–13. https://doi.org/10.21926/jept.2602008.

Srivastava, A., S. K. Jha, D. Gupta, and S. Singh. 2026. “A Critical Review of Microgrid Technologies: Storage and Control Systems with Effective Optimisation Techniques for Modeling Wind, Solar, Biomass, and Battery Energy Storage Systems.” Cureus Journals 3 (1): 1–23. https://doi.org/10.7759/s44388-025-00041-y.

Tasmant, H., B. Bossoufi, C. Alaoui, and P. Siano. 2025. “A Review of Machine Learning and IoT-Based Energy Management Systems for AC Microgrids.” Computers and Electrical Engineering 127: 1–23. https://doi.org/10.1016/j.compeleceng.2025.110563.

Ton, T. N., P. M. Le, and T. M. Le. 2026. “Multi-Objective Optimization of Distributed Generation Placement and Sizing in Active Distribution Networks Considering Harmonic Distortion.” International Journal of Electrical and Computer Engineering 16 (2): 598–607.

Wang, X., H. Zhang, H. Liu, and F. L. Lewis. 2026. “Control-Oriented Reinforcement Learning: A Survey of Recent Progress and Applications.” International Journal of Robust and Nonlinear Control 36 (2): 694–718. https://doi.org/10.1002/rnc.70152