On the application of reinforcement learning in the task of choosing the optimal trajectory

M. G. Gorodnichev

News of the Kabardin-Balkar scientific center of RAS, Tome 27 (2025) no. 2, pp. 86-102.

Résumé

This paper reviews state-of-the-art reinforcement learning methods, with a focus on their application in dynamic and complex environments. The study begins by analysing the main approaches to reinforcement learning such as dynamic programming, Monte Carlo methods, time-difference methods and policy gradients. Special attention is given to the Generalised Adversarial Imitation Learning (GAIL) methodology and its impact on the optimisation of agents' strategies. A study of model-free learning is presented and criteria for selecting agents capable of operating in continuous action and state spaces are highlighted. The experimental part is devoted to analysing the learning of agents using different types of sensors, including visual sensors, and demonstrates their ability to adapt to the environment despite resolution constraints. A comparison of results based on cumulative reward and episode length is presented, revealing improved agent performance in the later stages of training. The study confirms that the use of simulated learning significantly improves agent performance by reducing time costs and improving decision-making strategies. The present work holds promise for further exploration of mechanisms for improving sensor resolution and fine-tuning hyperparameters.

Détail

Keywords: Keywords: reinforcement learning, optimal trajectory, highly automated vehicles, policy-based learning, actor-critic architectures, simulated learning, sensors, continuous states, discrete states, PPO
Mots-clés : intelligent agents, SAC

Bibliographie

[1] S. Zhang, Q. Xia, M. Chen, S. Cheng, “Multi-Objective Optimal Trajectory Planning for Robotic Arms Using Deep Reinforcement Learning”, Sensors, 23 (2023), 5974 | DOI

[2] M. G. Tamizi, M. Yaghoubi, H. Najjaran, “A review of recent trend in motion planning of industrial robots”, International Journal of Intelligent Robotics and Applications, 7 (2023), 253–274 | DOI

[3] T. Kollar, N. Roy, “Trajectory Optimization using Reinforcement Learning for Map Exploration”, International Journal of Robotics Research, 27:2 (2008), 175–196 | DOI

[4] E. U. Acar, H. Choset, Y. Zhang, M. Schervish, “Path planning for robotic demining: robust sensor-based coverage of unstructured environments and probabilistic methods”, International Journal of Robotics Research, 22:7-8 (2003), 441–466

[5] D. A. Cohn, Z. Ghahramani, M. I. Jordan, “Active learning with statistical models”, Journal of Artificial Intelligence Research, 1996, no. 4, 705–712

[6] K. Axhausen et al., “Introducing MATSim.”, Multi-Agent Transport Simulation MATSim, eds. A. Horni, Ubiquity Press, London, 2016, 3–8 pp. | DOI

[7] G. Wu, D. Zhang, Z. Miao, W. Bao, J. Cao, “How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making”, Electronics, 13 (2024), 1281 | DOI

[8] T. Zhou, M. Lin, “Deadline-aware deep-recurrent-q-network governor for smart energy saving”, IEEE Transactions on Network Science and Engineering, 9 (2021), 3886–3895 | DOI

[9] Y. Yang, J. Wang, “An overview of multi-agent reinforcement learning from game theoretical perspective”, arXiv 2020, arXiv:2011 | DOI

[10] N. Mazyavkina, S. Sviridov, S. Ivanov, E. Burnaev, “Reinforcement learning for combinatorial optimization: A survey”, Comput. Oper. Res, 134 (2021), 105400 | DOI

[11] Junwei Zhang, Zhenghao Zhang, Shuai Han, Lu Shuai, “Proximal policy optimization via enhanced exploration efficiency”, Information Sciences, 609 (2022), 750 | DOI

[12] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul et al., “Rainbow: Combining improvements in deep reinforcement learning”, In AAAI Conference on Artificial Intelligence, 2018, 3215–3222 | DOI

[13] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor”, In International Conference on Machine Learning, 2018, 1856–1865 | DOI

[14] T. P. Lillicrap, J. J. Hunt, A. Pritzel et al., Continuous control with deep reinforcement learning, arXiv:1509.02971v1, 2015.

[15] Y. Chen, C. T. Lam, G. Pau, W. Ke, “From Virtual to Reality: A Deep Reinforcement Learning Solution to Implement Autonomous Driving with 3D-LiDAR”, Applied Sciences, 15:3 (2025), 1423 | DOI

[16] Guoyu Zuo, Kexin Chen, Jiahao Lu, Xiangsheng Huang, “Deterministic generative adversarial imitation learning \”, Neurocomputing, 388 (2020) | DOI

[17] R. Sawada, “Automatic Collision Avoidance Using Deep Reinforcement Learning with Grid Sensor”, Proceedings of the 23rd Asia Pacific Symposium on Intelligent and Evolutionary Systems, Proceedings in Adaptation, Learning and Optimization, v. 12, eds. H. Sato, I wanaga, A. Ishii, Springer, Cham, 2020, 17–32 pp. | DOI

[18] T. Hachaj, M. Piekarczyk, “On Explainability of Reinforcement Learning-Based Machine Learning Agents Trained with Proximal Policy Optimization That Utilizes Visual Sensor Data”, Applied Sciences, 15:2 (2025), 538 | DOI