DOI: 10.17587/prin.17.376-390

Intelligent Autopilot Agent based on Neural Network Technologies in an Autonomous Driving Simulation Environment

I. R. Lemeshev, Bachelor, igor.lemeshev.ai@gmail.com, S. S. Bulykin, Bachelor, sb17092004@mail.ru, A. A. Solovyev, Bachelor, alexsolovyvev@mail.ru, A. S. Mikhalev, Senior Lecturer at the Department of Software Engineering, asmikhalev@yandex.ru, Institute of space and information technologies, Siberian Federal University, Krasnoyarsk, 660041, Russian Federation

Corresponding author: Anton S. Mikhalev, Senior Lecturer at the Department of Software Engineering, Institute of space and information technologies, Siberian Federal University, Krasnoyarsk, 660041, Russian Federation, E-mail: asmikhalev@yandex.ru

The paper presents the architecture of an intelligent agent for end-to-end autonomous vehicle control based on the M3Drive neural network and multimodal data fusion in the CARLA simulation environment. The proposed approach combines visual features (depth maps and panoptic segmentation for reconstructing 3D geometry and scene semantics), navigation commands and spatial coordinates of route control points, as well as time series of agent states (history of control actions and critical event indicators). The model architecture includes specialized encoders for each data type: EfficientNet-lite0 for visual modalities followed by cross-attention fusion, bidirectional LSTM with an attention mechanism for processing temporal context, and a gating module for tabular features. The combined representations are fed into a multilayer perceptron with residual blocks and multi-task output "heads" that predict continuous control actions: steering angle, acceleration, and braking. Training was conducted in two stages with a weighted loss function. In the second stage, the model was fine-tuned using scenarios involving turns and reactions to traffic lights, which reduced steering errors by more than half. Experimental results at the independent test site confirmed the effectiveness of the system: a route completion rate (RC) of 89 % and an integral driving score (DS) of 67.6 were achieved at a high performance of 58 FPS. The results confirm the applicability of the proposed architecture for creating lightweight real-time autonomous driving systems and lay the foundation for transfer to the Mitsubishi i-MiEV physical platform.

Keywords: intelligent transportation systems, autonomous driving, intelligent agent, neural networks, multimodal data, panoptic segmentation, depth map, simulation learning

pp. 376—390

For citation:
Lemeshev I. R., Bulykin S. S., Solovyev A. A., Mikhalev A. S. Intelligent Autopilot Agent based on Neural Network Technologies in an Autonomous Driving Simulation Environment, Programmnaya Ingeneria, 2026, vol. 17, no. 7, pp. 376—390. DOI: 10.17587/prin.17.376-390 (in Russian).

References:

1. Chib P. S., Singh P. Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A survey, IEEE Transactions on Intelligent Vehicles, 2023, vol. 9, no. 1, pp. 103—118. DOI: 10.1109/TIV.2023.3318070.
2. Bojarski M., Del Testa D., Dworakowski D. et al. End to end learning for self-driving cars, arXiv preprint arXiv:1604.07316, 2016. DOI: 10.48550/arXiv.1604.07316.
3. Pomerleau D. A. ALVINN: An Autonomous Land Vehicle in a Neural Network, Advances in Neural Information Processing Systems, 1989, vol. 1, pp. 305—313.
4. Codevilla F., Miiller M., Lopez A. et al. End-to-end driving via conditional imitation learning, IEEE international conference on robotics and automation (ICRA), 2018, Brisbane, QLD, Australia, 2018, pp. 4693—4700. DOI: 10.1109/ICRA.2018.8460487.
5. Bansal M., Krizhevsky A., Ogale A. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst, arXiv preprint arXiv:1812.03079, 2018.
6. Ross S., Gordon G. J., Bagnell J. A. A reduction of imitation learning and structured prediction to no-regret online learning, Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011, pp. 627—635.
7. Hu A., Corrado G., Griffiths N. et al. Model-based imitation learning for urban driving, Advances in Neural Information Processing Systems, 2022, vol. 35, pp. 20703—20716.
8. Ho J., Ermon S. Generative adversarial imitation learning, Advances in neural information processing systems, 2016, vol. 29, pp. 4565—4573.
9. Kuefler A., Morton J., Wheeler T., Kochenderfer M. Imitating driver behavior with generative adversarial networks, Intelligent Vehicles Symposium (IV), IEEE, 2017, pp. 204—211.
10. Liang X., Wang T., Yang L., Xing E. Cirl: Controllable imitative reinforcement learning for vision-based self-driving, Pro­ceedings of the European conference on computer vision (ECCV), 2018, CCV 2018. Lecture Notes in Computer Science, Vol. 11211. Springer, Cham, 2018, pp. 584—599. DOI: 10.1007/978-3-030-01234-2_36.
11. Xiao Y., Codevilla F., Porres D., Lopez A. M. Scaling vision-based end-to-end autonomous driving with multi-view attention learning, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2023, pp. 1586—1593.
12. Chitta K., Prakash A., Jaeger B. et al. TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving, IEEE transactions on pattern analysis and machine intelligence, 2022, vol. 45, no. 11, pp. 12878—12895. DOI: 10.1109/TPAMI.2022.3200245.
13. Dosovitskiy A., Ros G., Codevilla F. et al. CARLA: An open urban driving simulator, Conference on robot learning, PMLR, 2017, pp. 1—16.
14. Tan M., Le Q. V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, International conference on machine learning, PMLR, 2019, pp. 6105—6114.