Journal "Software Engineering"
a journal on theoretical and applied science and technology
ISSN 2220-3397

Issue N7 2026 year

DOI: 10.17587/prin.17.355-365
Development of an Architecture for Analyzing a Person's Emotional State based on Facial Expression in an Immersive Learning Environment
R. E. Aslanov, Cand. Sc. (Eng.), Director, aslanov.boxing@mail.ru, State Budgetary Professional Educational Institution of Krasnodar Krai "Ust-Labinsk Social and Pedagogical College", Ust-Labinsk, 352330, Russian Federation, A. A. Bolshakov, D. Sc. (Eng.), Professor, aabolshakov57@gmail.com, Peter the Great St. Petersburg Polytechnic University, St. Petersburg, 195251, Russian Federation, I. V. Veshneva, D. Sc. (Eng.), Associate Professor, Professor, veshnevaiv@mail.ru, Saratov National Research State University named after N. G. Chernyshevsky, Saratov, 410012, Russian Federation, E. I. Malko, Middle Unity Developer, 3D and GameDev Department, i@malko-pro.ru, NANOSEMANTICS LABORATORY LLC, Moscow, 121357, Russian Federation
Corresponding author: Alexander A. Bolshakov, D. Sc. (Eng.), Professor, Peter the Great St. Petersburg Polytechnic University, St. Petersburg, 195251, Russian Federation, E-mail: aabolshakov57@gmail.com
Received on November 19, 2025
Accepted on February 13, 2026

A mathematical model and algorithm for emotion analysis are proposed. The results of developing a software package for analyzing the emotional state of learners in an immersive environment are described. A comparative analysis of existing solutions for recognizing emotional states was conducted, leading to the selection of the py-feat library (Detector module) as the one that best meets the requirements for comprehensive analysis and research re-producibility. An experiment assessing the user's emotional state while working with a multimedia simulator revealed the predominance of negative affective states. The obtained data are interpreted in the context of status functions concept, which are temporary complex-valued functions that form an orthonormal basis for acceptable system states. This revealed a systematic shift from neutral status to frustration.

Keywords: immersive technologies, emotional state, affective state, adaptive learning, emotional analysis, virtual reality, status functions
pp. 355—365
For citation:
Aslanov R. E., Bolshakov A. A., Veshneva I. V., Malko E. I. Development of an Architecture for Analyzing a Person's Emotional State based on Facial Expression in an Immersive Learning Environment, Programmnaya Ingeneria, 2026, vol. 17, no. 7, pp. 355—365. DOI: 10.17587/prin.17.7.355—365. (in Russian).
The work was supported by the Russian Science Foundation, grant No. 25-21-00334, "Prototype of a System for Analyzing a Person's Psychophysical State during Immersive Learning", https://rscf.ru/project/25-21-00334/, carried out at Peter the Great St. Petersburg Polytechnic University.
References:
  1. Lawson A., Martella A. M. Critically reflecting on the use of immersive virtual reality in educational settings: What is known and what has yet to be shown? J. Appl. Learn, 2023, vol. 6, no. 2, pp. 121—133. DOI: 10.37074/jalt.2023.6.2.35.
  2. Alhazzaa K., Yan W. Immersive technologies in Education: Exploring user experience and engagement in building energy simulations through AR and VR, Computers AND Education: X Reality, 2025, vol. 6, article 100097. DOI: 10.1016/j.cexr.2025.100097.
  3. Mahmoud K., Harris I., Yassin H. et al. Does Immersive VR Increase Learning Gain When Compared to a Non-immersive VR Learning Experience? Learning and Collaboration Technologies, Human and Technology Ecosystems, Springer International Publishing, Cham, 2020, pp. 480—498. DOI: 10.1007/978-3-030-50506-6_33.
  4. Tsybov N. N. Didactic Features of Using Training Systems in Technical Universities, Education and Self-Development, 2022, vol. 17, no. 4, pp. 267—287. DOI: 10.26907/esd.17.4.19 (in Russian).
  5. Silva L. B. P., Barreto B. P., Pontes J. et al. Evaluation of Soft Skills through Educational Testbed 4.0, Commun. Comput. Inf. Sci., 2021, pp. 678—690. DOI: 10.1007/978-3-030-91885-9_51.
  6. Chassidim H., Almog D., Mark S. Fostering soft skills in project-oriented learning within an agile atmosphere, Eur. J. Eng. Educ., 2018, vol. 43, no. 4, pp. 638—650. DOI: 10.1080/03043797.2017.1401595.
  7. Protasova Yu. V., Nemtinov V. A., Shipilova V. G. et al. Hybrid Educational Environment of the University's Engineer­ing Schools for the Implementation of Early Career Guidance for Schoolchildren, Programmnaya Ingeneria, 2024, vol. 15, no. 12, pp. 648—661. DOI: 10.17587/prin.14.648-661 (in Russian).
  8. Elshansky S. P. Psychophysiological Factors of Cognitive Learning Effectiveness, Psihologiya Obucheniya, 2020, no. 5, pp. 4—11 (in Russian).
  9. Azevedo R., Gasevic D. Analyzing Multimodal Multichannel Data about Self-Regulated Learning with Advanced Learning Technologies: Issues and Challenges, Computers in Human Behavior, 2020, vol. 96, pp. 207—210. DOI: 10.1016/j.chb.2019.03.025.
  10. Veshneva I. V. Methods for Analyzing the Psychophysical State of a Person During Learning with Immersive Technologies, Informacionnye Tekhnologii v Obrazovanii, 2024, no. 7, pp. 50—58 (in Russian).
  11. Shouq Al Awadhi, Noor Al Habib, Dalal Al-Murad et al. Interactive virtual reality educational application, Adv Sci, Technol Eng Syst, 2018, vol. 3, no. 4, pp. 72—82. DOI: 10.25046/aj030409.
  12. Asish S. M., Kulshreshth A. K., Borst C. W. Detecting distracted students in educational VR environments using machine learning on eye gaze data, Comput. Graph., 2022, vol. 109, pp. 75—87. DOI: 10.1016/j.cag.2022.10.007.
  13. Kurbako A. V., Khramkov A. N., Borovkova E. I. et al. Hardware and software complex for diagnosing the psychophysi-ological state of a person in the process of solving cognitive prob­lems, Izvestiya of Saratov University. Physics, 2024, vol. 24, no. 1, pp. 19—29. DOI: 10.18500/1817-3020-2024-24-1-19-29 (in Russian).
  14. Skvorchevsky K. A., Dyatlova O. V. Modern adaptive and intelligent digital learning systems: mechanisms and potential, Educational Studies, 2024, no. 3 (2). pp. 299—337. DOI: 10.17323/vo-2024-19751 (in Russian).
  15. Veshneva I. V., Chistyakova T. B., Bolshakov A. A. Method for processing and interpreting data on measuring interactions in an educational environment based on status functions, Proceedings of SPIIRAS, 2016, no. 6 (49), pp. 144—166. DOI: 10.15622/sp.49.8 (in Russian).
  16. Veshneva I. V., Bolshakov A. A., Fedorova A. E. Organi­zation of Engineering Education for the Development of Cyber-Physical Systems Based on the Assessment of Competences Using Status Functions, Studies in Systems, Decision and Control, 2020, vol. 260, pp. 277—288. DOI: 10.1007/978-3-030-32648-7_22.
  17. Bilyi A. M., Vinnichenko A. D., Kotelnikova E. A. et al. Implementation of a software system for the analysis and correction of a person's emotional state based on their personal characteristics of physiological indicators, Bulletin of Psychophysiology, 2021, no. 2, pp. 60—72. DOI: 10.34985/e2135-0262-9089-z (in Russian).
  18. Kharki K., Berrada K., Burgos D. Design and Implementation of a Virtual Laboratory for Physics Subjects in Moroccan Universities, Sustainability, 2021, vol. 13, no. 7, article 3711. DOI: 10.3390/su13073711.
  19. Kok Y.-Y., Er H.-M., Nadarajah V. D. An Analysis of Health Science Students' Preparedness and Perception of Interactive Virtual Laboratory Simulation, Med. Sci. Educ., 2021, vol. 1, pp. 1919—1929. DOI: 10.1007/s40670-021-01364-1.
  20. Tlegenova T. E., Shardakov V. M. Testing a Model of Be­havioral Classifiers in Personalized Learning, Perspectives of Science, 2020, no. 6 (129), pp. 15—18 (in Russian).
  21. Baig M., Kavakli M. A Survey on Psycho-Physiological Analysis and Measurement Methods in Multimodal Systems, Mul-timodal Technol. Interact., 2019, vol. 3, article 37. DOI: 10.3390/mti3020037.
  22. Shu L., Xie J., Yang M. et al. A review of emotion recognition using physiological signals, Sensors, 2018, vol. 18, no. 7, article 2074. DOI: 10.3390/s18072074.
  23. Egger M., Ley M., Hanke S. Emotion Recognition from Physiological Signal Analysis: A Review, Electron. Notes Theor. Comput. Sci., 2019, vol. 343, pp. 35—55. DOI: 10.1016/j.entcs.2019.04.009.
  24. Kiryukhin A. V., Alpatov A. V. Design of an automated system for assessing human affective states based on heart rate measurement while viewing video images, Programmnaya Ingeneria, 2015, no. 1, pp. 34—40 (in Russian).
  25. Maltsev A. V. Methods for Implementing Tactile Sensa­tions in Virtual Space by Integrating Physical Objects with Their Computer Models, Programmnaya Ingeneria, 2024, vol. 15, no. 7, pp. 331—339. DOI: 10.17587/prin.15.331-339 (in Russian).
  26. Aguilar Reyes C. I., Wozniak D., Ham A. et al. Design and evaluation of an adaptive virtual reality training system, Virtual Reality, 2023, vol. 27, pp. 2509—2528. DOI: 10.1007/s10055-023-00827-7.
  27. Asbee J., Kelly K., McMahan T., Parsons T. Machine learning classification analysis for an adaptive virtual reality Stroop task, Virtual Reality, 2023, vol. 27, pp. 1391—1407. DOI: 10.1007/s10055-022-00744-1.
  28. Beriev I. R., Alisultanova E. D. Adaptive learning system in an electronic environment, Bulletin of GGNTU, Technical sciences, 2021, vol. 17, no. 4 (26), pp. 5—12. DOI: 10.34708/GSTOU.2021.13.11.001 (in Russian).
  29. Matveev A. V., Paramonov A. I. Algorithm of the adap­tive education system based on the choice of an individual trajectory, Modern education: content, technology, quality, 2023, vol. 1, pp. 450—452 (in Russian).
  30. Tasueva Kh. Z. A., Alikhadzhiev S. Kh. The role of adaptive technologies in the modernization of educational processes, Pedagogical journal, 2024, vol. 14, no. 7-1, pp. 196—203 (in Russian).
  31. Pulijala Y., Ma M., Pears M., Peebles D., Ayoub A. Effectiveness of immersive virtual reality in surgical training - A randomized control trial, J. Oral Maxillofac. Surg., 2018, vol. 76, no. 5, pp. 1065—1072. DOI: 10.1016/j.joms.2017.10.002.
  32. Yoganathan S., Finch D. A., Parkin E., Pollard J. 360° virtual reality video for the acquisition of knot tying skills: a randomized controlled tria, Int. J. Surg. 54 (Pt A), 2018, pp. 24—27. DOI: 10.1016/j.ijsu.2018.04.002.