Journal "Software Engineering"
a journal on theoretical and applied science and technology
ISSN 2220-3397

Issue N1 2025 year

DOI: 10.17587/prin.16.28-38
Software Complex for Documenting Graphical Applications Based on Topological Decomposition
A. V. Abakumov, Graduate Student, artem210966@yandex.ru, Murom Institute of Vladimir State University, Murom, 602264, Russian Federation
Corresponding author: Artem V. Abakumov, Graduate Student, Murom Institute of Vladimir State University, Murom, 602264, Russian Federation E-mail: artem210966@yandex.ru
Received on September 06, 2024
Accepted on October 30, 2024

Contemporary software products and services necessitate extensive accompanying information, including user manuals and documentation. The task becomes more complex in the case of graphical applications, as it requires capturing screenshots and highlighting the described elements. Attaining high-quality visual information using conventional means demands considerable effort and time, since, unlike text, raster images do not provide an interface for retrieving metadata about individual elements, necessitating programmatic extraction of such information. This study presents a software complex designed to automate and expedite this process. The primary focus is on a topo­logical decomposition method that segments images based on topological features such as pixel brightness and proximity, enabling the segmentation of screenshots into control elements. This approach demonstrates high accuracy (96-99 %) in identifying graphical components across most tested images. The software product encompasses functions for the automatic capture of application windows, an algorithm for identifying control elements, and the capability to export the entire image or a specified region. Additionally, an algorithm for generating output templates for image insertion has been implemented. The decomposition method has exhibited effectiveness across the majority of test data. In instances with shadow effects, noise levels were recorded at a maximum of 34 %; however, in most cases, this figure did not exceed 3 %. Further research may contribute to enhancing these results. The findings of this study could prove valuable for the continued exploration of graphical user interface decomposition and the assessment of the method's effectiveness on diverse images. The developed software complex significantly simplifies and accelerates the documentation process, thereby opening new avenues in the automation of software development.

Keywords: image processing, documentation, persistent homology, image decomposition, graphical user interface
pp. 28—38
For citation:
Abakumov A. V. Software Complex for Documenting Graphical Applications Based on Topological Decomposition, Programmnaya Ingeneria, 2025, vol. 16, no. 1, pp. 28—38. DOI: 10.17587/prin.16.28-38. (in Russian).
This study was supported by the Russian Science Foundation, project no. 23-21-10064, https://rscf.ru/project/23-21-10064/.
References:
  1. Macchi F., Rosin P., Mervi J., Turchet L. Image-based approaches for automating GUI testing of interactive web-based applications, 28th Conference of Open Innovations Association (FRUCT), Moscow, Russia, 2021, pp. 278—285. DOI: 10.23919/ FRUCT50888.2021.9347592.
  2. Koltsov P. P., Osipov A. S., Kutsaev A. S. et al. On the quantitative performance evaluation of image analysis algorithms, Computer optics, 2015, vol. 39, no. 4, pp. 542—556. DOI: 10.18287/0134-2452-2015-39-4-542-556 (in Russian).
  3. Ganesan P., Sajiv G. A comprehensive study of edge detection for image processing applications, 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India, 2017, pp. 1—6. DOI: 10.1109/ICIIECS.2017.8275968.
  4. Subr K., Soler C., Durand F. Edge-preserving multiscale image decomposition based on local extrema, ACM Transactions on Graphics, 2009, vol 28, no. 5, pp. 1—9. DOI: 10.1145/1618452.161849.
  5. Guo T., Zhang T., Lim E. et al. A review of wavelet analysis and its applications: Challenges and opportunities, IEEE Access, 2022, no. 10, pp. 58869—58903. DOI: 10.1109/AC-CESS.2022.3179517.
  6. Oh H.-S., Choi G., Kim D. Decomp2d: An R solution for image decomposition, SoftwareX, 2022, no. 19, 101182 p. DOI: 10.1016/j.softx.2022.101182.
  7. Du S., Liu Y., Zhao M. et al. A new image decomposition approach using pixel-wise analysis sparsity model, Pattern Recognition, 2023, no. 136, 109241 p. DOI: 10.1016/j.patcog.2022. 109241.
  8. Chen Q., Koltun V. A simple model for intrinsic image decomposition with depth cues, Stanford University, Adobe Research, 2013, pp. 241—248. DOI: 10.1109/ICCV.2013.37.
  9. Rukhovich D. D. Iterative Scheme for Object Detection in Crowded Environments, Programmnaya Ingeneria, 2021, vol. 12, no. 1, pp. 31—39. DOI: 10.17587/prin.12.31-39 (in Russian).
  10. Li Z., Snavely N. Learning intrinsic image decomposition from watching the world, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018, pp. 9039—9048. DOI: 10.1109/CVPR.2018.00942/.
  11. Zhou F., Chen Q., Liu B. tructure and texture-aware image decomposition via training a neural network, IEEE Transactions on Image Processing, 2020, no. 29, pp. 3458—3473. DOI: 10.1109/ TIP.2019.2961232.
  12. Feng F., Hu Y., Li W. Improved YOLOv8 algorithms for small object detection in aerial imagery, Journal of King Saud University — Computer and Information Sciences, 2024, vol. 36, no. 6, pp. 102113. DOI: 10.1016/j.jksuci.2024.102113.
  13. Papernot N., McDaniel P., Jha S. et al. The Limitations of Deep Learning in Adversarial Settings, IEEE European Symposium on Security and Privacy (EuroS&P), Saarbrucken, Germany, 2016, pp. 372—387. DOI: 10.1109/EuroSP.2016.36.
  14. Stefanos T. Limitations of Deep Neural Networks: a dis­cussion of G. Marcus' critical appraisal of deep learning, arxiv. 2020.15754. DOI: 10.48550/arXiv.2012.15754.
  15. Wiyatno R., Xu A., Dia O., Berker A. Adversarial Exam­ples in Modern Machine Learning: A Review, arxiv.2019.05268. DOI: 10.48550/arXiv.1911.05268.
  16. Bons A., Marin B., Aho P., Vos T. E. J. Scripted and scriptless GUI testing for web applications: An industrial case, Information and Software Technology, 2023, no. 158, рр. 107172. DOI: 10.1016/j.infsof.2023.107172.
  17. Mariani L., Pezze M., Zuddas D. Recent advances in automatic black-box testing, Advances in Computers, 2015, no. 99, pp. 157—193. DOI: 10.1016/bs.adcom.2015.04.002.
  18. Honfi D., Micskei Z. White-box software test generation with Microsoft Pex on open source C# projects: A dataset, Data in Brief, 2020, no. 31. DOI: 10.1016/j.dib.2020.105962.
  19. Yuanchun L., Ziyue Y., Yao G. et al. Humanoid: A Deep Learning-based Approach to Automated Black-box Android App Testing, arXiv.2019. 02633. DOI: 10.48550/arXiv.1901.02633.
  20. Thomas D., White G., Guy J. Improving random GUI testing with image-based widget detection, Proceedings of the 28th ACM SIG-SOFT International Symposium on Software Testing and Analysis (ISSTA 2019), Beijing, China, 2019, pp. 307—317. DOI: 10.1145/3293882.333055/.
  21. Eremeev S. V., Abakumov A. V., Andrianov D. E., Titov D. V. Image decomposition method by topological features, Computer optics, 2022, vol. 46, no. 6, pp. 939—947. DOI: 10.18287/2412-6179-CO-1080 (in Russian).
  22. Eremeev S. V., Abakumov A. V., Andrianov D. E. et al. Detection of perennial heaving mounds in digital elevation model images using decomposition by topological features, Sovremennye problemy distantsionnogo zondirovaniya zemli iz kosmosa, 2023, vol. 20, no. 6, pp. 129—143. DOI: 10.21046/2070-7401-2023-20-6-129-143(in Russian).
  23. Abakumov A., Eremeev S. Segmentation of Graphical User Interface Elements Based on Topological Decomposition for GUI Testing Tasks, Data Analytics and Management in Data Intensive Domains. DAMDID/RCDL 2023. Communications in Computer and Information Science, 2024, no. 2086. DOI: 10.1007/978-3-031-67826-4_14.
  24. Carlson G. Topology and data, Bulletin (New Series) of the American Mathematical Society, 2009, vol. 46, no. 2, pp. 255—308. DOI:10.1090/S0273-0979-09-01249-X.
  25. Roy H., Shafiuzzaman М., Samsuddoha М. SRCS: A new proposed counting sort algorithm based on square root method, 2019 22nd International Conference on Computer and Information Technology (ICCIT), Dhaka, Bangladesh, 2019, pp. 1—6. DOI: 10.1109/ICCIT48885.2019.9038601.