A. V. Savchenko, Doctor of Sciences, Professor, e-mail: avsavchenko@hse.ru, I. S. Grechikhin, Postgraduate Student, Senior Lecturer, e-mail: igrechikhin@hse.ru, National Research University Higher School of Economics, Nizhny Novgorod, Russian Federation

Detection of Specialized Object Categories in Photos from Mobile Device Based on a Multi-Task Neural Network

In this paper we consider the task of user preferences analysis for recommender engines based on a gallery of his or her mobile device. In particular, we propose the novel three-phase method for simultaneous image-based detection and recognition of particular objects. Conventional object detection techniques cannot be applied if there are many categories of the same object (pet breeds, car models, etc.) and there is à of large dataset with Mown bounding boxes for each object category. In order to deal with this issue, we estimate the borders of base objects (dogs, cats, cars, etc.) by using such existing neural network  architectures as high precision Faster R- CNN or fast single-shot detectors. Secondly, the visual features (embeddings) of each object are extracted by using a multi-task convolutional neural network model with several outputs — one for each type of object. Finally, these embeddings are used to predict the concrete categories and group different photos of the same object by using cluster analysis techniques. The proposed approach is implemented in a special mobile application for Android. Experimental results for recognizing dog and cat breeds are presented. It is demonstrated that our method makes it possible to improve the accuracy of dog detection and recognition when compared to the Mown single-task neural nets. Moreover, we gather a special dataset of real photos with pets to estimate the clustering quality. It is shown that the L1-normed features extracted by our multi-task model may be grouped rather accurately if hierarchical agglomerative clustering or HDBSCAN method are used.
Keywords: image processing, convolutional neural networks, mobile systems, pet breed recognition, hierarchical clustering, multi-task learning, object detection

P. 586–593

Acknowledgments. The article was prepared within the framework of the Academic Fund Program at the National Research University Higher School of Economics (HSE University) in 2019-2020 (grant No. 19-04-004) and by the Russian Academic Excellence Project "5-100"