DOI: 10.17587/prin.14.82-92

Non-Rigid Registration of a Human Face from Images from a Stereo Camera

A. D. Borisov, Postgraduate Student, radiatus@yandex.ru, S. D. Makhortov, Professor, Head of Department of Programming and Information Technologies, msd_exp@outlook.com, Voronezh State University, Voronezh, 394018, Russian Federation

Corresponding author: Sergey D. Makhortov, Professor, Head of Department of Programming and Information Technologies, Voronezh State University, Voronezh, 394018, Russian Federation, E-mail: msd_exp@outlook.com

Capturing the facial expressions of an actor on the set to further transfer the scene to the digital space has become one of the most important tasks in the field of computer graphics. Most of the methods for solving this problem have a lower quality of reconstruction in comparison with the capture, which is carried out in videogrammetric installations. The reason is the inability to calculate high-quality three-dimensional scanned copies ("scans"). This article highlights the problematic issues of existing approaches and proposes a solution for capturing an actors facial expressions from images from a helmets stereo camera, which is comparable to stationary capture. To obtain this level of accuracy, it is proposed to use a local deformation model based on the geometries of key facial expressions, as well as stereo restriction, instead of explicitly calculating a 3D scan.

Keywords: soft registration, stereo camera, computer graphics, motion capture, digital doubles

pp. 82–92

For citation:
Borisov A. D., Makhortov S. D. Non-Rigid Registration of a Human Face from Images from a Stereo Camera, Programmnaya Ingeneria, 2023, vol. 14, no. 2, pp. 82—92. DOI: 10.7587/prin.14.82-92 (in Russian).

References:

1. Borshukov G., Piponi D., Larsen O. et al. Image-based Facial Animation for «The Matrix Reloaded», ACM SIGGRAPH computer graphics, N. Y., ACM, 2003, pp. 16. DOI: 10.1145/1198555.1198596.
2. Garrido P., Valgaert L., Wu C. et al. Reconstructing detailed dynamic face geometry from monocular video, ACM SIGGRAPH computer graphics, N. Y., ACM, 2013, pp. 1—10. DOI: 10.1145/2508363.2508380.
3. Debevec P., Hawkins T., Tchou C. et al. Acquiring the Re­flectance Field of a Human Face, ACM SIGGRAPH computer graphics. N. Y., ACM, 2020, pp. 145—156. DOI: 10.1145/344779.344855.
4. Hao L., Sumner R., Pauly M. et al. Global correspondence optimization for non-rigid registration of depth scans, Eurographics Symposium on Geometry Processing, N. Y., ACM, 2008, pp. 1421—1430.
5. Casas D., Tejera M., Guillemaut J. et al. Interactive Anima­tion of 4D Performance Capture, IEEE Transactions on Visualization and Computer Graphics, IEEE 2013, pp. 762—773. DOI: 10.1109/TVCG.2012.314.
6. Hongwei X., Leijia D., Jianxing F. et al High-Quality Real TIme Facial Capture Based on Single Camera, Creative Commons Attribution 4.0International, arXiv, 2021, pp. 1—9. DOI: 10.48550/arXiv.2111.07556.
7. Serra J., Moser L., McLean D. et al. Simplified facial capture with head mounted cameras, ACM SIGGRAPH computer graphics, N. Y., ACM, 2021, pp. 1—2. DOI: 10.1145/3450623. 3464637.
8. Eugen A., Bouaziz S., Pauly M. Dynamic 3D avatar cre­ation from hand-held video input, ACM SIGGRAPH computer graphics, N. Y., ACM, 2015, pp. 1—14. DOI: 10.1145/2766974.
9. Fyffe G., Nagano K., Huynh L. et al. Multi-View Stereo on Consistent Face Topology, ACM SIGGRAPH computer graphics, N. Y., ACM, 2017, pp. 1—2. DOI: 10.1111/cgf.13127.
10. Weinzaepfel P., Revaud J., Harchaoui Z. et al. DeepFlow: Large displacement optical flow with deep matching, IEEE International Conference on Computer Vision, IEEE 2013, pp. 1385—1392. DOI: 10.1109/ICCV.2013.175.
11. Brox T., Malik J. Large displacement optical flow: descrip­tor matching in variational motion estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE 2011, pp. 500—513. DOI: 10.1109/TPAMI.2010.143.
12. Sun D., Roth S., Black M. Secrets of optical flow estimation and their principles, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 2010, pp. 2432—2439. DOI: 10.1109/CVPR.2010.5539939.
13. Beeler T., Hahn F., Bradley D. et al. High-Quality Pas­sive Facial Performance Capture using Anchor Frames, ACM SIGGRAPH computer graphics, N. Y.: ACM, 2011, pp. 1—10. DOI: 10.1145/2010324.1964970.
14. Borisov A. D., Solomatin D. I. Stereo Constraints for the Problem of Non-Rigid Face Registration, Trudy molodyh uchyonyh fakulteta kompyuternyh nauk VGU, 2022, pp. 219—223 (in Russian).
15. Borisov A. D., Solomatin D. I. Razrabotka personalizirovan-nogo detektora kluchevyh nocher litsa, Trudy molodyh uchyonyh fakulteta kompyuternyh nauk VGU, 2020, pp. 16—22 (in Russian).
16. Gabriel P., Cohen L. Geodesic Methods for Shape and Surface Processing, Advances in Computational Vision and Medical Image Processing: Methods and Applications, Springer Verlag, 2009, pp. 29—56. DOI: 10.1007/978-1-4020-9086-8.
17. Beeler T., Bickel B., Beardsley P. et al. High-quality single-shot capture of facial geometry, ACM SIGGRAPH comput­er graphics, N. Y., ACM, 2010, pp. 1—9. DOI: 10.1145/1778765. 1778777.
18. Biancolini M. Fast RBF for Engineering Applications, Fast radial basis functions for engineering applications, Springer Interna­tional Publishing, 2018, pp. 7—78.
19. Sumner R. Mesh Modification Using Deformation Gradients, Department of Electrical Engineering and Computer Science, 2005, рр. 1—29.
20. Andelson E, Anderson C., Bergen J. et al. Pyramid methods in image processing, ACM SIGGRAPH computer graphics, N. Y., ACM, 1983, pp. 1—9.
21. Madsen K., Nielsen H., Tingleff O. Methods for Non-Linear Least Square Problems, Cobenhavn, Technical University of Denmark, 2004, рр. 1—30.