Browse > Article
http://dx.doi.org/10.5909/JBE.2019.24.3.440

A System for 3D Face Manipulation in Video  

Park, Jungsik (Department of Computer Science, Hanyang University)
Seo, Byung-Kuk (Electronics and Telecommunications Research Institute)
Park, Jong-Il (Department of Computer Science, Hanyang University)
Publication Information
Journal of Broadcast Engineering / v.24, no.3, 2019 , pp. 440-451 More about this Journal
Abstract
We propose a system that allows three dimensional manipulation of face in video. The 3D face manipulation of the proposed system overlays the 3D face model with the user 's manipulation on the face region of the video frame, and it allows 3D manipulation of the video in real time unlike existing applications or methods. To achieve this feature, first, the 3D morphable face model is registered with the image. At the same time, user's manipulation is applied to the registered model. Finally, the frame image mapped to the model as texture, and the texture-mapped and deformed model is rendered. Since this process requires lots of operations, parallel processing is adopted for real-time processing; the system is divided into modules according to functionalities, and each module runs in parallel on each thread. Experimental results show that specific parts of the face in video can be manipulated in real time.
Keywords
Face manipulation; face editing; video editing; face tracking; mesh deformation;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Virtual Plastic Surgery Simulator, https://www.plastic-surgery-simulator.com (accessed July 30, 2018)
2 Snow.me, https://snow.me (accessed July 30, 2018)
3 H.-V. Chung and I.-K. Lee, "Image-Based Deformation of Objects in R eal Scenes," Proceedings of International Symposium on Visual Computing, pp.159-166, 2005, https://doi.org/10.1007/11595755_ 20.
4 J. Park, B.-K. Seo and J.-I. Park, "[Poster] Interactive Deformation of Real Objects," Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.295-296, 2014, https://doi.org/10.1109/ISMAR.2014.6948457.
5 Y. Zheng, X. Chen, M.-M. Cheng, K. Zhou, S.-M. Hu, and N. J. Mitra, "Interactive Images: Cuboid Proxies for Smart Image Manipulation," ACM Transactions on Graphics, Vol.31, No.4, pp.99:1-99:11, Jul. 2012, https://doi.org/10.1145/2185520.2185595.   DOI
6 T. Chen, Z. Zhu, A. Shamir, S.-M. Hu, and D. Cohen-Or, "3-Sweep: E xtracting Editable Objects from a Single Photo," ACM Transactions on Graphics, Vol.32, No.6, pp.195:1-195:10, Nov. 2013, https://doi.org/10.1145/2508363.2508378.   DOI
7 N. Kholgade, T. Simon, A. Efros, and Y. Sheikh, "3D Object Manipula tion in a Single Photograph Using Stock 3D Models," ACM Transactions on Graphics, Vol.33, No.4, pp.127:1-127:12, Jul. 2014, https://doi.org/10.1145/2601097.2601209.   DOI
8 K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister, "Video Face Replacement," ACM Transactions on Graphics, Vol.30, No.6, pp.130:1-130:10, Dec. 2011, https://doi.org/10.1145/2070781.2024164.   DOI
9 N. Haouchine, A. Petit, F. Roy, and S. Cotin, "[Poster] Deformed Reality: Proof of Concept and Preliminary Results," Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.166-167, 2017, https://doi.org/10.1109/ISMAR-Adjunct.2017.56.
10 J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Niessner, "Face2Face: Real-Time Face Capture and Reenactment of RGB Videos," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2387-2395, 2016, https://doi.org/10.1109/CVPR.2016.262.
11 J. Park and J.-I. Park, "A Framework for Virtual 3D Manipulation of Fa ce in Video," Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp.649-650, 2018, https://doi.org/10.1109/VR.2018.8446445.
12 C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou, "FaceWarehouse: A 3D Facial Expression Database for Visual Computing," IEEE Transactions on Visualization and Computer Graphics, Vol.20, No.3, pp.413-425, Mar. 2014, https://doi.org/10.1109/TVCG.2013.249.   DOI
13 C. Cao, H. Wu, Y. Weng, T. Shao, and K. Zhou, "Real-Time Facial Animation with Image-Based Dynamic Avatars," ACM Transactions on Graphics, Vol.35, No.4, pp.126:1-126:12, Jul. 2016, https://doi.org/10.1145/2897824.2925873.   DOI
14 K. Nagano, J. Seo, J. Xing, L.Wei, Z. Li, S. Saito, A. Agarwal, J. Fursund, and H. Li, "paGAN: Real-Time Avatars Using Dynamic Textures," ACM Transactions on Graphics, Vol.37, No.6, pp.258:1-258:12, Dec. 2018, https://doi.org/10.1145/3272127.3275075.   DOI
15 S. Ren, X. Cao, Y. Wei, and J. Sun, "Face Alignment at 3000 FPS via Regressing Local Binary Features," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1685-1692, 2014, https://doi.org/10.1109/CVPR.2014.218.
16 A. Telea, "An image inpainting technique based on the fast marching method," Journal of Graphics Tools, Vol.9, No.1, pp.23-34, 2004, https://doi.org/10.1080/10867651.2004.10487596.   DOI
17 V. Kazemi and J. Sullivan, "One Millisecond Face Alignment with an Ensemble of Regression Trees," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1867-1874, 2014, https://doi.org/10.1109/CVPR.2014.241.
18 M. Kowalski, J. Naruniec, and T. Trzcinski, "Deep Alignment Networ k: A Convolutional Neural Network for Robust Face Alignment," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
19 P. Huber, G. Hu, R. Tena, P. Mortazavian, P. Koppen, W. J. Christmas, M. Ratsch, and J. Kittler, "A Multiresolution 3D Morphable Face Mod el and Fitting Framework," Proceedings of the 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2016.
20 O. Sorkine and M. Alexa, "As-Rigid-As-Possible Surface Modeling," Proceedings of the 5th Eurographics Symposium on Geometry Processing, pp.109-116, 2007.