Browse > Article
http://dx.doi.org/10.9717/kmms.2019.22.5.535

Comparison Analysis of Four Face Swapping Models for Interactive Media Platform COX  

Jeon, Ho-Beom (Dept of IT Convergence and Application Engineering, Pukyong University)
Ko, Hyun-kwan (Dept of IT Convergence and Application Engineering, Pukyong University)
Lee, Seon-Gyeong (Dept of IT Convergence and Application Engineering, Pukyong University)
Song, Bok-Deuk (Electronics and Telecommunications Research Institute)
Kim, Chae-Kyu (Dept of IT Convergence and Application Engineering, Pukyong University)
Kwon, Ki-Ryong (Dept of IT Convergence and Application Engineering, Pukyong University)
Publication Information
Journal of Korea Multimedia Society / v.22, no.5, 2019 , pp. 535-546 More about this Journal
Abstract
Recently, there have been a lot of researches on the whole face replacement system, but it is not easy to obtain stable results due to various attitudes, angles and facial diversity. To produce a natural synthesis result when replacing the face shown in the video image, technologies such as face area detection, feature extraction, face alignment, face area segmentation, 3D attitude adjustment and facial transposition should all operate at a precise level. And each technology must be able to be interdependently combined. The results of our analysis show that the difficulty of implementing the technology and contribution to the system in facial replacement technology has increased in facial feature point extraction and facial alignment technology. On the other hand, the difficulty of the facial transposition technique and the three-dimensional posture adjustment technique were low, but showed the need for development. In this paper, we propose four facial replacement models such as 2-D Faceswap, OpenPose, Deekfake, and Cycle GAN, which are suitable for the Cox platform. These models have the following features; i.e. these models include a suitable model for front face pose image conversion, face pose image with active body movement, and face movement with right and left side by 15 degrees, Generative Adversarial Network.
Keywords
Face Swap; OpenPose; Auto Encoder; Deepfake; Deep Learning; GAN; CyecleGAN;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 B.D. Song, S.Y. Kim, and C.K. Kim, "Design and Implementation of the Image Creation System Based on User-Media Interaction," Journal of Korea Multimedia Society, Vol. 19, No. 3, pp. 932-938, 2016.   DOI
2 J.H. Kim, J.L. Ha, B.S. Choi, Y.H. Jeong, and J.W. Hong, "Design and Implementation for Interactive Augmented Broadcasting System," IEEE Transactions on Broadcasting, Vol. 60, No. 2, pp. 217-226, 2014.   DOI
3 I.J. Lee, H.K. Lee, J.W. Hong, and J.H. Cha, "Interactive Contents Player for Rich Media Service," IEEE Transactions on Consumer Electronics, Vol. 55, No. 1, pp. 112-118, 2009.   DOI
4 S.K. Kim, S.H. Moon, J. Park, J.C. Chang, and S.Y. Han, "A Study on Interactive Storytelling Script Language for Generating the Stories," Journal of Korea Multimedia Society, Vol. 12, No. 2, pp. 313-322, 2009.
5 P. Garrido, L. Valgaerts, O. Rehmsen, T. Thormahlen, P. Perez, C. Theobalt, et al., "Automatic Face Reenactment," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4217-4224, 2014.
6 Y. Nirkin, I. Masi, A.T. Tran, T. Hassner, and G. Medioni, "On Face Segmentation, Face Swapping, and Face Perception," Proceedings of 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 98-105, 2018.
7 Switching Eds: Face swapping with Python, dlib, and OpenCV, http://matthewearl.github.io/2015/07/28/switching-eds-with-python/(accessed April, 09, 2019).
8 T. Hassner, "Viewing Real-World Faces in 3D," Proceeding of the IEEE International Conference on Computer Vision, pp. 3607-3614, 2013.
9 J.S. Song, X. Zhang, and J.I. Park, "Face Replacement under Different Illumination Condition," Journal of Business Ethics Special Issue Conference, Vol. 20, No. 4, pp. 606-618, 2015.
10 T. Cho, J.P. Jeong, and S.M Choi, "3D Emotional Avatar Creation and Animation Using Facial Expression Recognition," Journal of Korea Multimedia Society, Vol. 17, No. 9, pp. 1076-1083, 2014.   DOI
11 V. Blanz and T. Vetter, "A Morphable Model for the Synthesis of 3-D Faces," Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187-194, 1999.
12 T. Hassner, S. Harel, E. Paz, and R. Enbar, "Effective Face Frontalization in Unconstrained Images," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4295-4304, 2015.
13 Faceswap-GAN, GitHub, https://github.com/shaoanlu/faceswap-GAN (accessed April, 09, 2019).
14 Deepfakes-faceswap, GitHub, https://github.com/deepfakes/faceswap (accessed April, 09, 2019).
15 J.M. Saragih, S. Lucey, and J.F. Cohn, "Face Alignment through Subspace Constrained Mean-Shifts," Proceedings of 2009 IEEE 12th International Conference on Computer Vision IEEE, pp. 1034-1041, 2009.
16 V. Kazemi and J. Sullivan, "One Millisecond Face Alignment with An Ensemble of Regression Trees," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867-1874, 2014.
17 J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. NieBner, "Face2Face: Realtime Face Capture and Reenactment of RGBVideos," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387-2398, 2016.
18 I. Korshunova, W. Shi, J. Dambre, and L. Theis, "Fast Face-Swap Using Convolutional Neural Networks," Proceedings of the IEEE International Conference on Computer Vision, pp. 3677-3685, 2017.
19 R. Natsume, T. Yatagawa, and S. Morishima, "RSGAN: Face Swapping and Editing Using Face and Hair Representation in Latent Spaces," ArXiv Preprint ArXiv:1804.03447, 2018.
20 H. Dong, P. Neekhara, C. Wu, and Y. Guo, "Unsupervised Image-to-Image Translation with Generative Adversarial Networks," ArXiv Preprint ArXiv:1701.02676, 2017.
21 D.E. King, "Dlib-ml: A Machine Learning Toolkit," Journal of Machine Learning Research, Vol. 10, Issue 7, pp. 1755-1758, 2009.
22 J. Ahlberg, CANDIDE-3 - An Updated Parameterized Face, Report No. LiTH-ISY-R-2326, Department of Electrical Engineering, Linkoping University, Sweden, 2001.
23 F.D. Foresee and M.T. Hagan, "Gauss-Newton Approximation to Bayesian Learning," Proceedings of the 1997 International Joint Conference on Neural Networks, Vol. 3, pp. 1930-1935, 1997.
24 R.A. Hummel, B.B. Kimia, and S.W. Zucker, "Deblurring Gaussian Blur," Computer Vision, Graphics, and Image Processing, Vol. 38, No. 1, pp. 66-80, 1987.   DOI
25 K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, "Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks." IEEE Signal Processing Letters, Vol. 23, No. 10, pp. 1499-1503, 2016.   DOI
26 J.Y. Zhu, T. Park, P. Isola, and A.A. Efros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," Proceedings of IEEE International Conference on Computer Vision, pp. 2242-2251, 2017.
27 Z. Cao, T. Simon, S.E. Wei, and Y. Sheikh, "OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields," arXiv eprint arXiv:1812.08008, 2018.
28 Faceswap, GitHub, https://github.com/MarekKowalski/FaceSwap, (accessed April, 09, 2019).
29 I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair Bengio, et al., "Generative Adversarial Nets," NIPS'14 Proceedings of the 27th International Conference on Neural Information Processing Systems. Vol. 2. pp. 2672-2680, 2014.
30 I. Guyon, G. Dror, V. Lemaire, G. Taylor, and D. Silver, "Autoencoders, Unsupervised Learning, and Deep Architectures," Journal of Machine Learning Research : Workshop and Conference Proceedings, Vol. 27, pp. 37-50, 2012.