Browse > Article
http://dx.doi.org/10.6109/jkiice.2022.26.9.1305

A New Image Processing Scheme For Face Swapping Using CycleGAN  

Ban, Tae-Won (Department of Intelligent Communication Engineering, Gyeongsang National University)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.26, no.9, 2022 , pp. 1305-1311 More about this Journal
Abstract
With the recent rapid development of mobile terminals and personal computers and the advent of neural network technology, real-time face swapping using images has become possible. In particular, the cycle generative adversarial network made it possible to replace faces using uncorrelated image data. In this paper, we propose an input data processing scheme that can improve the quality of face swapping with less training data and time. The proposed scheme can improve the image quality while preserving facial structure and expression information by combining facial landmarks extracted through a pre-trained neural network with major information that affects the structure and expression of the face. Using the blind/referenceless image spatial quality evaluator (BRISQUE) score, which is one of the AI-based non-reference quality metrics, we quantitatively analyze the performance of the proposed scheme and compare it to the conventional schemes. According to the numerical results, the proposed scheme obtained BRISQUE scores improved by about 4.6% to 14.6%, compared to the conventional schemes.
Keywords
Face swapping; face translation; cycle generative adversarial network (cycleGAN); landmarks; dataset;
Citations & Related Records
연도 인용수 순위
  • Reference
1 A. Datta, O. K. Yadav, Y. Singh, S. S, K. M, and S. E, "Real-Time Face Swapping System using OpenCV," in Proceeding of the 3rd International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, pp. 1081-1086, 2021.
2 X. Jin, Y. Qi, and S. Wu, "CycleGAN Face-off," arXiv. [Internet]. Available: https://arxiv.org/abs/1712.03451.
3 J. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," in Proceeding of 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 2242-2251, 2017.
4 dlib C++ Library[Internet]. Available: http://dlib.net/.
5 Y. Zhang, L. Zheng, and V. L. L. Thing, "Automated Face Swapping and Its Detection," in Proceeding of IEEE 2nd International Conference on Signal and Image, Singapore, pp. 15-19, 2017.
6 D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, and S. K. Nayar, "Face Swapping: Automatically Replacing Faces in Photographs," ACM Transactions on Graphics (SIGGRAPH), vol. 27, no. 3, pp. 1-8, Aug. 2008.
7 K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister, "Video Face Replacement," ACM Transactions on Graphics (SIGGRAPH), vol. 30, no. 6, pp. 1-10, Dec. 2011.
8 I. Korshunova, W. Shi, J. Dambre, and L. Theis, "Fast Face-Swap Using Convolutional Neural Networks," in Proceeding of IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 3697-3705, 2017.
9 github, 2020. Github [Internet]. Available: https://github.com/JunHyeok96/DeepFake.
10 Q. Zhang, H. Zheng, T. Yan, and J. Li, "3D Large-Pose Face Alignment Method Based on the Truncated Alexnet Cascade Network," Advances in Condensed Matter Physics, vol. 2020, Article ID 6675014, Dec. 2020.
11 github, 2021. Github [Internet]. Available: https://github.com/nicolalandro/cyclegan-pretrained
12 A. Mittal, A. K. Moorthy, and A. C. Bovik, "No-Reference Image Quality Assessment in the Spatial Domain," IEEE Transactions on Image Processing, vol. 21, no. 12, pp. 4695-4708, Dec. 2012.   DOI
13 [Internet]. Available: https://youtu.be/DXdzTS6ZHhw.
14 github, 2022. Github [Internet]. Available: https://github.com/photosynthesis-team/piq.
15 S. Suwajanakorn, S. M. Seitz, and I. KemelmacherShlizerman, "What Makes Tom Hanks Look Like Tom Hanks," in Proceeding of IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 3952-3960, 2015.