Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Multi-attribute Face Editing using Facial Masks

얼굴 마스크 정보를 활용한 다중 속성 얼굴 편집

  • ;
  • 박인규 (인하대학교 전기컴퓨터공학과) ;
  • 홍성은 (인하대학교 전기컴퓨터공학과)
  • Received : 2022.06.30
  • Accepted : 2022.07.25
  • Published : 2022.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Although face recognition and face generation have been growing in popularity, the privacy issues of using facial images in the wild have been a concurrent topic. In this paper, we propose a face editing network that can reduce privacy issues by generating face images with various properties from a small number of real face images and facial mask information. Unlike the existing methods of learning face attributes using a lot of real face images, the proposed method generates new facial images using a facial segmentation mask and texture images from five parts as styles. The images are then trained with our network to learn the styles and locations of each reference image. Once the proposed framework is trained, we can generate various face images using only a small number of real face images and segmentation information. In our extensive experiments, we show that the proposed method can not only generate new faces, but also localize facial attribute editing, despite using very few real face images.

얼굴 인식 및 얼굴 생성이 다양한 분야에서 큰 주목을 받고 있지만, 얼굴 이미지를 모델 학습에 사용하는데 따른 개인 정보 문제는 최근 큰 문제가 되고 있다. 본 논문에서는 소수의 실제 얼굴 이미지와 안면 마스크 정보로부터 다양한 속성을 가진 얼굴 이미지를 생성함으로써 개인 정보 침해 이슈를 줄일 수 있는 얼굴 편집 네트워크를 제안한다. 다수의 실제 얼굴 영상을 이용하여 얼굴 속성을 학습하는 기존의 방법과 달리 제안하는 방법은 얼굴 분할 마스크와 얼굴 부분 텍스처 영상을 스타일 정보로 사용하여 새로운 얼굴 이미지를 생성한다. 이후 해당 이미지는 각 참조 이미지의 스타일과 위치를 학습하기 위한 훈련에 사용된다. 제안하는 네트워크가 학습되면 소수의 실제 얼굴 영상과 얼굴 분할 정보만을 사용하여 다양한 얼굴 이미지를 생성할 수 있다. 실험에서 제안 기법이 실제 얼굴 이미지를 매우 적게 사용함에도 불구하고 새로운 얼굴을 생성할 뿐만 아니라 얼굴 속성 편집을 지역화하여 수행할 수 있음을 보인다.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2021R1F1A1054569, No. 2022R1A4A1033549). This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-00155915, Artificial Intelligence Convergence Innovation Human Resources Development (Inha University)).

References

  1. Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative adversarial nets," Proceedings of Advances in Neural Information Processing Systems, 2014. doi: https://doi.org/10.48550/arXiv.1406.2661
  2. Wang, X. and Gupta, A., "Generative image modeling using style and structure adversarial networks," Proceedings of European Conference on Computer Vision, pp. 318-335, 2016. doi: https://doi.org/10.1007/978-3-319-10578-9_24
  3. Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434, 2015. doi: https://doi.org/10.48550/arXiv.1511.06434
  4. Choi, Yunjey, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8789-8797, 2018. doi: https://doi.org/10.1109/CVPR.2018.00916
  5. Zhu, Yuhao, Qi Li, Jian Wang, Cheng-Zhong Xu, and Zhenan Sun. "One shot face swapping on megapixels," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4834-4844, 2021. doi: https://doi.org/10.1109/CVPR46437.2021.00480
  6. Korshunova, Iryna, Wenzhe Shi, Joni Dambre, and Lucas Theis. "Fast face-swap using convolutional neural networks," Proceedings of the IEEE International Conference on Computer Vision, pp. 3677-3685, 2017. doi: https://doi.org/10.1109/ICCV.2017.397
  7. Xu, Yangyang, Bailin Deng, Junle Wang, Yanqing Jing, Jia Pan, and Shengfeng He. "High-resolution face swapping via latent semantics disentanglement," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7642-7651, 2022. doi: https://doi.org/10.48550/arXiv.2203.15958
  8. Gu, Shuyang, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan. "Mask-guided portrait editing with conditional gans," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3436-3445, 2019. doi: https://doi.org/10.1109/CVPR.2019.00355
  9. Zhu, Peihao, Rameen Abdal, Yipeng Qin, and Peter Wonka. "Sean: Image synthesis with semantic region-adaptive normalization," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5104-5113, 2020. doi: https://doi.org/10.48550/arXiv.1911.12861
  10. Kim, Hyunsu, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. "Exploiting spatial dimensions of latent in gan for real-time image editing," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 852-861, 2021. doi: https://doi.org/10.48550/arXiv.2104.14754
  11. Zhou, Hang, Jihao Liu, Ziwei Liu, Yu Liu, and Xiaogang Wang. "Rotate-and-render: Unsupervised photorealistic face rotation from single-view images," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5911-5920, 2020. doi: https://doi.org/10.48550/arXiv.2003.08124
  12. Wang, Ting-Chun, Arun Mallya, and Ming-Yu Liu. "One-shot free-view neural talking-head synthesis for video conferencing," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10039-10049, 2021. doi: https://doi.org/10.1109/CVPR46437.2021.00991
  13. Vitadhani, Agastya, Kalamullah Ramli, and Prima Dewi Purnamasari. "Detection of Clickbait Thumbnails on YouTube Using TesseractOCR, Face Recognition, and Text Alteration," Proceedings of International Conference on Artificial Intelligence and Computer Science Technology (ICAICST), pp. 56-61, 2021. doi: https://doi.org/10.1109/ICAICST53116.2021.9497811
  14. Mirza, Mehdi, and Simon Osindero. "Conditional generative adversarial nets." arXiv preprint arXiv:1411.1784, 2014. doi: https://doi.org/10.48550/arXiv.1411.1784
  15. Isola, Phillip, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. "Image-to-image translation with conditional adversarial networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134, 2017. doi: https://doi.org/10.1109/CVPR.2017.632
  16. Wang, Ting-Chun, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. "High-resolution image synthesis and semantic manipulation with conditional gans," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8798-8807, 2018. doi:https://doi.org/10.1109/CVPR.2018.00917
  17. Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. "Image style transfer using convolutional neural networks," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2414-2423, 2016. doi:https://doi.org/10.1109/CVPR.2016.265
  18. Huang, Xun, and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization," Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017. doi: https://doi.org/10.48550/arXiv.1703.06868
  19. Karras, Tero, Timo Aila, Samuli Laine, and Jaakko Lehtinen. "Progressive Growing of GANs for Improved Quality, Stability, and Variation," Proceedings of International Conference on Learning Representations, 2018. doi: https://doi.org/10.48550/arXiv.1710.10196
  20. Grassucci, Eleonora, Edoardo Cicero, and Danilo Comminiello. "Quaternion generative adversarial networks," In Generative Adversarial Learning: Architectures and Applications, pp. 57-86. Springer, 2022. doi: https://doi.org/10.1007/978-3-030-91390-8_4