Browse > Article
http://dx.doi.org/10.3837/tiis.2021.04.016

Eyeglass Remover Network based on a Synthetic Image Dataset  

Kang, Shinjin (School of Games, Hongik University)
Hahn, Teasung (NCsoft)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.4, 2021 , pp. 1486-1501 More about this Journal
Abstract
The removal of accessories from the face is one of the essential pre-processing stages in the field of face recognition. However, despite its importance, a robust solution has not yet been provided. This paper proposes a network and dataset construction methodology to remove only the glasses from facial images effectively. To obtain an image with the glasses removed from an image with glasses by the supervised learning method, a network that converts them and a set of paired data for training is required. To this end, we created a large number of synthetic images of glasses being worn using facial attribute transformation networks. We adopted the conditional GAN (cGAN) frameworks for training. The trained network converts the in-the-wild face image with glasses into an image without glasses and operates stably even in situations wherein the faces are of diverse races and ages and having different styles of glasses.
Keywords
Image-to-image Translation; Generative Adversarial Network; Eyeglass Remover;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Kim, M. Kim, H. Kang, and K. Lee, "U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-instance Normalization for Image-to-image Translation," arXiv preprint arXiv:1907.10830, 2020.
2 A. B. Larsen, S. K. Sonderby, H. Larochelle, and O. Winther, "Autoencoding Beyond Pixels Using a Learned Similarity Metric," in Proc. of International Conference on Machine Learning, pp. 1558-1566, 2016.
3 S. Zhou, T. Xiao, Y. Yang, D. Feng, Q. He, and W. He, "Genegan: Learning Object Transfiguration and Attribute Subspace from Unpaired Data," in Proc. of British Machine Vision Conference, 2017.
4 B. Hu, Z. Zheng, P. Liu, W. Yang, and M. Ren, "Unsupervised Eyeglasses Removal in the Wild," IEEE Transactions on Cybernetics, 2020.
5 M. Zhao, Z. Zhang, X. Zhang, L. Zhang, and B. Li, "Eyeglasses Removal Based on Attributes Detection and Improved TV Restoration Model," Multimedia Tools and Applications, vol. 80, no. 2, pp. 2691-2712, 2021.   DOI
6 J. A. Buolamwini, "Gender Shades: Intersectional Phenotypic and Demographic Evaluation of Face Datasets and Gender Classifiers," Massachusetts Institute of Technology, 2017.
7 M. Liu, Y. Ding, M. Xia, X. Liu, E. Ding, W. Zuo, and S. Wen, "Stgan: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3668-3677, 2019.
8 J. Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," in Proc. of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
9 Timestamp Camera, Artify. [Online]. Available: https://play.google.com/store/apps/details?id=com.artifyapp.timestamp&hl=en_US
10 Z. Liu, P. Luo, X. Wang, and X. Tang, "Large-scale Celebfaces Attributes (celeba) Dataset," Aug. 2018.
11 A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, and A. Courville, "Augmented Cyclegan: Learning Many-to-many Mappings from Unpaired Data," ArXiv preprint arXiv:1802.10151, 2018.
12 E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or, "Encoding in Style: a Stylegan Encoder for Image-to-image Translation," ArXiv preprint arXiv:2008.00951, 2020.
13 M. Liang, Y. Xue, K. Xue, and A. Yang, "Deep Convolution Neural Networks for Automatic Eyeglasses Removal," DEStech Transactions on Computer Science and Engineering, 2017.
14 G. Perarnau, J. Weijer, J. Raducanu, and J. M. A lvarez, "Invertible Conditional Gans for Image Editing," ArXiv preprint ArXiv:1611.06355, 2016.
15 Y. Choi, M. Choi, M. Kim, J. Ha, S. Kim, and J. Choo, "Stargan: Unified Generative Adversarial Networks for Multi-domain Image-to-image Translation," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789-8797, 2018.
16 Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen, "Attgan: Facial Attribute Editing by Only Changing What You Want," IEEE Transactions on Image Processing, 2019.
17 G. Zhang, M. Kan, S. Shan, and X. Chen, "Generative Adversarial Network with Spatial Attention for Face Attribute Editing," in Proc. of European Conference on Computer Vision, vo1. 11210, pp. 422-437, 2018.
18 Y. Chen, X. Shen, Z. Lin, X. Lu, I. Pao, and J. Jia, "Semantic Component Decomposition for Face Attribute Manipulation," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 9851-9859, 2019.
19 T. Xiao, J. Hong, and J. Ma, "Elegant: Exchanging Latent Encodings with Gan for Transferring Multiple Face Attributes," in Proc. of European Conference on Computer Vision, 2018.
20 T. Kim, B. Kim, M. Cha, and J. Kim, "Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks," ArXiv preprint ArXiv:1707.09798, 2017.
21 Y. Chen, H. Lin, M. Shu, R. Li, X. Tao, X. Shen, Y. Ye, and J. Jia, "Faceletbank for Fast Portrait Manipulation," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3541-3549, 2018.
22 N. Din, K. Javed, S. Bae, and J. Yi, "Effective Removal of User-Selected Foreground Object from Facial Images Using a Novel GAN-Based Network," IEEE Access, vol. 8, pp. 109648-109661, 2020.   DOI
23 J. S. Park, Y. H. Oh, S. C. Ahn, and S. W. Lee, "Glasses Removal from Facial Image Using Recursive Error Compensation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 805-811, 2005.   DOI
24 W. K. Wong and H. Zhao, "Eyeglasses Removal of Thermal Image based on Visible Information," Information Fusion, vol. 14, no. 2, pp. 163-176, 2013.   DOI
25 Y. H. Lee and S. H. Lai, "ByeGlassesGAN: Identity Preserving Eyeglasses Removal for Face Images," in Proc. of European Conference on Computer Vision, vol. 12374, pp. 243-258, 2020.
26 Dlib. [Online]. Aviailable: http://dlib.net/
27 R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-cam: Visual Explanations from Deep Networks via Gradient-based Localization," in Proc. of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
28 K. Karkkainen and J. Joo, "Fairface: Face Attribute Dataset for Balanced Race, Gender, and Age," ArXiv preprint arXiv:1908.04913, 2019.
29 W. Cao, V. Mirjalili, and S. Raschka, "Rank-Consistent Ordinal Regression for Neural Networks," Pattern Recognition Letters, pp. 325-331, Nov. 2020.
30 Snapchat. [Online]. Available: https://play.google.com/store/apps/details?id=com.snapchat.android&hl=ko
31 C. H. Lee, Z. Liu, L. Wu, and P. Luo, "Maskgan: Towards Diverse and Interactive Facial Image Manipulation," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549-5558, 2020.
32 Y. Lu, Y. W. Tai, and C. K. Tang, "Conditional Cyclegan for Attribute Guided Face Image generation," ArXiv preprint arXiv:1705.09966, 2017.
33 B. Hu, Z. Zheng, P. Liu, W. Yang, and M. Ren, "Unsupervised Eyeglasses Removal in the Wild," IEEE Transactions on Cybernetics, 2020.
34 G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. Denoyer, and M. A. Ranzato, "Fader Networks: Manipulating Images by Sliding Attributes," in Proc. of the 31st Conference on Neural Information Processing Systems, pp. 5967-5976, 2018.
35 T. Xiao, J. Hong, and J. Ma, "Dna-gan: Learning Disentangled Representations from MultiAttribute Images," in Proc. of International Conference on Learning Representations Workshops, 2018.
36 W. Yin, Z. Liu, and C. C. Loy, "Instance Level Facial Attributes Transfer with Geometry-aware Flow," in Proc. of AAAI Conference on Artificial Intelligence, vol. 33, no. 1, 2019.
37 C. Wu, C. Liu, H. Y. Shum, Y. Q. Xy, and Z. Zhang, "Automatic Eyeglasses Removal from Face Images," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 6, no. 3, pp. 322-336, 2004.
38 M. Smet, M. Fransens, and L. Gool, "A Generalized EM Approach for 3D Model based Face Recognition under Occlusions," in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1423-1430, 2006.