Browse > Article
http://dx.doi.org/10.3837/tiis.2022.03.007

Face inpainting via Learnable Structure Knowledge of Fusion Network  

Yang, You (National Center for Applied Mathematics in Chongqing)
Liu, Sixun (School of Computer and Information Science, Chongqing Normal University)
Xing, Bin (National Engineering Laboratory of Industrial Big-Data Application Technology)
Li, Kesen (School of Computer and Information Science, Chongqing Normal University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.16, no.3, 2022 , pp. 877-893 More about this Journal
Abstract
With the development of deep learning, face inpainting has been significantly enhanced in the past few years. Although image inpainting framework integrated with generative adversarial network or attention mechanism enhanced the semantic understanding among facial components, the issues of reconstruction on corrupted regions are still worthy to explore, such as blurred edge structure, excessive smoothness, unreasonable semantic understanding and visual artifacts, etc. To address these issues, we propose a Learnable Structure Knowledge of Fusion Network (LSK-FNet), which learns a prior knowledge by edge generation network for image inpainting. The architecture involves two steps: Firstly, structure information obtained by edge generation network is used as the prior knowledge for face inpainting network. Secondly, both the generated prior knowledge and the incomplete image are fed into the face inpainting network together to get the fusion information. To improve the accuracy of inpainting, both of gated convolution and region normalization are applied in our proposed model. We evaluate our LSK-FNet qualitatively and quantitatively on the CelebA-HQ dataset. The experimental results demonstrate that the edge structure and details of facial images can be improved by using LSK-FNet. Our model surpasses the compared models on L1, PSNR and SSIM metrics. When the masked region is less than 20%, L1 loss reduce by more than 4.3%.
Keywords
Face Inpainting; Image Edge; Gated Convolution; Region Normalization; Prior Knowledge;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 O. Le Meur, J. Gautier, C. Guillemot, "Examplar-based inpainting based on local geometry," in Proc. of 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium, pp. 3401-3404, 2011.
2 M. Sagong, Y. Shin, S. Kim, et al., "PEPSI: fast image inpainting with parallel decoding network," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 11352-11360, 2019.
3 N. Wang, J. Li, L. Zhang, et al., "MUSICAL: Multi-Scale image contextual attention learning for inpainting," in Proc. of Twenty-Eighth International Joint Conference on Artificial Intelligence, pp. 3748-3754, August 2019.
4 J. Yu, Z. Lin, J. Yang, et al., "Free-Form image inpainting with gated convolution," in Proc. of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, pp. 4470-4479, 2019.
5 J. Li, N. Wang, L. Zhang, et al., "Recurrent feature reasoning for image inpainting," in Proc. of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 7757-7765, 2020.
6 L. Song, J. Cao, L. Song, et al., "Geometry-aware face completion and editing," in Proc. of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 2506-2513, 2019.
7 Z. Xu, J. Sun, "Image inpainting by patch propagation using patch sparsity," IEEE Transactions on Image Processing, vol. 19, no. 5, pp. 1153-1165, May 2010.   DOI
8 M. Bertalmio, G. Sapiro, V. Caselles, et al., "Image inpainting," in Proc. of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New York, USA, ACM, 417-424, 2000.
9 A. V. D. Oord, N. Kalechbrenner, K. Kavukcuoglu, "Pixel recurrent neural networks," in Proc. of 33rd International Conference on Machine Learning, ICML 2016, vol. 4, pp. 2611-2620, 2016.
10 Y. Li, S. Liu, J. Yang et al., "Generative face completion," in Proc. of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 5892-5900, 2017.
11 L. A. Gatys, A. S. Ecker, M. Bethge, "Image style transfer using convolutional neural Networks," in Proc. of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 2414-2423, 2016.
12 S. Iizuka, E. Simo-Serra, H. Ishikawa, "Globally and locally consistent image completion," ACM Transactions on Graphics, vol. 36, no. 4, pp. 1-14, 2017.
13 K. Simonyan, A. Zisscrman, "Very deep convolutional networks for large-scale image recognition," in Proc. of 3rd International Conference on Learning Representations, 2015.
14 K. Nazeri, E. Ng, T. Joseph, et al., "EdgeConnect: structure guided image inpainting using edge prediction," in Proc. of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, South Korea, pp. 3265-3274, 2019.
15 D. Pathak, P. Krahenbuhl, J. Donahue, et al., "Context Encoders: feature learning by inpainting," in Proc. of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 2536-2544, 2016.
16 J. Yu, Z. Lin, J. Yang, et al., "Generative image inpainting with contextual attention," in Proc. of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 5505-5514, 2018.
17 Y. Zeng, J. Fu, H. Chao, et al., "Learning Pyramid-Context Encoder network for high-quality image inpainting," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 1486-1494, 2019.
18 C. Xie, S. Liu, C. Li, et al., "Image inpainting with learnable bidirectional attention maps," in Proc. of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, pp. 8857-8866, 2019.
19 W. Xiong, J. Yu, Z. Lin, et al., "Foreground-Aware image inpainting," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 5833-5841, 2019.
20 J. Yang, Z. Qi, Y. Shi, "Learning to incorporate structure knowledge for image inpainting," in Proc. of the AAAI Conference on Artificial Intelligence, vol. 34, no. 7, pp. 12605-12612, 2020.
21 D. P. Kingma, J. Ba, "Adam: a method for stochastic optimization," in Proc. of International Conference on Learning Representations, 2015.
22 W. Yi, T. Xin, X. Qi, et al., "Image inpainting via generative multi-column convolutional neural networks," Neural Information Processing Systems foundation, pp. 331-340, 2018.
23 G. Liu, F. A. Reda, K. J. Shih, et al., "Image inpainting for irregular holes using partial convolutions," in Proc. of 15th European Conference on Computer, LNCS, vol. 11215 pp. 89-105, 2018.
24 Z. Guo, Z. Chen, T. Yu, et al., "Progressive image inpainting with full-resolution residual network," in Proc. of ACM International Conference on Multimedia, New York, NY, USA, pp.2496-2504, 2019.
25 T. Miyato, T. Kataoka, M. Koyama, et al., "Spectral normalization for generative adversarial networks," in Proc. of International Conference on Learning Representations, 2018.
26 Z. Qiang, L. He, D. Xu, "Exemplar-based pixel by pixel inpainting based on patch shift," Communications in Computer and Information Science, vol. 772, pp. 370-382, 2017.   DOI
27 T. Karras, T. Aila, S. Laine, et al., "Progressive growing of GANs for improved quality, stability, and variation," in Proc. of in International Conference on Learning Representations, 2018.
28 T. Zhou, C. Ding, S. Lin, et al., "Learning oracle attention for high-fidelity face completion," in Proc. of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 7677-7686, 2020.
29 J. B. Huang, S. B. Kang, N. Ahuja, et al., "Image completion using planar structure guidance," ACM Transactions on Graphics, vol. 33, no. 4, pp. 1-10, 2014.
30 Y. G. Shin, M. C. Sagong, Y. J. Yeo, et al., "PEPSI++: fast and lightweight network for image inpainting," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 252-265, Jan. 2021.   DOI
31 L. Haofu, Gareth, Z. Yefeng, et al., "Face completion with semantic knowledge and collaborative Adversarial Learning," in Proc. of Computer Vision-ACCV 2018, pp. 382-397, 2018.
32 Y. Ren, X. Yu, R. Zhang, et al., "StructureFlow: image inpainting via structure-aware appearance flow," in Proc. of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, pp. 181-190, 2019.
33 J. Li, F. He, L. Zhang, et al., "Progressive reconstruction of visual structure for image inpainting," in Proc. of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, pp. 5961-5970, 2019.
34 T. Yu, Z. Guo, X. Jin, et al., "Region normalization for image inpainting," in Proc. of the AAAI Conference on Artificial Intelligence. vol. 34, no. 7, pp. 12733-12740, 2020.   DOI
35 P. Isola, J. Zhu, T. Zhou et al., "Image-to-Image translation with conditional adversarial networks," in Proc. of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 5967-5976, 2017.
36 K. He, X. Zhang, S. Ren, et al., "Deep residual learning for image recognition," in Proc. of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770-778, 2016.
37 C. Zheng, T. Cham, J. Cai, "Pluralistic image completion," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 1438-1447, 2019.
38 Xuewei Li, Xueming Li, X. Zhang, et al., "A method of inpainting moles and acne on the high-resolution face photos," IET Image Processing, vol. 15, no. 3, pp. 833-844, February 2021.   DOI
39 H. Yang, J. Yuan, C. Li et al., "BrainIoT: Brain-Like Productive Services Provisioning with Federated Learning in Industrial IoT," IEEE Internet of Things Journal, vol. 9, no. 3, pp. 2014-2024, 2022.   DOI
40 Y. H. Song, C. Yang, Y. J. Shen, et al., "SPG-Net: segmentation prediction and guidance network for image inpainting," in Proc. of 29th British Machine Vision Conference, BMVC 2018, 2018.
41 A. Criminisi, P. Perez, K. Toyama, "Object removal by exemplar-based inpainting," in Proc. of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceeding, Madison, WI, USA, pp. II-II, 2003.
42 C. Barnes, E. Shechtman, Finkelstein A, et al., "PatchMatch: a randomized correspondence algorithm for structural image editing," ACM Transactions on Graphics, vol. 28, no. 3, pp. 1-11, July 27, 2009.
43 S. Darabi, E. Shechtman, C. Barnes, et al., "Image melding: combining inconsistent images using patch-based synthesis," ACM Transactions on Graphics, vol. 31, no. 4, pp. 1-10, July 2012.