Browse > Article
http://dx.doi.org/10.3837/tiis.2021.10.010

Patch size adaptive image inpainting  

Liu, Huaming (School of Computer and Information Engineering, Fuyang Normal University)
Lu, Guanming (College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications)
Bi, Xuehui (School of Computer and Information Engineering, Fuyang Normal University)
Wang, Weilan (Key Laboratory of China's Ethnic Languages and Information Technology of Ministry of Education, Northwest Minzu University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.10, 2021 , pp. 3642-3667 More about this Journal
Abstract
Texture synthesis technology has the advantages of repairing texture and structure at the same time. However, during the filling process, the size of the patch is fixed, and the content of the filling is not fully considered. In order to be able to adaptively change the patch size, we used the exemplar-based inpainting technique as the test algorithm, considering the image structure and texture, calculated the image structure patch size and texture patch size, and comprehensively determined the image patch size. This can adaptively change the patch size according to the filling content. In addition, we use multi-layer images to calculate the priority, so that the order of image repair was more stable. The proposed repair algorithm is compared with other image repair algorithms. The experimental results showed that the proposed adaptive image repair algorithm can better repair the texture and structure of the image, which proved the effectiveness of the proposed algorithm.
Keywords
Texture synthesis; exemplar-based techniques; patch adaptive size; object removal;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Zdunek R, Sadowski T, "Image Completion with Hybrid Interpolation in Tensor Representation," Appl. Sci., vol. 10, no. 3, pp. 1-16, 2020.
2 F. Tony Chan, K. Michael, C. Andy Yau, M. Andy, Yip, "Superresolution image reconstruction using fast inpainting algorithms," Appl. and Comput. Harmonic Anal., vol.23, no.1, pp.3-24, 2007.   DOI
3 M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, "Image inpainting," in Proc. of ACM SIGGRAPH. Conf. Computer Graph., New Orleans, LA, USA, pp.417-424, 2000.
4 V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick, "Graphcut textures: image and video synthesis using graph cuts," ACM Trans. Graphics, vol. 22, no. 3, pp. 277-286, 2003.   DOI
5 C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, "PatchMatch: A randomized correspondence algorithm for structural image editing," ACM Transactions on Graphics, vol. 28, no. 3, pp. 1-11, 2009.
6 M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, "Simultaneous structure and texture image inpainting," in Proc. of IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit, vol. 12, no. 8, pp. 882-889, Aug. 2003.
7 A. Criminisi, P. Perez, and K. Toyama, "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200-1212, 2004.   DOI
8 P. Arias, G. Facciolo, V. Caselles G. Sapiro, "A Variational Framework for Exemplar-Based Image Inpainting," Int. J. Comput. Vision, vol. 93, no. 3, pp. 319-347, Jul. 2011.   DOI
9 D. Thanh, V. Prasath, S. Dvoenko, L. M. Hieu, "An adaptive image inpainting method based on euler's elastica with adaptive parameters estimation and the discrete gradient method," Signal Process., vol. 178, pp.1-19, 2021.
10 S. Yang, H. Liang, Y. Wang, H.Y. Cai, X.D. Chen, "Image Inpainting Based on Multi-Patch Match with Adaptive Size," Appl. Sci., vol. 10, no. 14, pp. 1-17, 2020.
11 Q. Fan, "Research on Exemplar-based Image Inpainting Algorithm," M.S. thesis, University of Electronic Science and Technique of China, Hefei, Anhui, China, 2018.
12 K. Nazeri, E. Ng, T. Joseph, F. Z. Qureshi, M. Ebrahimi, "Edgeconnect: Generative image inpainting with adversarial edge learning," arXiv preprint arXiv:1901.00212, 2019.
13 M. Thang, "Image Inpainting Algorithm Based on Patch Structure Sparsity," M.S. thesis, Huazhong University of Science & Technique, Wuhan, Hubei, China, 2016.
14 M. Bertalmio, A. Bertozzi, G. Sapiro, "Navier-stokes, fluid dynamics, and image and video inpainting," in Proc. of IEEE Comput Soc. Conf. Comput. Vision Pattern Recognit, Kauai, HI, USA, pp.355-362, 2001.
15 T. F. Chan, and J. Shen, "Nontexture inpainting by curvature-driven diffusions," J. Vis. Commun. Image R., vol. 12, no. 4, pp. 436-449, 2001.   DOI
16 A. Efros, W. Freeman, "Image quilting for texture synthesis and transfer," in Proc. of ACM SIGGRAPH Conf. Computer Graph., Los Angeles, CA, USA, pp.341-346, 2001.
17 H. Zhou, and J. Zheng, "Adaptive patch size determination for patch-based image completion," in Proc. of Int. Conf. Image Process., Hong Kong, China, pp. 421-424, 2010.
18 N. J. Butko, L. Zhang, G. W. Cottrell, and J. R. Movellan, "Visual saliency model for robot cameras," in Proc. of IEEE Int. Conf. Rob. Autom., Pasadena, CA, USA, pp. 2398-2403, 2008.
19 X. Zhou, X. Yan, "Adaptive image inpainitng based on structural correlation," Computer science, vol.47, no.4, pp.131-135, 2020.
20 J. Cao, Y. Li, H. Cui Q. Zhang, "An Adaptive Sample Block and Local Search Algorithm for Inpainting of Ancient Temple Murals," Journal of Computer-Aided Design & Computer Graphics, vol.31, no.11, pp.2030-2037, 2019.
21 C. Meng, K. He, Q. Jiao, "Image completion method with adaptive patch size," Journal of Image and Graphics, vol.17, no.3, pp.337-341, 2012.
22 C. Chang, "Research on image inpainting algorithm based on texture," M.S. Thesis, Fuzhou University, Fuzhou, Fujian, China, 2016.(in Chinese)
23 R. P. Borole, and S. V. Bonde, "Image Restoration using Prioritized Exemplar Inpainting with Automatic Patch Optimization," J. Inst. Eng. Ser. B, vol. 98, no. 3, pp. 311-319, 2017.
24 R. Zhao, L. Zhao, "Image Inpainting Research Based on Deep Learning," International Journal of Advanced Network, Monitoring and Controls, vol. 5, no. 2, pp. 23-30, 2020.
25 C. Chang, "Research on image inpainting algorithm based on texture," M.S. Thesis, Fuzhou University, Fuzhou, Fujian, China, 2016.
26 S. Ye, X. Lin, G. Wang, "Feature based adaptive error concealment for image transmission over wireless channel," Computer engineering and applications, vol.19, pp.61-64, 2002.
27 D. Z. Silvano, "A note on the gradient of a multi-image," Comput. Vision Graphics Image Process., vol.33, no.1, pp.116-125, 1986.   DOI
28 X. Li, Y. Qiong, X. Yang, J. Jia. "Structure extraction from texture via relative total variation," ACM Tran. Graphics, vol.31, no.6, pp.1-10, 2012.
29 L. Olivier, E. Mounira, G. Christine, "Hierarchical super-resolution-based inpainting," IEEE Trans. Image Process., vol.22, no.10, pp.3779-3790, 2013.   DOI
30 P. Tiefenbacher, V. Bogischef, D. Merget, G. Rigoll, "Subjective and objective evaluation of image inpainting quality," in Proc. of Int. Conf. Image Process., Quebec, QC, Canada, pp.447-451, 2015.
31 J. Herling, W. Broll, "Pixmix: A real-time approach to high-quality diminished reality," in Proc. of IEEE Int. Symp. Mix. Augmented Real., Sci. Technol. Pap., Atlanta, GA, USA, pp.141-150, 2012.
32 Z. Xu, J. Sun, "Image Inpainting by Patch Propagation Using Patch Sparsity," IEEE Trans. Image Process., vol.19, no.5, pp.1153-1165, 2010.   DOI
33 Y. He, "Research on Color Image Inpainting Based on Gradient Change and Structure Tensor," M.S. Thesis, Hangzhou Dianzi University, Hangzhou, Zhejiang, China, 2018.
34 O. Le Meur, J. Gautier, and C. Guillemot, "Examplar-based inpainting based on local geometry," in Proc. of Int. Conf. Image Process., Hongkong, China, pp. 3401-3404, 2011.
35 R. P. Borole, and S. V. Bonde, "Image Restoration and Object Removal Using Prioritized Adaptive Patch-Based Inpainting in a Wavelet Domain," in Proc. of J. Inf. Process. Syst., vol. 13, no. 5, pp.1183-1202, 2017.
36 K. He, J. Niu, C. Shen, W. Lu, "Image Inpainting Algorithm with Adaptive Patch Using SSIM," Journal of Tianjin University (Science and Technique), vol.51, no.07, pp.763-767, 2018.
37 X. Lin, X. Yang, "Effective exemplar-based image inpainting using low-rank matrix completion," in Proc. of IEEE Int. Conf. Aware. Sci. Technol., Qinhuangdao, China, pp.37-42, 2015.
38 A. Criminisi, P. Perez, K. Toyama, "Object removal by exemplar-based inpainting," in Proc. of IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognit, Madison, WI, USA, pp.1-8, 2003.
39 L. Meur, C. Olivier Guillemot, "Super-Resolution-Based Inpainting," Lect. Notes Comput. Sci., Florence, Italy, pp.554-567, 2012.
40 J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, T. Huang, "Free-form image inpainting with gated convolution," in Proc. of IEEE Int. Conf. Comput. Vision, Seoul, Korea, pp.4471-4480, 2019.
41 C. Ledig, T. Lucas, H. Ferenc, C. Jose, C. Andrew, A. Alejandro, A. Andrew, T. Alykhan, T. Johannes, W. Zehan, S. Wenzhe, "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network," in Proc. of IEEE Conf. Comput. Vis. Pattern Recognit, Honolulu, HI, USA, pp.105-114, 2017.
42 C. Yang, X. Lu, Z. Lin, S. Eli, W. Oliver, L. Hao, "High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis," in Proc. of IEEE Conf. Comput. Vis. Pattern Recognit, Honolulu, HI, USA, pp.4076-4084, 2017.
43 H. Liu, X. Bi, G. Lu, W. Wang, "Exemplar-Based Image Inpainting With Multi-Resolution Information and the Graph Cut Technique," IEEE Access, vol.7, pp.101641-101657, 2019.   DOI
44 L. Zhang, L. Zhang, X. Mou, D. Zhang, "FSIM: A feature similarity index for image quality assessment," IEEE trans. Image Process., vol.20, no.8, pp.2378-2386, 2011.   DOI
45 A. Telea, "An image inpainting technique based on the fast marching method," J. graphics tools, vol.9, no.1, pp.25-36, 2004.   DOI
46 A. Bugeau, M. Bertalmi o, V. Caselles, G. Sapiro, "A comprehensive framework for image inpainting," IEEE Trans. Image Process., vol.19, no.10, pp.2634-2645, 2010.   DOI
47 P. Getreuer, "Total variation inpainting using split Bregman," Image Processing on Line, vol.2, pp.147-157, 2012.   DOI
48 V. Fedorov, G. Facciolo, P. Arias, "Variational framework for non-local inpainting," Image Processing on Line, vol.5, pp.362-386, 2015.   DOI
49 J. Mati as, G. Facciolo, E. Meinhardt-Llopis, "Poisson Image Editing," Image Processing on Line, vol.6, pp.300-325, 2016.   DOI
50 D. Lee, S. Yoo, M. Choi, J. Ra, J. Kim, "Block Poisson Method and its application to large scale image editing," in Proc. of Int. Conf. Image Process., Orlando, FL, USA, pp.2121-2124, 2012.
51 P. Perez, M. Gangnet, A. Blake, "Poisson image editing," ACM Trans. Graphics, vol.22, no.3, pp.313-318, 2003.   DOI
52 S. Darabi, E. Shechtman, C. Barnes, B. G. Dan, P. Sen, "Image melding: combining inconsistent images using patch-based synthesis," ACM Trans. Graphics, vol.31, no.4, pp.1-10, 2012.