Browse > Article
http://dx.doi.org/10.3837/tiis.2020.01.011

An Interactive Perspective Scene Completion Framework Guided by Complanate Mesh  

Hao, Chuanyan (School of Education Science and Technology, Nanjing University of Posts and Telecommunications)
Jin, Zilong (School of Computer and Software, Nanjing University of Information Science and Technology)
Yang, Zhixin (Faculty of Science and Technology, University of Macau)
Chen, Yadang (School of Computer and Software, Nanjing University of Information Science and Technology)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.14, no.1, 2020 , pp. 183-200 More about this Journal
Abstract
This paper presents an efficient interactive framework for perspective scene completion and editing tasks, which are available largely in the real world but rarely studied in the field of image completion. Considering that it is quite hard to extract perspective information from a single image, this work starts from a friendly and portable interactive platform to obtain the basic perspective data. Then, in order to make this interface less sensitive, easier and more flexible, a perspective-rectification based correction mechanism is proposed to iteratively update the locations of the initial points selected by users. At last, a complanate mesh is generated by the geometry calculations from these corrected initial positions. This mesh must approximate the perspective direction and the structure topology as much as possible so that the filling process can be conducted under the constraint of the perspective effects of the original image. Our experiments show the results with good qualities and performances, and also demonstrate the validity of our approaches by various perspective scenes and images.
Keywords
image completion; image editing; exemplar-based synthesis; interaction; perspective rectification;
Citations & Related Records
연도 인용수 순위
  • Reference
1 K. He and J. Sun, "Statistics of patch offsets for image completion," in Proc. of the 12th European Conference on Computer Vision, pp. 16-29, October 7-13, 2012.
2 G. F. Zhang, J. Y. Jia, T. T. Wong and H. J. Bao, "Consistent depth maps recovery from a video sequence," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 1, no. 31, pp. 974-988, 2009.
3 Y. D. Chen, C. Y. Hao, Z. M. Cai, W. Wu and E. H. Wu, "Live accurate and dense reconstruction from a handheld camera," Journal of Computer Animation and Virtual Worlds, vol. 24, no. 3, pp. 387-397, 2013.   DOI
4 A. Criminisi, P. Perez, and K. Toyama, "Region filling and object removal by exemplar-based image inpainting," IEEE Transactions on Image Processing, vol. 13, no. 9, pp: 1200-1212, 2004.   DOI
5 I. Drori, D. Cohen-Or, and H. Yeshurun, "Fragment-based image completion," ACM Transactions on Graphics, vol. 22, no. 3, pp. 303-312, 2003.   DOI
6 J. Sun, L. Yuan, J. Jia, and H.-Y. Shum, "Image completion with structure propagation," ACM Transactions on Graphics, vol 24, no. 3, pp. 861-868, 2005.   DOI
7 M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image inpainting," in Proc. of the 27th Annual Conf. on Computer Graphics and Interactive Techniques, pp. 417-424, July 23-28, 2000.
8 M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, "Simultaneous structure and texture image inpainting," IEEE Transactions on Image Processing, vol. 12, no. 8, pp. 882-889, 2003.   DOI
9 A. A. Efros and T. K. Leung, "Texture synthesis by non-parametric sampling," in Proc. of the International Conf. on Computer Vision, pp. 1033-1038, September 20-25, 1999.
10 M. Ashikhmin, "Synthesizing natural textures," in Proc. of the 2001 Symposium on Interactive 3D Graphics, pp. 217-226, March 26-29, 2001.
11 L.-Y. Wei and M. Levoy, "Fast texture synthesis using tree-structured vector quantization," in Proc. of the 27th annual Conf. on Computer Graphics and Interactive Techniques, pp. 479-488, July 23-28, 2000.
12 A. A. Efros and W. T. Freeman, "Image quilting for texture synthesis and transfer," in Proc. of the 28th Annual Conf. on Computer Graphics and Interactive Techniques, pp. 341-346, August 12-17, 2001.
13 V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick, "Graphcut textures: image and video synthesis using graph cuts," ACM Transactions on Graphics, vol. 22, no. 3, pp. 277-286, 2003.   DOI
14 L. Liang, C. Liu, Y.-Q. Xu, B. Guo, and H.-Y. Shum, "Real-time texture synthesis by patch-based sampling," ACM Transactions on Graphics, vol. 20, no. 3, pp. 127-150, 2001.   DOI
15 A. Schodl, R. Szeliski, D. H. Salesin, and I. Essa, "Video textures," in Proc. of the 27th Annual Conf. on Computer Graphics and Interactive Techniques, pp. 489-498, July 23-28, 2000.
16 P. Perez, M. Gangnet, and A. Blake, "Poisson image editing," in Proc. of the 30th Annual Conf. on Computer Graphics and Interactive Techniques, pp. 313-318, July 27-31, 2003.
17 A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin, and M. Cohen, "Interactive digital photomontage," ACM Transactions on Graphics, vol. 23, no. 3, pp. 294-302, 2004.   DOI
18 C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, "Patchmatch: a randomized correspondence algorithm for structural image editing," ACM Transactions on Graphics, vol. 28, no. 3, pp. 1-11, 2009.
19 C. Barnes, E. Shechtman, D. B. Goldman, and A. Finkelstein, "The generalized patchmatch correspondence algorithm," in Proc. of the 11th European Conf. on Computer Vision, pp. 29-43, September 5-11, 2010.
20 S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and P. Sen, "Image melding: combining inconsistent images using patch-based synthesis," ACM Transactions on Graphics, vol. 31, no. 4, pp. 1-10, 2012.
21 Z. Zhou, Z. Zhu, X. Bai, D. Lischinski, D. Cohen-Or, and H. Huang, "Non-stationary texture synthesis by adversarial expansion," ACM Transactions on Graphics, vol. 37, no. 4, pp. 1-13, 2018.
22 Y. Wexler, E. Shechtman, and M. Irani, "Space-time completion of video," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 463-476, 2007.   DOI
23 M. Wilczkowiak, G. J. Brostow, B. Tordoff, and R. Cipolla, "Hole filling through photomontage," in 16th British Machine Vision Conf., pp. 492-501, September, 2005.
24 J. Hays and A. A. Efros, "Scene completion using millions of photographs," ACM Transactions on Graphics, vol. 26, no. 3, pp. 1-7, 2007.   DOI
25 J. -B. Huang, S. B. Kang, N. Ahuja, and J. Kopf, "Image Completion Using Planar Structure Guidance," ACM Transactions on Graphics, vol. 33, no. 4, pp. 1-10, 2016.
26 S. Lizuka, E. Simo-serra, and H. Ishikawa, "Globally and Locally Consistent Image Completion," ACM Transactions on Graphics, vol. 36, no. 4, pp. 1-14, 2017.
27 D. Pavić, V. Schönefeld, and L. Kobbelt, "Interactive image completion with perspective correction," The Visual Computer, vol. 22, no. 9, pp. 671-681, 2006.   DOI
28 Y. Liu, W.-C. Lin, and J. Hays, "Near-regular texture analysis and manipulation," ACM Transactions on Graphics, vol. 23, no. 3, pp. 368-376, 2004.   DOI
29 J. -B. Huang, J. Kopf, N. Ahuja, and S. B. Kang, "Transformation Guided Image Completion," in Proc. of IEEE International Conf. on Computational Photography, pp. 1-9, April 19-21, 2013.