Journal of International Society for Simulation Surgery

International Society for Simulation Surgery (국제컴퓨터가상수술학회)
권호 표지
DOI QR코드

DOI QR Code

Essential Computer Vision Methods for Maximal Visual Quality of Experience on Augmented Reality

  • Heo, Suwoong (The Department of Electrical and Electronic Engineering, Yonsei University Yonsei University) ;
  • Song, Hyewon (The Department of Electrical and Electronic Engineering, Yonsei University Yonsei University) ;
  • Kim, Jinwoo (The Department of Electrical and Electronic Engineering, Yonsei University Yonsei University) ;
  • Nguyen, Anh-Duc (The Department of Electrical and Electronic Engineering, Yonsei University Yonsei University) ;
  • Lee, Sanghoon (The Department of Electrical and Electronic Engineering, Yonsei University Yonsei University)
  • Received : 2016.11.11
  • Accepted : 2016.11.19
  • Published : 2016.12.10
Download PDF
⟨ Previous Next ⟩

Abstract

The augmented reality is the environment which consists of real-world view and information drawn by computer. Since the image which user can see through augmented reality device is a synthetic image composed by real-view and virtual image, it is important to make the virtual image generated by computer well harmonized with real-view image. In this paper, we present reviews of several works about computer vision and graphics methods which give user realistic augmented reality experience. To generate visually harmonized synthetic image which consists of a real and a virtual image, 3D geometry and environmental information such as lighting or material surface reflectivity should be known by the computer. There are lots of computer vision methods which aim to estimate those. We introduce some of the approaches related to acquiring geometric information, lighting environment and material surface properties using monocular or multi-view images. We expect that this paper gives reader's intuition of the computer vision methods for providing a realistic augmented reality experience.

Keywords

References

  1. Liao H, Inomata T, Sakuma I, Dohi T. 3-D augmented reality for MRI-guided surgery using integral videography autostereoscopic image overlay. IEEE Transactions on Biomedical Engineering 2010; 57(6):1476-1486 https://doi.org/10.1109/TBME.2010.2040278
  2. Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I, et al. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE Transactions on Biomedical Engineering 2014;61(4):1295-1304 https://doi.org/10.1109/TBME.2014.2301191
  3. Shen F, Chen B, Guo Q, Qi Y, Shen Y. Augmented reality patientspecific reconstruction plate design for pelvic and acetabular fracture surgery. International Journal of Computer Assisted Radiology and Surgery 2013;8(2):169-179 https://doi.org/10.1007/s11548-012-0775-5
  4. Yamamoto T, Abolhassani N, Jung S, Okamura AM, Judkins TN. Augmented reality and haptic interfaces for robot-assisted surgery. The International Journal of Medical Robotics and Computer Assisted Surgery 2012;8(1):45-56 https://doi.org/10.1002/rcs.421
  5. Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, et al. Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatric Surgery International 2012;28(4):341-346 https://doi.org/10.1007/s00383-011-3034-x
  6. Kamphuis C, Barsom E, Schijven M, Christoph N. Augmented reality in medical education?. Perspectives on Medical Education 2014; 3(4):300-311 https://doi.org/10.1007/s40037-013-0107-7
  7. Whitted T. An improved Illumination Model for Shaded Display. ACM SIGGRAPH Computer Graphics 1979:13.2
  8. Nah JH, Kwon HJ, Kim DS, Jeong CH, Park JH, Han TD, et al, RayCore: A ray-tracing hardware architecture for mobile devices. ACM Transactions on Graphics 2014;33(5):162 https://doi.org/10.1145/2629634
  9. Tobias OJ, Seara R, Image segmentation by histogram thresholding using fuzzy sets. IEEE Transactions on Image Processing 2002;11: 1457-1465 https://doi.org/10.1109/TIP.2002.806231
  10. Haris K, Efstratiadis SN, Maglaveras N, Katsaggelos AK, Hybrid image segmentation using watersheds and fast region merging. IEEE Transactions on Image Processing 1998;7:1684-1699 https://doi.org/10.1109/83.730380
  11. Wu X, Adaptive split-and-merge segmentation based on piecewise least-square approximation. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993;15:808-815 https://doi.org/10.1109/34.236248
  12. Sinha SN. Graph cut algorithms in vision. graphics and machine learning-an integrative paper, 2004
  13. Shi J, Malik J, Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000; 22:888-905 https://doi.org/10.1109/34.868688
  14. Rouhi R, Jafari M, Kasaei S, Keshavarzian P, Benign and malignant breast tumors classification based on region growing and CNN segmentation. Expert Systems with Applications 2015;42:990-1002 https://doi.org/10.1016/j.eswa.2014.09.020
  15. Shelhamer E, Long J, Darrell T. Fully Convolutional Networks for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
  16. Noh H, Hong S, Han B, Learning deconvolution network for semantic segmentation. IEEE International Conference on Computer Vision 2015:1520-1528
  17. Pinheiro PH, Collobert R, Recurrent Convolutional Neural Networks for Scene Labeling. ICML, 2014:82-90
  18. Bruce NDB, Catton C, Janjic S. A Deeper Look at Saliency: Feature Contrast, Semantics, and Beyond. IEEE Conference on Computer Vision and Pattern Recognition 2016:516-524
  19. Felzenszwalb PF, Huttenlocher DP, Efficient graph-based image segmentation. International Journal of Computer Vision 2004;59: 167-181 https://doi.org/10.1023/B:VISI.0000022288.19776.77
  20. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, et al. Backpropagation applied to handwritten zip code recognition. Neural Computation 1989;1:541-551 https://doi.org/10.1162/neco.1989.1.4.541
  21. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. NIP, 2014:1097-1105
  22. Karsch K, Hedau V, Forsyth D, Hoiem D. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics (TOG) 2011;30:6
  23. Rother C. A new approach to vanishing point detection in architectural environments. Image and Vision Computing 2002;20(9):647- 655 https://doi.org/10.1016/S0262-8856(02)00054-9
  24. Coughlan JM, Yuille AL. Manhattan world: Compass direction from a single image by bayesian inference. Computer Vision, 1999:2
  25. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, et al. KinectFusion: Real-time dense surface mapping and tracking. Mixed and augmented reality (ISMAR), 2011
  26. Whelan T, Leutenegger S, Salas-Moreno RF, Glocker B, Davison AJ. ElasticFusion: Dense SLAM without a pose graph. Proc. Robotics: Science and Systems, Rome, 2015
  27. Perley RA. High dynamic range imaging. Synthesis Imaging in Radio Astronomy II. 1999:180
  28. Khan EA, Reinhard E, Fleming RW, Bulthofff HH. Image-based material editing. ACM Transactions on Graphics (TOG) 2006;25(3): 654-663 https://doi.org/10.1145/1141911.1141937
  29. Nishino K, Zhang Z, Ikeuchi K. Determining reflectance parameters and illumination distribution from a sparse set of images for viewdependent image synthesis. ICCV 2001;1:599-606
  30. Tunwattanapong B, Fyffe G, Graham P, Busch J, Yu X, Ghosh A, et al. Acquiring reflectance and shape f rom continuous spherical harmonic illumination. ACM Transactions on Graphics (TOG) 2013; 32(4):109