Journal of information and communication convergence engineering

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Three-Dimensional Imaging and Display through Integral Photography

  • Navarro, Hector (3D Imaging and Display Lab., Department of Optics, University of Valencia) ;
  • Dorado, Adrian (3D Imaging and Display Lab., Department of Optics, University of Valencia) ;
  • Saavedra, Genaro (3D Imaging and Display Lab., Department of Optics, University of Valencia) ;
  • Corral, Manuel Martinez (3D Imaging and Display Lab., Department of Optics, University of Valencia)
  • Received : 2014.04.10
  • Accepted : 2014.05.23
  • Published : 2014.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Here, we present a review of the proposals and advances in the field of three-dimensional (3D) imaging acquisition and display made in the last century. The most popular techniques are based on the concept of stereoscopy. However, stereoscopy does not provide real 3D experience, and produces discomfort due to the conflict between convergence and accommodation. For this reason, we focus this paper on integral imaging, which is a technique that permits the codification of 3D information in an array of 2D images obtained from different perspectives. When this array of elemental images is placed in front of an array of microlenses, the perspectives are integrated producing 3D images with full parallax and free of the convergence-accommodation conflict. In the paper we describe the principles of this technique, together with some new applications of integral imaging.

Keywords

References

  1. W. Rollmann, "Notiz zur stereoskopie," Annalen der Physik, vol. 165, no. 6, pp. 350-351, 1853. https://doi.org/10.1002/andp.18531650614
  2. E. H. Land, "Polarizing optical system," U.S. Patent 2,099,694, 1937.
  3. D. W. Byatt, "Stereoscopic television system," U.S. Patent 4,281,341, 1981.
  4. P. J., Bos and K. R. Koehler/Beran, "The pi-cell: a fast liquidcrystal optical-switching device," Molecular Crystals and Liquid Crystals, vol. 113, no. 1, pp. 329-339, 1984. https://doi.org/10.1080/00268948408071693
  5. W. Hess, "Stereoscopic picture," U.S. Patent 1,128,979, 1915.
  6. A. Berthier, "Images stereoscopiques de grand format," Le Cosmos, no. 34, pp. 205-210, 1896.
  7. F. E. Ives, "A novel stereogram," Journal of the Franklin Institute, vol. 153, no. 1, pp. 51-52, 1902. https://doi.org/10.1016/S0016-0032(02)90195-X
  8. B. Julesz, "Stereopsis and binocular rivalry of contours," Journal of the Optical Society of America, vol. 53, no. 8, pp. 994-998, 1963. https://doi.org/10.1364/JOSA.53.000994
  9. C. W. Kanolt, "Photographic method and apparatus," U. S. Patent 1,260,682, 1918.
  10. H. Imai, M. Imai, Y. Ogura, and K. Kubota, "Eye-position tracking stereoscopic display using image-shifting optics," Proceedings of SPIE, vol. 2653, pp. 49-55, 1996.
  11. B. G. Blundell and A. J. Schwarz, Volumetric Three-Dimensional Display Systems. New York, NY: John Wiley & Sons, 2000.
  12. D. Gabor, D. "A new microscopic principle," Nature, vol. 161, no. 4098, pp. 777-778, 1948. https://doi.org/10.1038/161777a0
  13. G. Lippmann, "Epreuves reversibles photographies integrals," Comptes Rendus de l'Academie des Sciences, vol. 146, pp. 446-451, 1908.
  14. G. Lippmann, "Epreuves reversibles donnant la sensation du relief," Journal de Physique Theorique et Appliquee, vol. 7, no. 1, pp. 821-825, 1908. https://doi.org/10.1051/jphystap:019080070082100
  15. G. Lippmann, "L'etalon international de radium," Radium (Paris), vol. 9, no. 4, pp. 169-170, 1912. https://doi.org/10.1051/radium:0191200904016901
  16. Y. Kim et al., "Accommodative response of integral imaging in near distance," Journal of Display Technology, vol. 8, no. 2, pp. 70-78, 2012. https://doi.org/10.1109/JDT.2011.2163701
  17. H. Hiura, S. Yano, T. Mishina, J. Arai, K. Hisatomi, Y. Iwadate, and T. Ito, "A study on accommodation response and depth perception in viewing integral photography," in Proceedings of the 5th International Conference on 3D Systems and Applications (3DSA), Osaka, Japan, 2013.
  18. A. P. Sokolov, Autostereoscopy and integral photography by Professor Lippmann's method. Moskva: Izd-vo MGU (Moscow State University), 1911.
  19. M. Martinez-Corral, R. Martinez-Cuenca, G. Saavedra, H. Navarro, A. Pons, and B. Javidi, "Progresses in 3D integral imaging with optical processing," Journal of Physics: Conference Series, vol. 139, no. 1, p. 012012, 2008. https://doi.org/10.1088/1742-6596/139/1/012012
  20. H. E. Ives, "Optical properties of a Lippman lenticulated sheet," Journal of the Optical Society of America, vol. 21, no. 3, pp. 171-176, 1931. https://doi.org/10.1364/JOSA.21.000171
  21. H. E. Ives, "Parallax panoramagrams made with a large diameter lens," Journal of the Optical Society of America, vol. 20, no. 6, pp. 332-340, 1930. https://doi.org/10.1364/JOSA.20.000332
  22. D. F. w. Coffey, "Apparatus for making a composite stereograph," U.S. Patent 2,063,985, 1936.
  23. J. T. Gruetzner, "Means for obtaining three-dimensional photography," U.S. Patent 2,724,312, 1955.
  24. R. L. De Montebello, "Integral photography," U.S. Patent 3,503,315, 1970.
  25. R. L. De Montebello, "Process of making reinforced lenticular sheet," U.S. Patent 3,584,369, 1971.
  26. R. L. De Montebello, "Wide-angle integral photography: the integram system," Proceedings of the SPIE, vol. 120, pp. 73-91, 1977
  27. H. S. Buck, R. L. de Montebello, and R. P. Globus, "Integral photography apparatus and method of forming same," U.S. Patent 4,732,453, 1988.
  28. C. B. Burckhardt, R. J. Collier, and E. T. Doherty, "Formation and inversion of pseudoscopic images," Applied Optics, vol. 7, no. 4, pp. 627-631, 1968. https://doi.org/10.1364/AO.7.000627
  29. C. B. Burckhardt, "Optimum parameters and resolution limitation of integral photography," Journal of the Optical Society of America, vol. 58, no. 1, pp. 71-74, 1968. https://doi.org/10.1364/JOSA.58.000071
  30. C. B. Burckhardt and E. T. Doherty, "Beaded plate recording of integral photographs," Applied Optics, vol. 8, no. 11, pp. 2329-2331, 1969. https://doi.org/10.1364/AO.8.002329
  31. L. P. Dudley, "Integral photography," U.S. Patent 3,613,539, 1971.
  32. L. P. Dudley, "Methods of integral photography," U.S. Patent 3,675,553, 1972.
  33. T. Okoshi, "Optimum design and depth resolution of lens-sheet and projection-type three-dimensional displays," Applied Optics, vol. 10, no. 10, pp. 2284-2291, 1971. https://doi.org/10.1364/AO.10.002284
  34. T. Okoshi, A. Yano, and Y. Fukumori, "Curved triple-mirror screen for projection-type three-dimensional display," Applied Optics, vol. 10, no. 3, pp. 482-489, 1971. https://doi.org/10.1364/AO.10.000482
  35. T. Okoshi, Three-Dimensional Imaging Techniques. New York, NY: Academic Press, 1976.
  36. Y. A. Dudnikov, "Autostereoscopy and integral photography," Optical Technology, vol. 37, no. 7, pp. 422-426, 1970.
  37. Y. A. Dudnikov, "Elimination of pseudoscopy in integral photography," Optical Technology, vol. 38, no. 3, pp. 140-143, 1971.
  38. Yu. A. Dudnikov, "Effect of three-dimensional moire in integral photography," Soviet Journal of Optical Technology, vol. 41, no. 5, pp. 260-262, 1974.
  39. Y. A. Dudnikov and B. K. Rozhkov, "Selecting the parameters of the lens-array photographing system in integral photography," Soviet Journal of Optical Technology, vol. 45, no. 6, pp. 349-351, 1978.
  40. Y. A. Dudnikov and B. K. Rozhkov, "Limiting capabilities of photographing various subjects by the integral photography method," Soviet Journal of Optical Technology, vol. 46, no. 12, pp. 736-738, 1979.
  41. Y. A. Dudnikov, B. K. Rozhkov, and E. N. Antipova, "Obtaining a portrait of a person by the integral photography method," Soviet Journal of Optical Technology, vol. 47, no. 9, pp. 562-563, 1980.
  42. A. Chutjian and R. J. Collier, "Recording and reconstructing three-dimensional images of computer-generated subjects by Lippmann integral photography," Applied Optics, vol. 7, no. 1, pp. 99-103, 1968. https://doi.org/10.1364/AO.7.000099
  43. L. Yang, M. McCormick, and N. Davies, "Discussion of the optics of a new 3-D imaging system," Applied Optics, vol. 27, no. 21, pp. 4529-4534, 1988. https://doi.org/10.1364/AO.27.004529
  44. N. Davies, M. McCormick, and L. Yang, "Three-dimensional imaging systems: a new development," Applied Optics, vol. 27, no. 21, pp. 4520-4528, 1988. https://doi.org/10.1364/AO.27.004520
  45. N. Davies and M. McCormick, "Imaging system," US Patent 5,040,871, 1991.
  46. N. A. Davies, M. Brewin, and M. McCormick, "Design and analysis of an image transfer system using microlens arrays," Optical Engineering, vol. 33, no. 11, pp. 3624-3633, 1994. https://doi.org/10.1117/12.181580
  47. N. Davies and M. McCormick, "Imaging system," US Patent 5,615,048, 1997.
  48. N. Davies and M. McCormick, "Lens system with intermediate optical transmission microlens screen," U.S. Patent 5,650,876, 1997.
  49. N. Davies, and M. McCormick, "Imaging arrangements," U.S. Patent No. 5,655,043, 1997.
  50. N. Davies and M. McCormick, "Imaging arrangements," U.S. Patent 6,097,541, 2000.
  51. E. H. Adelson and J. R. Bergen, "The plenoptic function and the elements of early vision," Computational Models of Visual Processing, vol. 1, no. 2, pp. 3-20, 1991.
  52. J. J. Gibson, The Senses Considered as Perceptual Systems. Boston, MA: Houghton Mifflin, 1966.
  53. E. H. Adelson and J. Y A. Wang, "Single lens stereo with a plenoptic camera," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 99-106, 1992. https://doi.org/10.1109/34.121783
  54. M. Levoy and P. Hanrahan, "Light field rendering," in Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, pp. 31-42, 1996
  55. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," in Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, pp. 43-54, 1996
  56. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, "Real-time pickup method for a three-dimensional image based on integral photography," Applied Optics, vol. 36, no. 7, pp. 1598-1603, 1997. https://doi.org/10.1364/AO.36.001598
  57. B. Javidi and F. Okano, Three-Dimensional Television, Video, and Display Technologies. Berlin: Springer, 2002.
  58. A. Isaksen, L. McMillan, and S. J. Gortler, "Dynamically reparameterized light fields," in Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, pp. 297-306, 2000.
  59. V. Vaish, G. Garg, E. V. Talvala, E. Antunez, B. Wilburn, M. Horowitz, and M. Levoy, "Synthetic aperture focusing using a shear-warp factorization of the viewing transform," in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop, San Diego, CA, pp. 129-129, 2005.
  60. R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, and P. Hanrahan, "Light field photography with a hand-held plenoptic camera," Computer Science Technical Report, vol. 2, pp. 1-11, 2005.
  61. A. Lumsdaine and T. Georgiev, "The focused plenoptic camera," in Proceedings of IEEE International Conference on Computational Photography, San Francisco, CA, pp. 1-8, 2009.
  62. K. Fife, A. El Gamal, and H. S. P. Wong, "A multi-aperture image sensor with 0.7 ${\mu}$m pixels in 0.11 ${\mu}m$ CMOS technology," IEEE Journal of Solid-State Circuits, vol. 43, no. 12, pp. 2990-3005, 2008. https://doi.org/10.1109/JSSC.2008.2006457
  63. J. Arai, F. Okano, H. Hoshino, and I. Yuyama, "Gradient-index lens-array method based on real-time integral photography for three-dimensional images," Applied Optics, vol. 37, no. 11, pp. 2034-2045, 1998. https://doi.org/10.1364/AO.37.002034
  64. M. S. Min, J. Hong, and B. Lee, "Analysis of an optical depth converter used in a three-dimensional integral imaging system," Applied Optics, vol. 43, no. 23, pp. 4539-4549, 2004. https://doi.org/10.1364/AO.43.004539
  65. J. Arai, H. Kawai, M. Kawakita, and F. Okano, "Depth-control method for integral imaging," Optics Letters, vol. 33, no. 3, pp. 279-281, 2008. https://doi.org/10.1364/OL.33.000279
  66. H. Navarro, R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, "3D integral imaging display by smart pseudoscopic-to-orthoscopic conversion (SPOC)," Optics Express, vol. 18, no. 25, pp. 25573-25583, 2010. https://doi.org/10.1364/OE.18.025573
  67. J. Arai, M. Okui, M. Kobayashi, and F. Okano, "Geometrical effects of positional errors in integral photography," Journal of the Optical Society of America A, vol. 21, no. 6, pp. 951-958, 2004.
  68. B. Tavakoli, M. Daneshpanah, B. Javidi, and E. Watson, "Performance of 3D integral imaging with position uncertainty," Optics Express, vol. 15, no. 19, pp. 11889-11902, 2007. https://doi.org/10.1364/OE.15.011889
  69. A. Aggoun, "Pre-processing of integral images for 3-D displays," Journal of Display Technology, vol. 2, no. 4, pp. 393-400, 2006. https://doi.org/10.1109/JDT.2006.884691
  70. N. P. Sgouros, S. S. Athineos, M. S. Sangriotis, P. G. Papageorgas, and N. G. Theofanous, "Accurate lattice extraction in integral images," Optics Express, vol. 14, no. 22, pp. 10403-10409, 2006. https://doi.org/10.1364/OE.14.010403
  71. S. Jung, J. H. Park, H. Choi, and B. Lee, "Viewing-angleenhanced integral three-dimensional imaging along all directions without mechanical movement," Optics Express, vol. 11, no. 12, pp. 1346-1356, 2003. https://doi.org/10.1364/OE.11.001346
  72. B. Lee, S. Jung, and J. H. Park, "Viewing-angle-enhanced integral imaging by lens switching," Optics Letters, vol. 27, no. 10, pp. 818-820, 2002. https://doi.org/10.1364/OL.27.000818
  73. J. S. Jang and B. Javidi, "Improvement of viewing angle in integral imaging by use of moving lenslet arrays with low fill factor," Applied Optics, vol. 42, no. 11, pp. 1996-2002, 2003. https://doi.org/10.1364/AO.42.001996
  74. R. Martinez-Cuenca, H. Navarro, G. Saavedra, B. Javidi, and M. Martinez-Corral, "Enhanced viewing-angle integral imaging by multiple-axis telecentric relay system," Optics Express, vol. 15, no. 24, pp. 16255-16260, 2007. https://doi.org/10.1364/OE.15.016255
  75. M. Martinez-Corral, B. Javidi, R. Martinez-Cuenca, and G. Saavedra, "Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays," Applied Optics, vol. 43, no. 31, pp. 5806-5813, 2004. https://doi.org/10.1364/AO.43.005806
  76. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, "Extended depth-of-field 3-D display and visualization by combination of amplitude-modulated microlenses and deconvolution tools," Journal of Display Technology, vol. 1, no. 2, pp. 321-327, 2005. https://doi.org/10.1109/JDT.2005.858883
  77. A. Castro, Y. Frauel, and B. Javidi, "Integral imaging with large depth of field using an asymmetric phase mask," Optics Express, vol. 15, no. 16, pp. 10266-10273, 2007. https://doi.org/10.1364/OE.15.010266
  78. R. Martinez-Cuenca, G. Saavedra, A. Pons, B. Javidi, and M. Martinez-Corral, "Facet braiding: a fundamental problem in integral imaging," Optics Letters, vol. 32, no. 9, pp. 1078-1080, 2007. https://doi.org/10.1364/OL.32.001078
  79. H. Navarro, R. Martinez-Cuenca, A. Molina-Martin, M. Martinez-Corral, G. Saavedra, and B. Javidi, "Method to remedy image degradations due to facet braiding in 3D integral-imaging monitors," Journal of Display Technology, vol. 6, no. 10, pp. 404-411, 2010. https://doi.org/10.1109/JDT.2010.2052347
  80. F. Okano, J. Arai, K. Mitani, and M. Okui, "Real-time integral imaging based on extremely high resolution video system," Proceedings of the IEEE, vol. 94, no. 3, pp. 490-501, 2006. https://doi.org/10.1109/JPROC.2006.870687
  81. T. Mishina, "3D television system based on integral photography," in Proceedings of the 28th Picture Coding Symposium, Nagoya, Japan, pp. 20-20, 2010.
  82. J. Arai, F. Okano, M. Kawakita, M. Okui, Y. Haino, M. Yoshimura,... and M. Sato, "Integral three-dimensional television using a 33-megapixel imaging system," Journal of Display Technology, vol. 6, no. 10, pp. 422-430, 2010. https://doi.org/10.1109/JDT.2010.2050192
  83. O. Matoba, E. Tajahuerce, and B. Javidi, "Real-time threedimensional object recognition with multiple perspectives imaging," Applied Optics, vol. 40, no. 20, pp. 3318-3325, 2001. https://doi.org/10.1364/AO.40.003318
  84. S. Kishk and B. Javidi, "Improved resolution 3D object sensing and recognition using time multiplexed computational integral imaging," Optics Express, vol. 11, no. 26, pp. 3528-3541, 2003. https://doi.org/10.1364/OE.11.003528
  85. S. H. Hong and B. Javidi, "Distortion-tolerant 3D recognition of occluded objects using computational integral imaging," Optics Express, vol. 14, no. 25, pp. 12085-12095, 2006. https://doi.org/10.1364/OE.14.012085
  86. R. Schulein, C. M. Do, and B. Javidi, "Distortion-tolerant 3D recognition of underwater objects using neural networks," Journal of the Optical Society of America A, vol. 27, no. 3, pp. 461-468, 2010. https://doi.org/10.1364/JOSAA.27.000461
  87. S. Yeom, B. Javidi, and E. Watson, "Three-dimensional distortiontolerant object recognition using photon-counting integral imaging," Optics Express, vol. 15, no. 4, pp. 1513-1533, 2007. https://doi.org/10.1364/OE.15.001513
  88. B. Tavakoli, B. Javidi, and E. Watson, "Three dimensional visualization by photon counting computational integral imaging," Optics Express, vol. 16, no. 7, pp. 4426-4436, 2008. https://doi.org/10.1364/OE.16.004426
  89. M. DaneshPanah, B. Javidi, and E. Watson, "Three dimensional object recognition with photon counting imagery in the presence of noise," Optics Express, vol. 18, no. 25, pp. 26450-26460, 2010. https://doi.org/10.1364/OE.18.026450
  90. I. Moon and B. Javidi, "Three-dimensional recognition of photonstarved events using computational integral imaging and statistical sampling," Optics Letters, vol. 34, no. 6, pp. 731-733, 2009. https://doi.org/10.1364/OL.34.000731
  91. D. Aloni, A. Stern, and B. Javidi, "Three-dimensional photon counting integral imaging reconstruction using penalized maximum likelihood expectation maximization," Optics Express, vol. 19, no. 20, pp. 19681-19687, 2011. https://doi.org/10.1364/OE.19.019681
  92. S. H. Hong and B. Javidi, "Three-dimensional visualization of partially occluded objects using integral imaging," Journal of Display Technology, vol. 1, no. 2, pp. 354-359, 2005. https://doi.org/10.1109/JDT.2005.858879
  93. I. Moon and B. Javidi, "Three-dimensional visualization of objects in scattering medium by use of computational integral imaging," Optics Express, vol. 16, no. 17, pp. 13080-13089, 2008. https://doi.org/10.1364/OE.16.013080
  94. J. S. Jang and B. Javidi, "Three-dimensional integral imaging of micro-objects," Optics Letters, vol. 29, no. 11, pp. 1230-1232, 2004. https://doi.org/10.1364/OL.29.001230
  95. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, "Light field microscopy," ACM Transactions on Graphics, vol. 25, no. 3, pp. 924-934, 2006. https://doi.org/10.1145/1141911.1141976
  96. B. Javidi, I. Moon, and S. Yeom, "Three-dimensional identification of biological microorganism using integral imaging," Optics Express, vol. 14, no. 25, pp. 12096-12108, 2006. https://doi.org/10.1364/OE.14.012096
  97. M. Levoy, Z. Zhang, and I. McDowall, "Recording and controlling the 4D light field in a microscope using microlens arrays," Journal of Microscopy, vol. 235, no. 2, pp. 144-162, 2009. https://doi.org/10.1111/j.1365-2818.2009.03195.x
  98. D. Shin, M. Cho, and B. Javidi, "Three-dimensional optical microscopy using axially distributed image sensing," Optics Letters, vol. 35, no. 21, pp. 3646-3648, 2010. https://doi.org/10.1364/OL.35.003646
  99. H. Navarro, M. Martinez-Corral, B. Javidi, E. Sanchez-Ortiga, A. Doblas, and G. Saavedra, "Axial segmentation of 3D images through syntetic-apodization integral-imaging microscopy," in Proceedings of Focus on Microscopy Conference, Konstanz, Germany, 2011.

Cited by

  1. Enhanced volumetric imaging in 2-photon microscopy via acoustic lens beam shaping vol.11, pp.2, 2014, https://doi.org/10.1002/jbio.201700050
  2. Ownership protection of plenoptic images by robust and reversible watermarking vol.107, pp.None, 2014, https://doi.org/10.1016/j.optlaseng.2018.03.028
  3. A Cost Effective Process for Large Area Photonic Array Applied in Light Emitting Diodes for Light Extraction Enhancement vol.8, pp.9, 2019, https://doi.org/10.1149/2.0081909jss
  4. Examining the utility of pinhole-type screens for lightfield display vol.29, pp.21, 2014, https://doi.org/10.1364/oe.438827