Korean Journal of Optics and Photonics (한국광학회지)

Optical Society of Korea (한국광학회)
권호 표지
DOI QR코드

DOI QR Code

Three-Dimensional Conversion of Two-Dimensional Movie Using Optical Flow and Normalized Cut

Optical Flow와 Normalized Cut을 이용한 2차원 동영상의 3차원 동영상 변환

  • Jung, Jae-Hyun (School of Electrical Engineering, Seoul National University) ;
  • Park, Gil-Bae (School of Electrical Engineering, Seoul National University) ;
  • Kim, Joo-Hwan (School of Electrical Engineering, Seoul National University) ;
  • Kang, Jin-Mo (School of Electrical Engineering, Seoul National University) ;
  • Lee, Byoung-Ho (School of Electrical Engineering, Seoul National University)
  • Published : 2009.02.25
Download PDF
⟨ Previous Next ⟩

Abstract

We propose a method to convert a two-dimensional movie to a three-dimensional movie using normalized cut and optical flow. In this paper, we segment an image of a two-dimensional movie to objects first, and then estimate the depth of each object. Normalized cut is one of the image segmentation algorithms. For improving speed and accuracy of normalized cut, we used a watershed algorithm and a weight function using optical flow. We estimate the depth of objects which are segmented by improved normalized cut using optical flow. Ordinal depth is estimated by the change of the segmented object label in an occluded region which is the difference of absolute values of optical flow. For compensating ordinal depth, we generate the relational depth which is the absolute value of optical flow as motion parallax. A final depth map is determined by multiplying ordinal depth by relational depth, then dividing by average optical flow. In this research, we propose the two-dimensional/three-dimensional movie conversion method which is applicable to all three-dimensional display devices and all two-dimensional movie formats. We present experimental results using sample two-dimensional movies.

본 논문에서는 2차원 동영상을 normalized cut과 optical flow를 이용하여 3차원 동영상으로 변환하는 방법을 제안하였다. 이를 통해 특정 디스플레이 장치와 특정 동영상 포맷에 국한되지 않는 2차원 동영상의 3차원 동영상 변환 방법을 제안하였다. 본 연구에서는 2차원 동영상의 3차원 변환을 위하여 먼저 영상을 객체로 분할하고, 분할된 객체의 깊이를 추정하는 방법을 사용하였다. Normalized cut은 영상분할의 한 방법으로, 본 연구에서는 연산속도 향상을 위하여 기존 방법에 watershed 알고리즘을 적용하였고, 정확도 향상을 위하여 가중치에 optical flow를 추가하였다. Normalized cut을 이용하여 분할된 영상의 깊이 정보를 추정하기 위하여 optical flow를 이용하였다. Optical flow의 차이를 통해 정의할 수 있는 가려진 영역의 분할 영상 변화를 통해 순서적 깊이 정보를 추정한다. 추정된 순서적 깊이를 보정하기 위해 optical flow의 절대적 크기를 이용해 운동시차로 상대적 깊이를 추정하였다. 최종적으로 추정된 깊이 정보는 순서적 깊이와 상대적 깊이의 곱을 평균 optical flow로 나누어, 순서적 깊이의 차이를 보정하였다. 제안한 방법의 검증을 위하여 2차원 동영상을 3차원 동영상으로 변환하여 깊이 정보가 추정됨을 확인하였다.

Keywords

References

  1. T. Okoshi, “Three-dimensional displays,” Proc. IEEE, vol. 68, no. 5, pp. 548-564, 1980 https://doi.org/10.1109/PROC.1980.11695
  2. J.-H. Park, H.-R. Kim, Y. Kim, J. Kim, J. Hong, S.-D. Lee, and B. Lee, “Depth-enhanced three-dimensional-twodimensional convertible display based on modified integral imaging,” Opt. Lett., vol. 29, no. 23, pp. 2734-2736, 2004 https://doi.org/10.1364/OL.29.002734
  3. H. Choi, J. Kim, S.-W. Cho, Y. Kim, J. B. Park, and B. Lee, “Three-dimensional-two-dimensional mixed display system using integral imaging with an active pinhole array on a liquid crystal panel,” Appl. Opt., vol. 47, no. 13, pp. 2207-2214, 2008 https://doi.org/10.1364/AO.47.002207
  4. T. Okino, G. Murata, K. Taima, T. Iinuma, and K. Oketani, “New television with 2D/3D image conversion technologies,” Proc. SPIE, vol. 2653, pp. 96-103, 1996 https://doi.org/10.1117/12.237421
  5. B. J. Garcia, “Approaches to stereoscopic video based on spatiotemporal interpolation,” Proc. SPIE, vol. 2653, pp. 85-95, 1996 https://doi.org/10.1117/12.237420
  6. Y. Matsumoto, H. Terasaki, K. Sugumoto, and T. Arakawa, “Conversion system of monocular image sequence to stereo using motion parallax,” Proc. SPIE, vol. 3012, pp. 108-115, 1997 https://doi.org/10.1117/12.274446
  7. M. B. Kim, M. S. Song, D. K. Kim, and K. C. Choi, “Stereoscopic conversion of monoscopic video by the transformation of vertical-to-horizontal disparity,” Proc. SPIE, vol. 3295, pp. 65-75, 1998 https://doi.org/10.1117/12.307194
  8. J. Ross, “Stereopsis by binocular delay,” Nature, vol. 248, pp. 363-364, 1974 https://doi.org/10.1038/248363a0
  9. J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, 2000 https://doi.org/10.1109/34.868688
  10. B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, no. 1-3, pp. 185-204, 1981 https://doi.org/10.1016/0004-3702(81)90024-2
  11. 장세일, 황선규, 김회율, “Markov Random Fields를 이용한 움직이는 객체 추출 및 추적,” 대한 전자공학회 하계학술대회, vol. 26, no. 1, pp. 2100-2103, 2003
  12. L. Vincent and P. Soille, “Watersheds in digital spaces: an efficient algorithm based on immersion simulations,” IEEE Transactions of Pattern Analysis and Machine Intelligence, vol. 13, no. 6, pp. 583-598, 1991 https://doi.org/10.1109/34.87344
  13. A. S. Ogale, C. Fermuller, and Y. Aloimonos, “Motion segmentation using occlusions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 988-992, 2005 https://doi.org/10.1109/TPAMI.2005.123