Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Depth Map Correction Algorithm based on Segmentation in Multi-view Systems

다중시점 환경에서의 슈퍼픽셀 세그먼테이션 기반 깊이 영상 개선 알고리즘

  • 정우경 (세종대학교 전자정보통신공학과) ;
  • 한종기 (세종대학교 전자정보통신공학과)
  • Received : 2020.09.01
  • Accepted : 2020.10.30
  • Published : 2020.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

In immersive media, the most important factor that provides immersion is depth information. Therefore, it is essential to obtain high quality depth information in order to produce high quality immersive media. In this paper we propose an algorithm to improve depth map, considering the segmentation of images and the relationship between multiple views in multi-view systems. The proposed algorithm uses a super-pixel segmentation technique to divide the depth map of the reference view into several segments, and project each segment into adjacent view. Subsequently, the depth map of the adjacent view is improved using plane estimation using the information of the projected segment, and then reversed to the reference view. This process is repeated for several adjacent views to improve the reference depth map by updating the values of the improved adjacent views and the initial depth map of the reference view. Through simulation, the proposed algorithm is shown to surpass the conventional algorithm subjectively and objectively.

실감형 미디어에서 현실감을 느끼게 하는 가장 중요한 요소는 깊이 정보이다. 따라서 고품질의 실감형 미디어를 제작하기 위해서는 고품질의 깊이 정보를 획득하는 것이 필수적이다. 본 논문에서는 고품질의 깊이 정보를 획득하기 위하여 다중 시점 환경에서 깊이 지도를 개선하기 위하여 깊이 지도를 여러 개의 세그먼트로 분할 및 다중 시점간의 관계를 고려하는 알고리즘을 제안한다. 제안된 알고리즘은 슈퍼픽셀 세그먼테이션 기법을 사용하여 기준 시점의 깊이 지도를 여러 세그먼트로 나누고, 각 세그먼트를 인접 시점으로 투영한다. 이후 투영된 세그먼트의 정보를 이용하여 인접 시점의 깊이 지도를 평면 추정을 이용하여 개선한 후, 기준 시점으로 역투영된다. 여러 개의 인접 시점에 대해 이 과정을 반복하여 개선된 인접 시점들의 값들과 기준 시점의 초기 깊이 지도를 가중치 합으로 갱신하여 깊이 지도를 개선한다. 기존 다중 시점 스테레오 비전 알고리즘에 제안된 알고리즘을 적용한 시뮬레이션을 통해 제안된 알고리즘의 결과가 주관적 및 객관적으로 기존 알고리즘을 능가하는 것을 보인다.

Keywords

Acknowledgement

This work was partly supported by the National Research Foundation of Korea (NRF) under Grant NRF-2018R1A2A2A05023117 and partly by Institute for Information & communications Technology Promotion (IITP) under Grant 2017-0-00486 funded by the Korea government (MSIT).

References

  1. H. Hirschmuller, "Stereo Processing by Semiglobal Matching and Mutual Information," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328-341, Feb. 2008. https://doi.org/10.1109/TPAMI.2007.1166
  2. A. Dziembowski, A. Grzelka, D. Mieloch, O. Stankiewicz and M. Domanski, "Depth map upsampling and refinement for FTV systems", 2016 International Conference on Signals and Electronic Systems (ICSES), Krakow, pp. 89-92, 2016.
  3. T. Chang, J. Kuo and J. Yang, "Efficient hole filling and depth enhancement based on texture image and depth map consistency," 2016 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), Jeju, pp. 192-195, 2016.
  4. J. Lu, D. Min, R. S. Pahwa and M. N. Do, "A revisit to MRF-based depth map super-resolution and enhancement," 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, pp. 985-988, 2011.
  5. D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms" Int. Jour. Computer Vision, 47(1/2/3):7-42, 2002 https://doi.org/10.1023/A:1014573219977
  6. T. Schops, J. L. Schonberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, A. Geiger, "A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos", Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  7. H. Shih and H. Hsiao, "A depth refinement algorithm for multi-view video synthesis," 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, Dallas, TX, pp. 742-745, 2010
  8. M. Kurc, O. Stankiewicz and M. Domanski, "Depth map inter-view consistency refinement for multiview video," 2012 Picture Coding Symposium, Krakow, pp. 137-140, 2012.
  9. S. M. Seitz, B. Curless, J. Diebel, D. Scharstein and R. Szeliski, "A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms," 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), New York, NY, USA, 2006, pp. 519-528, doi: 10.1109/CVPR.2006.19.
  10. R. Xiaofeng and J. Malik. "Learning a classification model for segmentation." Proceedings Ninth IEEE International Conference on Computer Vision, pp. 10-17 vol.1, 2003.
  11. Arno Knapitsch, , Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. "Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction".ACM Transactions on Graphics 36, no.4 (2017).
  12. M. A. Fischler and R. C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography'. Commun. ACM 24, pp. 381-395, 1981. https://doi.org/10.1145/358669.358692
  13. T. Schops, J. L. Schonberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, A. Geiger, "A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos", Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  14. https://github.com/cdcseacave/openMVS