Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Light Field Angular Super-Resolution Algorithm Using Dilated Convolutional Neural Network with Residual Network

잔차 신경망과 팽창 합성곱 신경망을 이용한 라이트 필드 각 초해상도 기법

  • Kim, Dong-Myung (School of Electronics Engineering, Chung-Buk National University) ;
  • Suh, Jae-Won (School of Electronics Engineering, Chung-Buk National University)
  • Received : 2020.10.16
  • Accepted : 2020.10.29
  • Published : 2020.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Light field image captured by a microlens array-based camera has many limitations in practical use due to its low spatial resolution and angular resolution. High spatial resolution images can be easily acquired with a single image super-resolution technique that has been studied a lot recently. But there is a problem in that high angular resolution images are distorted in the process of using disparity information inherent among images, and thus it is difficult to obtain a high-quality angular resolution image. In this paper, we propose light field angular super-resolution that extracts an initial feature map using an dilated convolutional neural network in order to effectively extract the view difference information inherent among images and generates target image using a residual neural network. The proposed network showed superior performance in PSNR and subjective image quality compared to existing angular super-resolution networks.

마이크로렌즈 어레이 기반의 카메라로 촬영된 라이트필드 영상은 낮은 공간해상도 및 각해상도로 인하여 실제 사용하기에는 많은 제약이 따른다. 고해상도의 공간해상도 영상은 최근 많이 연구되고 있는 단일 영상 초해상도 기법으로 쉽게 얻을 수 있으나 고해상도의 각해상도 영상은 영상사이에 내재된 시점차 정보를 이용하는 과정에서 왜곡이 발생하여 좋은 품질의 각해상도 영상을 얻기 힘든 문제가 있다. 본 논문에서는 영상 사이에 내재된 시점차 정보를 효과적으로 추출하기 위해서 팽창 합성곱 신경망을 이용하여 초기 특징맵을 추출하고 잔차 신경망으로 새로운 시점 영상을 생성하는 라이트 필드 각 초해상도 영상 기법을 제안한다. 제안하는 네트워크는 기존의 각 초해상도 네트워크와 비교하여 PSNR 및 주관적 화질 비교에서 우수한 성능을 보였다.

Keywords

References

  1. M. Levoy and P. Hanrahan, "Light field rendering," in Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 31-42, 1996.
  2. J. Yu, "A Light-Field Journey to Virtual Reality," IEEE MultiMedia, vol. 24, no. 2, pp. 104-112, 2017. https://doi.org/10.1109/MMUL.2017.24
  3. Williem and I. K. Park, "Robust light field depth estimation for noisy scene with occlusion," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4396-4404, 2016.
  4. H. Schilling, M. Diebold, C. Rother, and B. Jahne, "Trust your model: Light field depth estimation with inline occlusion handling," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4530-4538, 2018.
  5. O. Johannsen, A. Sulc, N. Marniok, and B. Goldluecke, "Layered scene reconstruction from multiple light field camera views," in Asian Conference on Computer Vision, pp. 3-18, 2016.
  6. A. Vianello, J. Ackermann, R. B. Campus, M. Diebold, and B. Jahne, "Robust hough transform based 3D reconstruction from circular light fields," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7327-7335, 2018.
  7. C. Dong, C. C. Loy, K. He, and X. Tang. "Image Super-Resolution Using Deep Convolutional Networks," IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 2, pp. 295-307, Feb. 2016. https://doi.org/10.1109/TPAMI.2015.2439281
  8. B. Lim, S. Son, H. Kim, S. Nah and K. M. Lee, "Enhanced Deep Residual Networks for Single Image Super-Resolution," in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136-144, 2017.
  9. M. Haris, G. Shakhnarovich, and N. Ukita, "Deep Back-Projection Networks for Super-Resolution," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1664-1673, 2018.
  10. N. K. Kalantari, T. C. Wang, and R. Ramamoorthi, "Learning-based view synthesis for light field cameras," ACM Transactions on Graphics, vol. 35, no. 6, pp. 193:1 -193:10, Nov. 2016.
  11. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, pp. 1097-1105, 2012.
  12. Y. Yoon, H. G. Jeon, D. Yoo, J. Y. Lee, and I. S. Kweon, "Light-Field Image Super-Resolution Using Convolutional Neural Network," IEEE Signal Processing Letters, vol. 24, no. 6, pp. 848-852, Jun. 2017. https://doi.org/10.1109/LSP.2017.2669333
  13. F. Yu and V. Koltun, "Multi-Scale Context Aggregation by Dilated Convolutions," in Proceedings of 4th International Conference on Learning Representations, 2016.
  14. J. Zbontar and Y. Lecun, "Stereo matching by training a convolutional neural network to compare image patches," International Journal of Machine Learning Research, vol. 17, no. 1, pp. 2287-2318, Apr. 2016.
  15. M. Rerabek and T. Ebrahimi, "New Light Field Image Dataset," in 8th International Conference on Quality of Multimedia Experience, 2016.