Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

HVS-Aware Single-Shot HDR Imaging Using Deep Convolutional Neural Network

시각 인지 특성과 딥 컨볼루션 뉴럴 네트워크를 이용한 단일 영상 기반 HDR 영상 취득

  • Vien, An Gia (Pukyong National University, Department of Computer Engineering) ;
  • Lee, Chul (Pukyong National University, Department of Computer Engineering)
  • Received : 2018.03.22
  • Accepted : 2018.04.26
  • Published : 2018.05.30
Download PDF
⟨ Previous Next ⟩

Abstract

We propose a single-shot high dynamic range (HDR) imaging algorithm using a deep convolutional neural network (CNN) for row-wise varying exposures in a single image. The proposed algorithm restores missing information resulting from under- and/or over-exposed pixels in an input image and reconstructs the raw radiance map. The main contribution of this work is the development of a loss function for the CNN employing the human visual system (HVS) properties. Then, the HDR image is obtained by applying a demosaicing algorithm. Experimental results demonstrate that the proposed algorithm provides higher-quality HDR images than conventional algorithms.

본 논문은 딥 컨볼루션 뉴럴 네트워크(CNN)를 이용하여 행 별로 서로 다른 노출로 촬영된 단일 영상을 HDR 영상으로 변환하는 기법을 제안한다. 제안하는 알고리즘은 먼저 입력 영상에서 저조도 또는 포화로 인해 발생하는 정보 손실 영역을 CNN을 이용하여 복원하여 휘도맵을 생성한다. 또한, CNN 학습 과정에서 인간의 시각 인지 특성을 고려할 수 있는 손실 함수를 제안한다. 마지막으로 복원된 휘도맵에 디모자이킹 필터를 적용하여 최종 HDR 영상을 획득한다. 컴퓨터 모의실험을 통해 제안하는 알고리즘이 기존의 기법에 비해서 높은 품질의 HDR 영상을 취득하는 것을 확인한다.

Keywords

References

  1. P. Sen and C. Aguerrebere, "Practical High Dynamic Range Imaging of Everyday Scenes: Photographing The World as We Can See It with Our Own Eyes," IEEE Signal Processing Magazine, Vol. 33, pp. 36-44, Sep. 2016.
  2. P. E. Debevec and J. Malik, "Recovering High Dynamic Range Radiance Maps from Photographs," Proceeding of ACM SIGGRAPH, Aug. pp. 369-378, 1997.
  3. T. Mitsunaga and S. K. Nayar, "Radiometric Self-Calibration," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp.374-380, Jun. 1999.
  4. M. Aggarwal and N. Ahuja, "Split Aperture Imaging for High Dynamic Range," Proceeding of IEEE International Conference on Computer Vision, Vol. 2, pp. 10-17, Jul. 2001.
  5. H. Zimmer, A. Bruhn, and J. Weickert, "Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement," Computer Graphics Forum, Vol. 30, pp. 405-414, Apr. 2011. https://doi.org/10.1111/j.1467-8659.2011.01870.x
  6. P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, "Robust Patch-Based HDR Reconstruction of Dynamic Scenes," ACM Transactions on Graphics, Vol. 31, No. 6, pp. 203:1-203:11, Nov. 2012.
  7. C. Lee, Y. Li, and V. Monga, "Ghost-free High Dynamic Range Imaging via Rank Minimization," IEEE Signal Processing Letters, Vol. 21, No.9, pp. 1045-1049, Sep. 2014. https://doi.org/10.1109/LSP.2014.2323404
  8. Y. Li, C. Lee, and V. Monga, “A Maximum a Posteriori Estimation Framework for Robust High Dynamic Range Video Synthesis,” IEEE Transactions on Image Processing, Vol. 26, No. 3, pp. 1143-1157, Mar. 2017. https://doi.org/10.1109/TIP.2016.2642790
  9. C. Lee and E. Y. Lam, “Computationally Efficient Truncated Nuclear Norm Minimization for High Dynamic Range Imaging,” IEEE Transactions on Image Processing, Vol. 25, No. 9, pp. 4145-4157, Sep. 2016. https://doi.org/10.1109/TIP.2016.2585047
  10. S. K. Nayar and T. Mitsunaga, "High Dynamic Range Imaging: Spatially Varying Pixel Exposures," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 472-479, Jun. 2000.
  11. J. Gu, Y. Hitomi, T. Mitsunaga, and S. Nayar, "Coded Rolling Shutter Photography: Flexible Space-Time Sampling," Proceeding of IEEE International Conference on Computational Photography, pp. 1-8, Mar. 2010.
  12. K. Hirakawa and P. M. Simon, "Single-Shot High Dynamic Range Imaging with Conventional Camera Hardware," Proceeding of IEEE International Conference on Computer Vision, pp. 1339-1346, Nov. 2011.
  13. C. Aguerrebere, A. Almansa, Y. Gousseau, J. Delon, and P. Muse, "Single Shot High Dynamic Range Imaging Using Piecewise Linear Estimators," Proceeding of IEEE International Conference on Computational Photography, pp. 1-10, May 2014.
  14. H. Cho, S. J. Kim, and S. Lee, “Single-Shot High Dynamic Range Imaging Using Coded Electronic Shutter,” Computer Graphics Forum, Vol. 33, No. 7, pp. 329-338, Oct. 2014. https://doi.org/10.1111/cgf.12501
  15. A. Serrano, F. Heide, D. Gutierrez, G. Wetzstein, and B. Masia, “Convolutional Sparse Coding for High Dynamic Range Imaging,” Computer Graphics Forum, Vol. 35, No. 2, pp. 153-163, May 2016. https://doi.org/10.1111/cgf.12819
  16. J. Li, C. Bai, Z. Lin, and J. Yun, "Penrose High-Dynamic-Range Imaging," Journal of Electronic Imaging, Vol. 25, No. 3, pp. 033 024:1-13, Jun. 2016.
  17. V. G. An and C. Lee, "Single-Shot High Dynamic Range Imaging via Deep Convolutional Neural Network," Proceeding of APSIPA ASC, pp. 1768-1772, Dec. 2017.
  18. S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," Proceeding of International Conference on Machine Learning, pp. 448-456, Jul. 2015.
  19. V. Nair and G. E. Hilton, "Rectified Linear Units Improve Restricted Boltzman Machines," Proceeding of International Conference on Machine Learning, pp. 807-814, Jun. 2010.
  20. J. Johson, A. Alahi, and F. F. Li, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution," Proceeding of European Conference on Computer Vision, pp. 694-711, Oct. 2016.
  21. N. K. Kalantari and R. Ramamoorthi, "Deep High Dynamic Range Imaging of Dynamic Scenes," ACM Transactions on Graphics, Vol. 36, No. 4, Jul. 2017.
  22. G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, "HDR Image Reconstruction from a Single Exposure Using Deep CNNs," ACM Transactions on Graphics, Vol. 36, No. 6, Nov. 2017.
  23. R. Mantiuk, K. Myszkowski, and H. P. Seidel, "Lossy Compression of High Dynamic Range Images and Video," Proceeding of SPIE, Vol. 6057, Feb. 2006.
  24. O. Blackwell and H. Blackwell, “Visual Performance Data for 156 Normal Observers of Various Ages,” Journal of the Illuminating Engineering Society, Vol. 1, No. 1, pp. 3-13, 1971. https://doi.org/10.1080/00994480.1971.10732194
  25. J. E. Adams, "Design of Practical Color Filter Array Interpolation Algorithms for Digital Cameras," Proceeding of SPIE, Vol. 3028, pp. 117-125, Feb. 1997.
  26. T. O. Aydin, R. Mantiuk, and H. P. Seidel, "Extending Quality Metrics to Full Luminance Range Images," Proceeding of SPIE, Vol. 6806, p. 68060B, Mar. 2008.
  27. M. Narwaria, R. K. Mantiuk, M. P. D. Silva, and P. L. Callet, "HDR-VDP-2.2: A Calibrated Method for Objective Quality Prediction of High Dynamic Range and Standard Images," Journal of Electronic Imaging, Vol. 24, No. 1, Jan. 2015.