Browse > Article
http://dx.doi.org/10.15701/kcgs.2021.27.3.31

Deep Learning-Based Lighting Estimation for Indoor and Outdoor  

Lee, Jiwon (KAIST, Visual Media Lab)
Seo, Kwanggyoon (KAIST, Visual Media Lab)
Lee, Hanui (KAIST, Visual Media Lab)
Yoo, Jung Eun (KAIST, Visual Media Lab)
Noh, Junyong (KAIST, Visual Media Lab)
Publication Information
Journal of the Korea Computer Graphics Society / v.27, no.3, 2021 , pp. 31-42 More about this Journal
Abstract
We propose a deep learning-based method that can estimate an appropriate lighting of both indoor and outdoor images. The method consists of two networks: Crop-to-PanoLDR network and LDR-to-HDR network. The Crop-to-PanoLDR network predicts a low dynamic range (LDR) environment map from a single partially observed normal field of view image, and the LDR-to-HDR network transforms the predicted LDR image into a high dynamic range (HDR) environment map which includes the high intensity light information. The HDR environment map generated through this process is applied when rendering virtual objects in the given image. The direction of the estimated light along with ambient light illuminating the virtual object is examined to verify the effectiveness of the proposed method. For this, the results from our method are compared with those from the methods that consider either indoor images or outdoor images only. In addition, the effect of the loss function, which plays the role of classifying images into indoor or outdoor was tested and verified. Finally, a user test was conducted to compare the quality of the environment map created in this study with those created by existing research.
Keywords
lighting estimation; deep learning; HDR environment map; indoor lighting; outdoor lighting;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Zhang, K. Sunkavalli, Y. Hold-Geoffroy, S. Hadap, J. Eisenmann, and J.-F. Lalonde, "All-weather deep outdoor lighting estimation," 2019.
2 P. Debevec, "Image-based lighting," IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 26-34, 2002.   DOI
3 P. Kuo, C. Tang, and S. Chien, "Content-adaptive inverse tone mapping," in 2012 Visual Communications and Image Processing, 2012, pp. 1-6.
4 J. Zhang and J.-F. Lalonde, "Learning high dynamic range from outdoor panoramas," 2017.
5 T. Rhee, L. Petikam, B. Allen, and A. Chalmers, "Mr360: Mixed reality rendering for 360° panoramic videos," IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 4, pp. 1379-1388, 2017.   DOI
6 S. Ning, H. Xu, L. Song, R. Xie, and W. Zhang, "Learning an inverse tone mapping network with a generative adversarial regularizer," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 1383-1387.
7 F. Banterle, P. Ledda, K. Debattista, and A. Chalmers, "Inverse tone mapping," in Proceedings of the 4th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia, ser. GRAPHITE '06. New York, NY, USA: Association for Computing Machinery, 2006, p. 349-356. [Online]. Available: https://doi.org/10.1145/1174429.1174489   DOI
8 A. G. Rempel, M. Trentacoste, H. Seetzen, H. D. Young, W. Heidrich, L. Whitehead, and G. Ward, "Ldr2hdr: On-thefly reverse tone mapping of legacy video and photographs," ACM Trans. Graph., vol. 26, no. 3, p. 39-es, July 2007. [Online]. Available: https://doi.org/10.1145/1276377.1276426   DOI
9 Y. Kinoshita and H. Kiya, "itm-net: Deep inverse tone mapping using novel loss function considering tone mapping operator," IEEE Access, vol. 7, pp. 73 555-73 563, 2019.   DOI
10 P. Debevec, P. Graham, J. Busch, and M. Bolas, "A single-shot light probe," in ACM SIGGRAPH 2012 Talks, ser. SIGGRAPH '12. New York, NY, USA: Association for Computing Machinery, 2012. [Online]. Available: https://doi.org/10.1145/2343045.2343058   DOI
11 M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagne, and J.-F. Lalonde, "Deep parametric indoor lighting estimation," 2019.
12 K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," 2015.
13 J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired imageto-image translation using cycle-consistent adversarial networks," 2020.
14 D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," 2017.
15 M. Fan, D.-H. Lee, S. Kim, and S.-J. Ko, "An optimization framework for inverse tone mapping using a single low dynamic range image," Signal Process. Image Commun., vol. 78, pp. 274-283, 2019.   DOI
16 Y.-L. Liu, W.-S. Lai, Y.-S. Chen, Y.-L. Kao, M.-H. Yang, Y.- Y. Chuang, and J.-B. Huang, "Single-image hdr reconstruction by learning to reverse the camera pipeline," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
17 J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba, "Recognizing scene viewpoint using panoramic place representation," in 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 2695-2702.
18 M. Garon, K. Sunkavalli, S. Hadap, N. Carr, and J.-F. Lalonde, "Fast spatially-varying indoor lighting estimation," 2019.
19 Y. Hold-Geoffroy, K. Sunkavalli, S. Hadap, E. Gambaretto, and J. Lalonde, "Deep outdoor illumination estimation," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2373-2382.
20 M.-A. Gardner, K. Sunkavalli, E. Yumer, X. Shen, E. Gambaretto, C. Gagne, and J.-F. Lalonde, "Learning to predict indoor illumination from a single image," 2017.
21 Y. Hold-Geoffroy, A. Athawale, and J.-F. Lalonde, "Deep sky modeling for single image outdoor lighting estimation," 2019.
22 S. Song and T. Funkhouser, "Neural illumination: Lighting prediction for indoor environments," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6911-6919, 2019.
23 P. E. Debevec and J. Malik, "Recovering high dynamic range radiance maps from photographs," in Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, ser. SIGGRAPH '97. USA: ACM Press/Addison-Wesley Publishing Co., 1997, p. 369-378. [Online]. Available: https://doi.org/10.1145/258734.258884   DOI
24 X. Li and Y. Huo, "Hdr image reconstruction using locally weighted linear regression," in 2018 IEEE International Conference on Multimedia Expo Workshops (ICMEW), 2018, pp. 1-6.
25 P. Debevec, "Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography," 1998.
26 S. Lee, G. H. An, and S.-J. Kang, "Deep recursive hdri: Inverse tone mapping using generative adversarial networks," in Proceedings of the European Conference on Computer Vision (ECCV), September 2018.