Browse > Article
http://dx.doi.org/10.15701/kcgs.2020.26.3.87

Neural Relighting using Specular Highlight Map  

Lee, Yeonkyeong (Visual Computing Lab., Kookmin University)
Go, Hyunsung (Visual Computing Lab., Kookmin University)
Lee, Jinwoo (Visual Computing Lab., Kookmin University)
Kim, Junho (Visual Computing Lab., Kookmin University)
Publication Information
Journal of the Korea Computer Graphics Society / v.26, no.3, 2020 , pp. 87-97 More about this Journal
Abstract
In this paper, we propose a novel neural relighting that infers a relighted rendering image based on the user-guided specular highlight map. The proposed network utilizes a pre-trained neural renderer as a backbone network learned from the rendered image of a 3D scene with various lighting conditions. We jointly optimize a 3D light position and its associated relighted image by back-propagation, so that the difference between the base image and the relighted image is similar to the user-guided specular highlight map. The proposed method has the advantage of being able to explicitly infer the 3D lighting position, while providing the artists' preferred 2D screen-space interface. The performance of the proposed network was measured under the conditions that can establish ground truths, and the average error rate of light position estimations is 0.11, with the normalized 3D scene size.
Keywords
Neural Relighting; Neural Renderer; Specular Highlight Map; Light Position Estimation;
Citations & Related Records
연도 인용수 순위
  • Reference
1 M. Pharr, W. Jakob, and G. Humphreys, Physically Based Rendering: From Theory to Implementation, 3rd ed. Morgan Kaufmann, 2017.
2 A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. NieBner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B. Goldman, and M. Zollhofer, "State of the Art on Neural Rendering," Eurographics STAR, vol. 39, no. 2, pp. 701-727, 2020.
3 A. KT, P. Sakurikar, S. Saini, and P. J. Narayanan, "A Flexible Neural Renderer for Material Visualization," in SIGGRAPH Asia 2019 Technical Briefs, 2019, pp. 83-86.
4 J. Thies, M. Zollhofer, and M. NieBner, "Deferred Neural Rendering: Image Synthesis using Neural Textures," ACM Trans. Graphics, vol. 38, no. 4, pp. 66:1-66:12, 2019.
5 Z. Chen, A. Chen, G. Zhang, C. Wang, Y. Ji, K. N. Kiriakos, and J. Yu, "A Neural Rendering Framework for Free-Viewpoint Relighting," in Proc. CVPR, 2020, pp. 5599-5610.
6 M.-A. Gardner, K. Sunkavalli, E. Yumer, X. Shen, E. Gambaretto, C. Gagne, and J.-F. Lalonde, "Learning to Predict Indoor Illumination from a Single Image," ACM Trans. Graphics, vol. 36, no. 6, pp. 176:1-176:14, 2017.
7 M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagne, and J.-F. Lalondeonde, "Deep Parametric Indoor Lighting Estimation," in Proc. ICCV, 2019, pp. 7175-7183.
8 T. Y. Wang, T. Ritschel, and N. J. Mitra, "Joint Material and Illumination Estimation from Photo Sets in the Wild," in Proc. 3DV, 2018, pp. 22-31.
9 H. Zhou, S. Hadap, K. Sunkavalli, and D. W. Jacobs, "Deep Single-Image Portrait Relighting," in Proc. ICCV, 2019, pp. 7194-7202.
10 P. Kan and H. Kafumann, "DeepLight: light source estimation for augmented reality using deep learning," The Visual Computer, vol. 35, no. 6-8, pp. 873-883, 2019.   DOI
11 L. Hosek and A. Wilkie, "An Analytic Model for Full Spectral Sky-Dome Radiance," ACM Trans. Graphics, vol. 31, no. 4, pp. 95:1-95:9, 2012.
12 P.-P. J. Sloan, W. Martin, A. Gooch, and B. Gooch, "The Lit Sphere: A Model for Capturing NPR Shading from Art," in Proc. Graphics Interface, 2001, pp. 143-150.
13 A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, "Image Analogies," in Proc. ACM SIGGRAPH 2001, 2001, pp. 327-340.
14 P. Benard, F. Cole, M. Kass, I. Mordatch, J. Hegarty, M. S. Senn, K. Fleischer, D. Pesare, and K. Breeden, "Stylizing Animation By Example," ACM Trans. Graphics, vol. 32, no. 4, pp. 119:1-119:12, 2013.
15 J. Fiser, O. Jamriska, M. Lukac, E. Shechtman, P. Asente, J. Lu, and D. Sykora, "StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings," ACM Trans. Graphics, vol. 35, no. 4, pp. 92:1-92:11, 2016.
16 L. Zhang, E. Simo-Serra, Y. Ji, and C. Liu, "Generating Digital Painting Lighting Effects via RGB-space Geometry," ACM Trans. Graphics, vol. 39, no. 2, pp. 13:1-13:13, 2020.
17 "Blender," https://www.blender.org/.