Browse > Article
http://dx.doi.org/10.5909/JBE.2022.27.1.104

Super High-Resolution Image Style Transfer  

Kim, Yong-Goo (Dept. of AI Software Eng., Seoul Media Institute of Technology)
Publication Information
Journal of Broadcast Engineering / v.27, no.1, 2022 , pp. 104-123 More about this Journal
Abstract
Style transfer based on neural network provides very high quality results by reflecting the high level structural characteristics of images, and thereby has recently attracted great attention. This paper deals with the problem of resolution limitation due to GPU memory in performing such neural style transfer. We can expect that the gradient operation for style transfer based on partial image, with the aid of the fixed size of receptive field, can produce the same result as the gradient operation using the entire image. Based on this idea, each component of the style transfer loss function is analyzed in this paper to obtain the necessary conditions for partitioning and padding, and to identify, among the information required for gradient calculation, the one that depends on the entire input. By structuring such information for using it as auxiliary constant input for partition-based gradient calculation, this paper develops a recursive algorithm for super high-resolution image style transfer. Since the proposed method performs style transfer by partitioning input image into the size that a GPU can handle, it can perform style transfer without the limit of the input image resolution accompanied by the GPU memory size. With the aid of such super high-resolution support, the proposed method can provide a unique style characteristics of detailed area which can only be appreciated in super high-resolution style transfer.
Keywords
Deep Learning; Style Transfer; Super High-Resolution; Receptive Field; Gradient Descent;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 H. Wang, Y. Li, H. Ju, and M.-H. Yang, "Collaborative distillation for ultra-resolution universal style transfer," arXiv preprint arXiv:2003.08436, 2020.
2 R. Hecht-Nielsen, "Theory of the backpropagation neural network," in Proc. of the Int. Joint Conf. on Neural Networks(IJCNN), pp. 593-605, 1989. doi: 10.1109/IJCNN.1989.118638.   DOI
3 Fast photo style, https://github.com/NVIDIA/FastPhotoStyle (accessed 13 Dec. 2021)
4 E. Risser, P. Wilmot, and C. Barnes, "Stable and controllable neural texture synthesis and style transfer using histogram losses," arXiv preprint arXiv:1701.08893, 2018.
5 M. Abadi, et. al., "Tensorflow: A system for large-scale machine learning," arXiv preprint arXiv:1605.08695, 2016.
6 T. Strothotte and S. Schlechtweg, "Non-photorealistic computer graphics: modeling, rendering, and animation", Elsevier Science, USA, 2002.
7 Choi and Y.-G. Kim, "A normalized loss function of style transfer network for more diverse and more stable transfer results," J. of Broadcast Engineering, vol.25, no.6, pp.980-993, 2020. doi:10.5909/JBE.2020.25.6.980.   DOI
8 J. Johnson, A. Alahi, and L. Fei-Fei, "Perceptual losses for real-time style transfer and super-resolution," Proc. of the European Conf. on Computer Vision(ECCV), pp.694-711, 2016. doi:10.1007/978-3-319-46475-6_43.   DOI
9 H. Zhang and K. Dana, "Multi-style generative network for real-time transfer," arXiv preprint arXiv:1703.06953, 2017.
10 L. Sheng, Z. Lin, J. Shao, and X. Wang, "Avartar-net: multi-scale zero- shot style transfer by feature decoration," arXiv preprint arXiv:1805.03857, 2018.
11 K. Simonyan and A. Zissermann, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556v6, 2015.
12 Neural style transfer using tf.keras, https://www.tensorflow.org/tutorials/generative/style_transfer (accessed 13 Dec. 2021)
13 A. Sanakoyeu, D. Kotovenko, S. Lang, and B. Ommer, "A style-aware content loss for real-time HD style transfer," Proc. of the European Conf. on Computer Vision(ECCV), pp.698-714, 2018. doi:10.1007/978-3-030-01237-3_43.   DOI
14 P. Rosin and J. Collomosse, "Image and video-based artistic stylization", Springer-Verlag, London, 2013.
15 D. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980v9, 2014.
16 L. A. Gatys, A. S. Ecker, and M. Bethge, "Image style transfer using convolutional neural networks," Proc. of the IEEE int. Conf. on Computer Vis. and Patt. Recog.(CVPR), pp.2414-2423, 2016. doi:10.1109/CVPR.2016.265.   DOI
17 A. Efros and T. Leung, "Texture synthesis by non-parametric sampling," Proc. of the IEEE Int. Conf. on Computer Vision(ICCV), pp.1033-1038, 1999. doi:10.1109/ICCV.1999.790383.   DOI
18 Y. Li, N. Wang, J. Liu, and X. Hou, "Demystifying neural style transfer," Proc. of the 26th Int. Joint Conf. on Artificial Intelligence(IJCAI), pp. 2230-2236, 2017. doi:10.5555/3172077.3172198.   DOI
19 H. Huang, H. Wang, W. Luo, L. Ma, W. Jiang, X. Zhu, Z. Li, and W. Liu, "Real-time neural style transfer for videos," Proc. of the IEEE Conf. on Computer Vis. and Patt. Recog.(CVPR), pp. 7044-7052, 2017. doi: 10.1109/CVPR.2017.745.   DOI
20 M. Ruder, A. Dosovitskiy, and T. Brox, "Artistic style transfer for videos and spherical images," Int. J. of Computer Vision, vol.126, no.11, pp.1199-1219, 2018. doi:10.1007/s11263-018-1089-z.   DOI
21 V. Dumoulin, J. Shlens, and M. Kudlur, "A learned representation for artistic style," Proc. of the Int. Conf. on Learning Representations, arXiv preprint arXiv:1610.07629, 2016.
22 L. Gatys, A. Ecker, M. Bethge, A. Hertzman, and E. Shechtman, "Controlling perceptual factors in neural style transfer," arXiv preprint arXiv: 1611.07865, 2016.
23 X. Huang and S. Belongie, "Arbitrary style transfer in real-time with adaptive instance normalization," Proc. of the IEEE Int. Conf. on Computer Vision(ICCV), pp.1510-1519, 2017. doi:10.1109/ICCV.2017.167.   DOI
24 Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M. Yang, "Universal style transfer via feature transforms," arXiv preprint arXiv:1705.08086, 2017.
25 D. Heeger and J. Bergen, "Pyramid-based texture analysis/synthesis," Proc. of the 22nd annual conf. on computer graphics and interactive techniques, pp.229-238, 1995. doi:10.1145/218380.218446.   DOI
26 Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song, "Neural Style Transfer: A Review," IEEE Trans. on Visualization and Computer Graphics, vol.26, no.11, pp.3365-3385, 2020. doi:10.1109/TVCG.2019.2921336.   DOI