Browse > Article
http://dx.doi.org/10.3837/tiis.2021.04.015

Stylized Image Generation based on Music-image Synesthesia Emotional Style Transfer using CNN Network  

Xing, Baixi (Zhejiang University of Technology)
Dou, Jian (Hangzhou Dianzi University)
Huang, Qing (Hangzhou Dianzi University)
Si, Huahao (Hangzhou Dianzi University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.4, 2021 , pp. 1464-1485 More about this Journal
Abstract
Emotional style of multimedia art works are abstract content information. This study aims to explore emotional style transfer method and find the possible way of matching music with appropriate images in respect to emotional style. DCNNs (Deep Convolutional Neural Networks) can capture style and provide emotional style transfer iterative solution for affective image generation. Here, we learn the image emotion features via DCNNs and map the affective style on the other images. We set image emotion feature as the style target in this style transfer problem, and held experiments to handle affective image generation of eight emotion categories, including dignified, dreaming, sad, vigorous, soothing, exciting, joyous, and graceful. A user study was conducted to test the synesthesia emotional image style transfer result with ground truth user perception triggered by the music-image pairs' stimuli. The transferred affective image result for music-image emotional synesthesia perception was proved effective according to user study result.
Keywords
Affective Computing; Image Style Transfer; Deep Convolutional Neural Networks;
Citations & Related Records
연도 인용수 순위
  • Reference
1 C. Zhou, Z. Gu, Y. Gao, and J. Wang, "An Improved Style Transfer Algorithm Using Feedforward Neural Network for Real Time Image Conversion," Sustainability, vol. 11, no. 20, Oct. 2019.
2 D. Liang, D. Liang, S. Xing, P. Li, and X. Wu, "A robot calligraphy writing method based on style transferring algorithm and similarity evaluation," Intelligent Service Robotics, vol. 13, no. 1, pp. 137-146, Jan. 2020.   DOI
3 F. Luan, S. Paris, E. Shechtman, and K. Bala, "Deep Photo Style Transfer," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp.6997-7005, 2017.
4 Y. Liu, W. Chen, L. Liu, and M. Lew, "SwapGAN: A Multistage Generative Approach for Person-to-Person Fashion Style Transfer," IEEE Transactions on Multimedia, vol. 21, no. 9, pp. 2209-2222, Sep. 2019.   DOI
5 P. Andreini, S. Bonechi, M. Bianchini, A. Mecocci, and F. Scarselli, "Image generation by GAN and style transfer for agar plate image segmentation," Computer Methods and Programs in Biomedicine, vol. 184, no. 105268, Feb. 2020.
6 Z. Zhong, L. Zheng, Z. Zheng, S. Li, and Y. Yang, "CamStyle: A Novel Data Augmentation Method for Person Re-Identification," IEEE Transactions on Image Processing, vol. 28, no. 3, pp. 1176-1191, Mar. 2019.   DOI
7 J. Chen, G. Yang, H. Zhao, and M. Ramasamy, "Audio style transfer using shallow convolutional networks and random filters," Multimedia Tools and Applications, vol. 79, no. 21-22, pp. 15043-15057, June 2020.   DOI
8 K. Zsolnai-Feher, P. Wonka, and M. Wimmer, "Gaussian Material Synthesis," ACM Transactions on Graphics, vol. 37, no. 4, pp. 1-14, Aug. 2018.
9 B. Xing, K. Zhang, S. Sun, L. Zhang, Z. Gao, J. Wang, and S. Chen, "Emotion-driven Chinese folk music-image retrieval based on DE-SVM," Neurocomoputing, vol. 148, pp. 619-627, Jan. 2015.   DOI
10 J. Zhu, T. Park, P. Isola, and A. Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2242-2251, 2017.
11 H. Huang, H. Wang, W. Luo, L. Ma, W. Jiang, X. Zhu, Z. Li, and W. Liu, "Real-Time Neural Style Transfer for Videos," in Proc. of IEEE Conference on Computer Vision & Pattern Recognition, pp. 7044-7052, 2017.
12 H. Wu, Z. Sun, Y. Zhang, and Q. Li, "Direction-aware neural style transfer with texture enhancement," Neurocomputing, vol. 370, no. 22, pp. 39-55, Dec. 2019.   DOI
13 B. Kim, G. Kim, and S. Lee, "Style-controlled synthesis of clothing segments for fashion image manipulation," IEEE Transactions on Multimedia, vol. 22, no. 2, pp. 298-310, Feb. 2020.   DOI
14 L. Zhan and Y. Wang, "Stable and Refined Style Transfer Using Zigzag Learning Algorithm," Neural Processing Letters, vol. 50, pp. 2481-2492, Mar. 2019.   DOI
15 Y. Zhou, R. Jiang, X. Wu, J. He, S. Weng, and Q. Peng, "BranchGAN: Unsupervised Mutual Image-to-Image Transfer with A Single Encoder and Dual Decoders," IEEE Transactions on Multimedia, vol. 21, no. 12, pp. 3136-3149, Dec. 2019.   DOI
16 J. Johnson, A. Alahi, and F. Li, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution," in Proc. of European Conference on Computer Vision, pp. 694-711, 2016.
17 M. Guo and J. Jiang, "A robust deep style transfer for headshot portraits," Neurocomputing, vol. 361, pp.164-172, Oct. 2019.   DOI
18 L. Gatys, A. Ecker, and M. Bethge, "Image Style Transfer Using Convolutional Neural Networks," in Proc. of IEEE Computer Society Conference on Computer Vision & Pattern Recognition, pp. 2414-2424, 2016.
19 M. Cheng, X. Liu, J. Wang, S. Lu, Y. Lai, and P. Rosin, "Structure-preserving neural style transfer," IEEE Transactions on Image Processing, vol. 29, pp. 909-920, Aug. 2019.   DOI
20 Z. Li, F. Zhou, L. Yang,X. Li , and J. Li, "Accelerate neural style transfer with super-resolution," Multimedia Tools and Applications, vol. 79, pp. 4347-4364, Feb. 2020.   DOI
21 J. Yaniv, Y. Newman, and A. Shamir, "The face of art: landmark detection and geometric style in portraits," ACM Transactions on Graphics, vol. 38, no. 4, pp. 1-15, July 2019.
22 M. Huzaifah and L. Wyse, "Applying Visual Domain Style Transfer and Texture Synthesis Techniques to Audio - Insights and Challenges," Neural Computing and Applications, vol. 32, no. 4 , pp. 1051-1065, Feb. 2019.   DOI
23 R. Novak and Y. Nikulin, "Improving the neural algorithm of artistic style," arXiv:1605.04603, pp. 1-15, May 2016.
24 Z. Lian, B. Zhao, X. Chen, and J. Xiao, "Easyfont: a style learning-based system to easily build your large-scale handwriting fonts," ACM Transactions on Graphics, vol. 38, no. 1, pp.1-18, Feb. 2019.
25 K. Hevner, "Experimental studies of the elements of expression in music," The American Journal of Psychology, vol. 48, no. 2, pp. 246-268, Apr. 1936.   DOI
26 O. Jamriska, S. Sochorova, O. Texler, M. Lukac, J. Fiser, J. Lu, E. Shechtman, and D. Sykora, "Stylizing video by example," ACM Transactions on Graphics, vol. 38, no.4, pp. 1-11, July 2019.
27 M. Ruder, A. Dosovitskiy, and T. Brox, "Artistic style transfer for videos," in Proc. of German Conference on Pattern Recognition, pp. 26-36, 2016.
28 D. Aliaga, P. Rosen, and D. Bekins, "Style Grammars for Interactive Visualization of Architecture," IEEE Transactions on visualization and computer graphics, vol. 13, no. 4, pp. 786-798, July 2007.   DOI
29 H. Kwon, H. Yoon, and K. Park, "CAPTCHA Image Generation: Two-Step Style-Transfer Learning in Deep Neural Networks," Sensors, vol. 20, no. 5, Mar. 2020.
30 A. Khan, M. Ahmad, N. Naqvi, F. Yousafzai, and J. Xiao, "Photographic painting style transfer using convolutional neural networks," Multimedia Tools and Applications, vol. 78, pp. 19565-19586, Feb. 2019.   DOI
31 L. Gatys, A. Ecker, and M. Bethge, "A Neural Algorithm of Artistic Style," Journal of Vision, vol. 16, no. 12, pp. 1-16, Aug. 2016.   DOI
32 Y. Shih, W. Lai, and C. Liang, "Distortion-free wide-angle portraits on camera phones," ACM Transactions on Graphics, vol. 38, no. 4, pp. 1-12, July 2019.   DOI
33 D. Guo and T. Sim, "Digital face makeup by example," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 73-79, 2009.
34 X. Zhang, X. Zhang, and Z. Xiao, "Deep photographic style transfer guided by semantic correspondence," Multimedia Tools and Applications, vol. 78, pp. 34649-34672, Dec. 2019.   DOI
35 Y. Gao, Y. Guo, Z. Lian, M. Tang, and J. Xiao, "Artistic Glyph Image Synthesis via One-Stage Few-Shot Learning," ACM Transactions on Graphics, vol. 38, no. 6, pp. 1-12, Nov. 2019.