Browse > Article
http://dx.doi.org/10.3745/JIPS.01.0082

A Survey on Image Emotion Recognition  

Zhao, Guangzhe (College of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture)
Yang, Hanting (College of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture)
Tu, Bing (School of Information Science and Technology, Hunan Institute of Science and Technology)
Zhang, Lei (College of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture)
Publication Information
Journal of Information Processing Systems / v.17, no.6, 2021 , pp. 1138-1156 More about this Journal
Abstract
Emotional semantics are the highest level of semantics that can be extracted from an image. Constructing a system that can automatically recognize the emotional semantics from images will be significant for marketing, smart healthcare, and deep human-computer interaction. To understand the direction of image emotion recognition as well as the general research methods, we summarize the current development trends and shed light on potential future research. The primary contributions of this paper are as follows. We investigate the color, texture, shape and contour features used for emotional semantics extraction. We establish two models that map images into emotional space and introduce in detail the various processes in the image emotional semantic recognition framework. We also discuss important datasets and useful applications in the field such as garment image and image retrieval. We conclude with a brief discussion about future research trends.
Keywords
Emotion Semantics; Image Emotion Recognition; Image Feature Extraction; Machine Learning;
Citations & Related Records
연도 인용수 순위
  • Reference
1 R. Plutchik, "The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice," American Scientist, vol. 89, no. 4, pp. 344-350, 2001.   DOI
2 Q. Dai and Y. Yu, "A kind of image retrieval based on texture features and BP neural network," Computer Science, vol. 27, no. 6, pp. 55-57, 2000.   DOI
3 Z. Sun, G. Bebis, and R. Miller, "Quantized wavelet features and support vector machines for on-road vehicle detection," in Proceedings of 7th International Conference on Control, Automation, Robotics and Vision, Singapore, 2002, pp. 1641-1646.
4 S. Corchs, E. Fersini, and F. Gasparini, "Ensemble learning on visual and textual data for social image emotion classification," International Journal of Machine Learning and Cybernetics, vol. 10, no. 8, pp. 2057-2070, 2019.   DOI
5 L. Poretski, J. Lanir, and O. Arazy, "Feel the image: the role of emotions in the image-seeking process," Human-Computer Interaction, vol. 34, no. 3, pp. 240-277, 2019.   DOI
6 D. Kong, X. Shen, L. Cao, and G. Jin, "Phase retrieval for attacking fractional Fourier transform encryption," Applied Optics, vol. 56, no. 12, pp. 3449-3456, 2017.   DOI
7 S. Wang, E. Chen, J. Li, and X. Wang, "Kansei-based image evaluation and retrieval," Pattern Recognition and Artificial Intelligence, vol. 14, no. 3, pp. 297-301, 2001   DOI
8 R. V. Priya, "Emotion recognition from geometric fuzzy membership functions," Multimedia Tools and Applications, vol. 78, no. 13, pp. 17847-17878, 2019.   DOI
9 W. Wang and Q. He, "Emotion-based image semantic query through color description," Journal of South China University of Technology (Natural Science Edition), vol. 36, no. 1, pp. 60-66, 2008.   DOI
10 C. C. Chang and C. J. Lin, "LIBSVM: a library for support vector machines," ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 3, article no. 27, 2011. https://doi.org/10.1145/1961189. 1961199   DOI
11 S. Wang, K. Han, and J. Jin, "Review of image low-level feature extraction methods for content-based image retrieval," Sensor Review, vol. 39, no. 6, pp. 783-809, 2019.   DOI
12 J. Yi, A. Chen, Z. Cai, Y. Sima, M. Zhou, and X. Wu, "Facial expression recognition of intercepted video sequences based on feature point movement trend and feature block texture variation," Applied Soft Computing, vol. 82, article no. 105540, 2019. https://doi.org/10.1016/j.asoc.2019.105540   DOI
13 A. Barman and P. Dutta, "Facial expression recognition using distance and texture signature relevant features," Applied Soft Computing, vol. 77, pp. 88-105, 2019.   DOI
14 S. M. Alarcao, "Reminiscence therapy improvement using emotional information," in Proceedings of 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, 2017, pp. 561-565.
15 J. Yao, Y. Yu, and X. Xue, "Sentiment prediction in scene images via convolutional neural networks," in Proceedings of 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC), 2 Wuhan, China, 016, pp. 196-200.
16 C. Colombo, A. Del Bimbo, and P. Pala, "Semantics in visual information retrieval," IEEE Multimedia, vol. 6, no. 3, pp. 38-53, 1999.   DOI
17 H. Sadeghi and A. A. Raie, "Human vision inspired feature extraction for facial expression recognition," Multimedia Tools and Applications, vol. 78, no. 21, pp. 30335-30353, 2019.   DOI
18 A. Hernandez-Garcia, "Perceived emotion from images through deep neural networks," in Proceedings of 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, 2017, pp. 566-570.
19 S. Zhao, Y. Gao, X. Jiang, H. Yao, T. S. Chua, and X. Sun, "Exploring principles-of-art features for image emotion recognition," in Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, 2014, pp. 47-56.
20 N. Jamil, S. Lqbal, and N. Iqbal, "Face recognition using neural networks," in Proceedings of IEEE International Multi Topic Conference: Technology for the 21st Century (INMIC), Lahore, Pakistan, 2001, pp. 277-281.
21 S. Zhao, H. Yao, Y. Gao, R. Ji, and G. Ding, "Continuous probability distribution prediction of image emotions via multitask shared sparse regression," IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 632-645, 2016.   DOI
22 T. Hayashi and M. Hagiwara, "Image query by impression words-the IQI system," IEEE Transactions on Consumer Electronics, vol. 44, no. 2, pp. 347-352, 1998.   DOI
23 F. Liu, B. Wang, and Q. Zhang, "Deep learning of pre-classification for fast image retrieval," in Proceedings of the 2018 International Conference on Algorithms, Computing and Artificial Intelligence, Sanya, China, 2018, pp. 1-5.
24 T. C. Merlo, I. Soletti, E. Saldana, B. S. Menegali, M. M. Martins, A. C. B. Teixeira, S. D. S. Harada-Padermo, M. D. B. Dargelio, and C. J. Contreras-Castillo, "Measuring dynamics of emotions evoked by the packaging colour of hamburgers using Temporal Dominance of Emotions (TDE)," Food Research International, vol. 124, pp. 147-155, 2019.   DOI
25 S. Zhao, X. Zhao, G. Ding, and K. Keutzer, "EmotionGAN: unsupervised domain adaptation for learning discrete probability distributions of image emotions," in Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Korea, 2018, pp. 1319-1327.
26 K. Stallman, Emotional Psychology. Liaoning, China: Liaoning People Press, 1986.
27 M. Y. Tsalamlal, M. A. Amorim, J. C. Martin, and M. Ammi, "Combining facial expression and touch for perceiving emotional valence," IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 437-449, 2016.   DOI
28 D. Palani and K. Venkatalakshmi, "An IoT based predictive modelling for predicting lung cancer using fuzzy cluster based segmentation and classification," Journal of Medical Systems, vol. 43, article no. 21, 2019. https://doi.org/10.1007/s10916-018-1139-7   DOI
29 C. Singh, E. Walia, and K. P. Kaur, "Enhancing color image retrieval performance with feature fusion and non-linear support vector machine classifier," Optik, vol. 158, pp. 127-141, 2018.   DOI
30 A. R. Kurup, M. Ajith, and M. M. Ramon, "Semi-supervised facial expression recognition using reduced spatial features and deep belief networks," Neurocomputing, vol. 367, pp. 188-197, 2019.   DOI
31 L. Cai, H. Xu, Y. Yang, and J. Yu, "Robust facial expression recognition using RGB-D images and multichannel features," Multimedia Tools and Applications, vol. 78, no. 20, pp. 28591-28607, 2019.   DOI
32 S. Wang, E. Chen, J. Li, and X. Wang, "Kansei-based image evaluation and retrieval," Pattern Recognition and Artificial Intelligence, vol. 14, no. 3, pp. 297-301, 2001.   DOI
33 Y. Yu, Z. Tian, and Z. Cai, "Research of perceptive information of image," Acta Electronica Sinica, vol. 29, no. 10, pp. 1373-1375, 2001.   DOI
34 Z. Li, Z. Tan, L. Cao, H. Chen, L. Jiao, and Y. Zhong, "Directive local color transfer based on dynamic lookup table," Signal Processing: Image Communication, vol. 79, pp. 1-12, 2019.   DOI
35 H. Zhang, T. Huang, and L. Liu, "Study on the emotion factor space of fabric," in Proceedings of the International Conference on Kansei Engineering and Emotion Research, Penghu, Taiwan, 2012, pp. 1174-1178.
36 J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, UK: Cambridge University Press, 2004.
37 S. Wang, E. Chen, Z. Wang, and Z. Wang, "Research of emotion semantic image annotation and retrieval algorithm using support vector machine," Pattern Recognition and Artificial Intelligence, vol. 17, no. 1, pp. 27-33, 2004.   DOI
38 Y. Ye, X. Zhang, Y. Lin, and H. Wang, "Facial expression recognition via region-based convolutional fusion network," Journal of Visual Communication and Image Representation, vol. 62, pp. 1-11, 2019.   DOI
39 Q. You, J. Luo, H. Jin, and J. Yang, "Building a large scale dataset for image emotion recognition: the fine print and the benchmark," in Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, 2016, pp. 308-314.
40 J. Lee and E. Park, "Fuzzy similarity-based emotional classification of color images," IEEE Transactions on Multimedia, vol. 13, no. 5, pp. 1031-1039, 2011.   DOI
41 Y. Y. Gao, X. P. Wang, and Y. X. Yin, "Research on affective annotation for natural scene images," Journal of Chinese Computer Systems, vol. 32, no. 4, pp. 767-771, 2011.