The Journal of Information Systems (한국정보시스템학회지:정보시스템연구)

Korea Association of Information Systems (한국정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Multimodal Sentiment Analysis for Investigating User Satisfaction

  • Hwang, Gyo Yeob (Department of Management Information System, Dong-A University) ;
  • Song, Zi Han (Department of Management Information System, Dong-A University) ;
  • Park, Byung Kwon (Department of Management Information System, Dong-A University)
  • Received : 2023.03.27
  • Accepted : 2023.07.18
  • Published : 2023.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Purpose The proliferation of data on the internet has created a need for innovative methods to analyze user satisfaction data. Traditional survey methods are becoming inadequate in dealing with the increasing volume and diversity of data, and new methods using unstructured internet data are being explored. While numerous comment-based user satisfaction studies have been conducted, only a few have explored user satisfaction through video and audio data. Multimodal sentiment analysis, which integrates multiple modalities, has gained attention due to its high accuracy and broad applicability. Design/methodology/approach This study uses multimodal sentiment analysis to analyze user satisfaction of iPhone and Samsung products through online videos. The research reveals that the combination model integrating multiple data sources showed the most superior performance. Findings The findings also indicate that price is a crucial factor influencing user satisfaction, and users tend to exhibit more positive emotions when content with a product's price. The study highlights the importance of considering multiple factors when evaluating user satisfaction and provides valuable insights into the effectiveness of different data sources for sentiment analysis of product reviews.

Keywords

Acknowledgement

This work was supported by the Dong-A University research fund.

References

  1. Ahuja, S. and Dubey, G., Clustering and sentiment analysis on Twitter data. 2017 2nd International Conference on Telecommunication and Networks (TEL-NET), IEEE, 2017, pp. 1-5. Audio Speech Sentiment.
  2. Chen, M. and Li, X. SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis. Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Stroudsburg, PA, USA, 2020, pp. 1067-1077.
  3. Cui, Z., Qiu, Q., Yin, C., Yu, J., Wu, Z. and Deng, A., "A Barrage Sentiment Analysis Scheme Based on Expression and Tone", IEEE Access, Vol. 7, 2109, pp. 180324-180335.
  4. Duleba, S. and Moslem, S., "User Satisfaction Survey on Public Transport by a New PAHP Based Model", Applied Sciences, Vol. 11, No. 21, 2021, p. 10256.
  5. Hasson, S.G., Piorkowski, J. and McCulloh, I., Social media as a main source of customer feedback. Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ACM, New York, NY, USA, 2109, pp. 829-832.
  6. Hayes, B.E., Measuring customer satisfaction and loyalty: survey design, use, and statistical analysis methods, Quality Press, 2008.
  7. Hu, M. and Liu, B., "Opinion extraction and summarization on the Web", Proceedings of the National Conference on Artificial Intelligence, Vol. 2, 2006, pp. 1621-1624.
  8. Huddar, M.G., Sannakki, S.S. and Rajpurohit, V.S., "A Survey of Computational Approaches and Challenges in Multimodal Sentiment Analysis", International Journal of Computer Sciences and Engineering, Vol. 7, No. 1, 2019, pp. 876-883. https://doi.org/10.26438/ijcse/v7i1.876883
  9. Khan, S.A., Liang, Y. and Shahzad, S., "An Empirical Study of Perceived Factors Affecting Customer Satisfaction to Re-Purchase Intention in Online Stores in China:" Journal of Service Science and Management, Vol. 08, No. 03, 2015, pp. 291-305. https://doi.org/10.4236/jssm.2015.83032
  10. Li, H., Ye, Q. and Law, R., "Determinants of Customer Satisfaction in the Hotel Industry: An Application of Online Review Analysis", Asia Pacific Journal of Tourism Research, Vol. 18, No. 7, 2013, pp. 784-802. https://doi.org/10.1080/10941665.2012.708351
  11. Li, R., Zhao, J., Hu, J., Guo, S. and Jin, Q., Multi-modal Fusion for Video Sentiment Analysis. Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop, ACM, New York, NY, USA, 2020, pp. 19-25.
  12. librosa. Available at: https://librosa.org/doc/latest/index.html (Accessed: February 7, 2023).
  13. Lin, C.-C., Wu, H.-Y. and Chang, Y.-F., "The critical factors impact on online customer satisfaction", Procedia Computer Science,Elsevier Vol. 3, 2011, pp. 276-281. https://doi.org/10.1016/j.procs.2010.12.047
  14. Liu, Y., Jiang, C. and Zhao, H., "Assessing product competitive advantages from the perspective of customers by mining user-generated content on social media:" Decision Support Systems, Vol. 123, 2019, p. 113079.
  15. Madaleno, R., Wilson, H. and Palmer, R., "Determinants of Customer Satisfaction in a Multi-Channel B2B Environment", Total Quality Management & Business Excellence, Vol. 18, No. 8, 2007, pp. 915-925. https://doi.org/10.1080/14783360701350938
  16. Mansor, S.N., Mostafa, S.A., Mustapha, A. and Darman, R., An Emotional Agent for the Analysis of Customer Satisfaction Surveys., 2018 International Symposium on Agent, Multi-Agent Systems and Robotics (ISAMSR), IEEE, 2018, pp. 1-6.
  17. Morency, L.-P., Mihalcea, R. and Doshi, P., Towards multimodal sentiment analysis. Proceedings of the 13th international conference on multimodal interfaces - ICMI '11, ACM Press, New York, New York, USA, 2011, p. 169.
  18. Naas, S.A. and Sigg, S., "Real-time emotion recognition for sales:" Proceedings - 2020 16th International Conference on Mobility, Sensing and Networking, MSN 2020, pp. 584-591.
  19. Perez Rosas, V., Mihalcea, R. and Morency, L.-P., "Multimodal Sentiment Analysis of Spanish Online Videos", IEEE Intelligent Systems, Vol. 28, No. 3, 2013, pp. 38-45. https://doi.org/10.1109/MIS.2013.9
  20. Prentice, C., Dominique Lopes, S. and Wang, X., "The impact of artificial intelligence and employee service quality on customer satisfaction and loyalty", Journal of Hospitality Marketing & Management, Vol. 29, No. 7, 2020, pp. 739-756. https://doi.org/10.1080/19368623.2020.1722304
  21. Schwemmer, C. and Ziewiecki, S., "Social Media Sellout: The Increasing Role of Product Promotion on YouTube", Social Media + Society, Vol. 4, No. 3, 2018, p. 205630511878672.
  22. Seng, K.P. and Ang, L.M., "Video analytics for customer emotion and satisfaction at contact centers", IEEE Transactions on Human-Machine Systems, Vol. 48, No. 3, 2018, pp. 266-278. https://doi.org/10.1109/THMS.2017.2695613
  23. Serengil, S.I. and Ozpinar, A., HyperExtended LightFace: A Facial Attribute Analysis Framework, 2021 International Conference on Engineering and Emerging Technologies (ICEET), IEEE, 2021, pp. 1-4.
  24. Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.F. and Pantic, M., "A survey of multimodal sentiment analysis", Image and Vision Computing,Elsevier B.V. Vol. 65, 2107, pp. 3-14.
  25. TextBlob Homepage. Available at: https://textblob.readthedocs.io/en/dev/(Accessed : June 9, 2022).
  26. Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K. and Morency, L.-P., "YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual Context", IEEE Intelligent Systems, Vol. 28, No. 3, 2013, pp. 46-53. https://doi.org/10.1109/MIS.2013.34
  27. Xu, G., Li, W. and Liu, J., "A social emotion classification approach using multimodel fusion", Future Generation Computer Systems, Vol. 102, 2020, pp. 347-356. https://doi.org/10.1016/j.future.2019.07.007
  28. Yadollahi, A., Shahraki, A.G. and Zaiane, O.R., "Current State of Text Sentiment Analysis from Opinion to Emotion Mining", ACM Computing Surveys, Vol. 50, No. 2, 2018, pp. 1-33. https://doi.org/10.1145/3057270