Smart Media Journal (스마트미디어저널)

THE KOREAN INSTITUTE OF SMART MEDIA (한국스마트미디어학회)
권호 표지
DOI QR코드

DOI QR Code

Sentiment Analysis on 'HelloTalk' App Reviews Using NRC Emotion Lexicon and GoEmotions Dataset

  • Simay Akar (Department of Management Information Systems in Keimyung University) ;
  • Yang Sok Kim (Department of Management Information Systems, Keimyung University) ;
  • Mi Jin Noh (Department of Business Big Data, Keimyung University)
  • Received : 2024.04.29
  • Accepted : 2024.06.10
  • Published : 2024.06.28
Download PDF
⟨ Previous Next ⟩

Abstract

During the post-pandemic period, the interest in foreign language learning surged, leading to increased usage of language-learning apps. With the rising demand for these apps, analyzing app reviews becomes essential, as they provide valuable insights into user experiences and suggestions for improvement. This research focuses on extracting insights into users' opinions, sentiments, and overall satisfaction from reviews of HelloTalk, one of the most renowned language-learning apps. We employed topic modeling and emotion analysis approaches to analyze reviews collected from the Google Play Store. Several experiments were conducted to evaluate the performance of sentiment classification models with different settings. In addition, we identified dominant emotions and topics within the app reviews using feature importance analysis. The experimental results show that the Random Forest model with topics and emotions outperforms other approaches in accuracy, recall, and F1 score. The findings reveal that topics emphasizing language learning and community interactions, as well as the use of language learning tools and the learning experience, are prominent. Moreover, the emotions of 'admiration' and 'annoyance' emerge as significant factors across all models. This research highlights that incorporating emotion scores into the model and utilizing a broader range of emotion labels enhances model performance.

Keywords

References

  1. Forbes(2020).https://www.forbes.com/sites/mergermarket/2020/04/15/language-learning-apps-are-seeing-a-surge-in-interest-during-the-covid-19-pandemic/?sh=1d1a9eb048f4(accessed Feb., 5, 2024).
  2. Language Exchange App Market - Growth, Trends and Forecast (2024-2032)(2024). https://www.linkedin.com/pulse/language-exchange-app-market-growth-trends-39fvf/ (accessed Mar., 21, 2024).
  3. A. V. Rivera, "HelloTalk," CALICO Journal, vol. 34, no. 3, pp. 384- 92, 2017. https://doi.org/10.1558/cj.32432
  4. A. Yadav, and D. K. Vishwakarma, "Sentiment Analysis Using Deep Learning Architectures: A Review," Artificial Intelligence Review, vol. 53, no. 6, pp. 4335- 4385, 2020. https://doi.org/10.1007/s10462-019-09794-5
  5. A. Iqbal, R. Amin, J. Iqbal, R. Alroobaea, A. Binmahfoudh, and M. Hussain, "Sentiment Analysis of Consumer Reviews Using Deep Learning," Sustainability, vol. 14, no. 17: 10844, Aug. 2022.
  6. R. Das, and T. D. Singh, "Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges," ACM Computing Surveys, vol. 55, no. 13s, pp. 1- 38, Jul. 2023. https://doi.org/10.1145/3494520
  7. D. M. Blei, "Latent Dirichlet Allocation," Journal of Machine Learning Research, 3, pp. 993-1022, 2003.
  8. B. C. Lee, and D. S. Kim, "Deriving the Determinants of Hotel Service Quality using OTA Reviews - LDA Topic Modeling -," Journal of Hotel & Resort, vol. 19, no. 4, pp. 41-58, Aug. 2020.
  9. P. Nandwani, & R. Verma, "A review on sentiment analysis and emotion detection from text, " Social Network Analysis and Mining, vol. 11, no. 1, 81, Dec. 2021.
  10. M. S. I. Malik, and A. Hussain, "Helpfulness of Product Reviews as a Function of Discrete Positive and Negative Emotions," Computers in Human Behavior, vol. 73, pp. 290- 302, Mar. 2017. https://doi.org/10.1016/j.chb.2017.03.053
  11. P. Ekman, "An Argument for Basic Emotions," Cognition and Emotion, vol. 6, no. 3, pp. 169- 200, 1992. https://doi.org/10.1080/02699939208411068
  12. S. M. Mohammad, and P. D Turney, "NRC Emotion Lexicon," National Research Council Canada, Nov. 2013.
  13. Y. L. S. Krishna, P. Paramesh, Y. T. Kumar and A. Gopi, "Sentiment Analysis of Product Reviews by using Naive Bayes and Vader Models," ICSSIT, Tirunelveli, India, pp. 1-6, 2023.
  14. D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, "GoEmotions: A Dataset of Fine-Grained Emotions," arXiv, 2005.00547, Jun. 2020.
  15. R. W. Picard, "Affective Computing," MIT Press, 1997.
  16. L. A. M. Bostan and R. Klinger, "An analysis of annotated corpora for emotion classification in text," ICCL, pp. 2104- 2119, 2018.
  17. H. B. Jung, S. J. Jang, and B.C. Bae, "A Text-Based Emotion Analysis of the Audiences in a Webtoon-Based TV Drama: Focusing on Uncanny Counter>," Journal of Digital Contents Society, vol. 24, no. 4, pp. 755- 64, Apr. 2023. https://doi.org/10.9728/dcs.2023.24.4.755
  18. T. Sosea, C. Pham, A. Tekle, C. Caragea, and J. J. Li, "Emotion Analysis and Detection during COVID-19," arXiv, Jul. 2022.
  19. Kantorov, Vadim, and Ivan Laptev, "Efficient Feature Extraction, Encoding, and Classification for Action Recognition," CVPR, pp. 2593- 2600, 2014.