Browse > Article
http://dx.doi.org/10.15207/JKCS.2018.9.1.419

Monitoring Mood Trends of Twitter Users using Multi-modal Analysis method of Texts and Images  

Kim, Eun Yi (Visual Information Lab., Department of Software, Konkuk University)
Ko, Eunjeong (Visual Information Lab., Department of Software, Konkuk University)
Publication Information
Journal of the Korea Convergence Society / v.9, no.1, 2018 , pp. 419-431 More about this Journal
Abstract
In this paper, we propose a novel method for monitoring mood trend of Twitter users by analyzing their daily tweets for a long period. Then, to more accurately understand their tweets, we analyze all types of content in tweets, i.e., texts and emoticons, and images, thus develop a multimodal sentiment analysis method. In the proposed method, two single-modal analyses first are performed to extract the users' moods hidden in texts and images: a lexicon-based and learning-based text classifier and a learning-based image classifier. Thereafter, the extracted moods from the respective analyses are combined into a tweet mood and aggregated a daily mood. As a result, the proposed method generates a user daily mood flow graph, which allows us for monitoring the mood trend of users more intuitively. For evaluation, we perform two sets of experiment. First, we collect the data sets of 40,447 data. We evaluate our method via comparing the state-of-the-art techniques. In our experiments, we demonstrate that the proposed multimodal analysis method outperforms other baselines and our own methods using text-based tweets or images only. Furthermore, to evaluate the potential of the proposed method in monitoring users' mood trend, we tested the proposed method with 40 depressive users and 40 normal users. It proves that the proposed method can be effectively used in finding depressed users.
Keywords
Convergence; Twitter user recognition; multi-modal analysis; mood trend monitoring; support vector machine; context recognition;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Yuan, Q. You, J. Luo, "Sentiment Analysis Using Social Multimedia," In Multimedia Data Mining and Analytics, Springer International Publishing, pp. 31-59, 2015.
2 S. Tsugawa, Y. Kikuchi, F. Kishino, K. Nakajima, Y. Itoh, H. Ohsaki, "Recognizing depression from Twitter activity," ACM CHI, pp. 3187-3196, 2015.
3 A. Go, L. Huang, R. Bhayani, "Twitter sentiment analysis," Final Projects from CS224N for Spring 2008/2009 at The Stanford Natural Language Processing Group, 2009.
4 Y. R. Tausczik, J. W. Pennebaker, "The psychological meaning of words: LIWC and computerized text analysis methods," Journal of Language and Social Psychology, Vol. 29, pp. 24-54, 2010.   DOI
5 M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, A. Kappas, "Sentiment strength detection in short informal text," Journal of the Association for Information Science and Technology, Vol. 61, pp. 2544-2558, 2010.
6 L. Zhang, R. Ghosh, M. Dekhil, M. Hsu, B. Liu, "Combining Lexicon-based and Learning-based Methods for Twitter Sentiment Analysis," In HP Laboratories Technical Report (89 ed.), 2011.
7 J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, J. Tang, "Can we understand van Gogh's mood?: Learning to infer affects from images in social networks," MM2012, pp. 857-860, 2012.
8 A. Mogadala, V. Varma, "Twitter user behavior understanding with mood transition prediction," The 2012 workshop on Data-driven user behavioral modelling and mining from social media, pp. 31-34. 2012.
9 S. Park, S. W. Lee, J. Kwak, M. Cha, B. Jeong, "Activities on Facebook reveal the depressive state of users," Journal of Medical Internet Research, Vol. 15, 2013.
10 A. Hogenboom, B. Daniella, F. Flavius, B. Malissa, J. Franciska, K. Uzay, "Exploiting emoticons in sentiment analysis," Annual ACM Symposium on Applied Computing 2013, pp. 703-710. 2013.
11 S. M. Mohammad, Saif M., and D. T. Peter, "Crowdsourcing a word-emotion association lexicon," Computational Intelligence Vol. 29, pp. 436-465, 2013.   DOI
12 Y. Yang, C. Peng, Z. Wenwu, Y. Shiqiang, "User interest and social influence based emotion prediction for individuals," MM2013, pp. 785-788, 2013.
13 C. Xu, C. Suleyman, K. Lee, L. Li, "Visual sentiment prediction with deep convolutional neural networks," arXiv preprint arXiv:1411.5731, 2014.
14 H. Lin, J. Jia, Q. Guo, Y. Xue, Q. Li, J. Huang, "User-level psychological stress detection from social media using deep neural network," MM2014, pp. 507-516, 2014.
15 Q. You, J. Luo, H. Jin, J. Yang, "Robust image sentiment analysis using progressively trained and domain transferred deep networks," AAAI2015, pp. 381-388, 2015.
16 https://en.wikipedia.org/wiki/List_of_emoticons Wikipedia
17 S. Tsugawa, Y. Kikuchi, F. Kishino, K. Nakajima, Y. Itoh, H. Ohsaki, "Recognizing depression from Twitter activity," CHI2015, pp. 3187-3196, 2015.
18 D. Borth, C. Tao, J. Rongrong, C. Shih-Fu, "Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content," MM2013, pp. 459-460. 2013.
19 http://sentistrength.wlv.ac.uk/Sentistrength.
20 E.S. Dan-Glauser, K. R. Scherer, "The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance," Behavior research methods, Vol. 43, pp. 468-477, 2011.   DOI
21 http://photo.net Photo.Net.
22 S. Kobayashi, COLOR IMAGE SCALE, Kodansha. 1991.
23 J. Bollen, H. Mao, X. Zeng, "Twitter mood predicts the stock market," Journal of Computational Science, Vol. 2, pp. 1-8, 2010.
24 M. De Chounhury, S. Counts, E. Horvitz, "Predicting postpartum changes in emotion and behavior via social media," CHI 2013, pp. 3267-3276, 2013.
25 H. Sueki, "The association of suicide-related Twitter use with suicidal behaviour: A cross-sectional study of young internet users in Japan," Journal of Affective Disorder, Vol. 170, pp. 155-160, Jan. 2015.   DOI
26 S. Zhao, H. Yao, Y. Yang, Y. Zhang, "Affective image retrieval via multi-graph learning," MM2014, pp. 1025-1028. 2014.
27 M. Marcus, M. T. Yasamy, M. van Ommeren, D. Chisholm, S. Saxena, "Depression: A global public health concern," WHO Department of Mental Health and Substance Abuse Vol. 1, pp. 6-8, 2012.
28 Z. Rihmer, "Can better recognition and treatment of depression reduce suicide rates? A brief review," European Psychiatry, Vol. 16, pp. 406-409, 2001.   DOI
29 L. S. Radloff L S, "The CES-D scale a self-report depression scale for research in the general population," Applied Psychological Measurement, Vol. 1, pp. 385-401, 1977.   DOI
30 G. E. Simon, D. B. Goldberg, G. Tiemens, T. B. Ustun, "Outcomes of recognized and unrecognized depression in an international primary care study," General Hospital Psychiatry, Vol. 21, pp. 97-105, 1999.   DOI
31 M. De Choudhury, M. Gamon, S. Counts, E. Horvitz, "Predicting depression via social media," ICWSM 2013, pp. 1-10, 2013.
32 C.K. Chung, J. W. Pennebaker, "Counting little words in Big Data: The psychology of communities, culture, and history," Social Conition and Communication, Psychology Press, pp. 25-42, 2014.
33 G. Katz, N. Ofek, B. Shapira, "ConSent: Context-based sentiment analysis," Knowledge-Based Systems, Vol. 84, pp. 162-178, 2015.