Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on Enhancing Personalization Recommendation Service Performance with CNN-based Review Helpfulness Score Prediction

CNN 기반 리뷰 유용성 점수 예측을 통한 개인화 추천 서비스 성능 향상에 관한 연구

  • Li, Qinglong (Department of Big Data Analytics, Kyung Hee University) ;
  • Lee, Byunghyun (Department of Big Data Analytics, Kyung Hee University) ;
  • Li, Xinzhe (Department of Big Data Analytics, Kyung Hee University) ;
  • Kim, Jae Kyeong (School of Management & Department of Big Data Analytics, Kyung Hee University)
  • 이청용 (경희대학교 빅데이터응용학과) ;
  • 이병현 (경희대학교 빅데이터응용학과) ;
  • 이흠철 (경희대학교 빅데이터응용학과) ;
  • 김재경 (경희대학교 경영대학 & 빅데이터응용학과)
  • Received : 2021.05.21
  • Accepted : 2021.07.12
  • Published : 2021.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, various types of products have been launched with the rapid growth of the e-commerce market. As a result, many users face information overload problems, which is time-consuming in the purchasing decision-making process. Therefore, the importance of a personalized recommendation service that can provide customized products and services to users is emerging. For example, global companies such as Netflix, Amazon, and Google have introduced personalized recommendation services to support users' purchasing decisions. Accordingly, the user's information search cost can reduce which can positively affect the company's sales increase. The existing personalized recommendation service research applied Collaborative Filtering (CF) technique predicts user preference mainly use quantified information. However, the recommendation performance may have decreased if only use quantitative information. To improve the problems of such existing studies, many studies using reviews to enhance recommendation performance. However, reviews contain factors that hinder purchasing decisions, such as advertising content, false comments, meaningless or irrelevant content. When providing recommendation service uses a review that includes these factors can lead to decrease recommendation performance. Therefore, we proposed a novel recommendation methodology through CNN-based review usefulness score prediction to improve these problems. The results show that the proposed methodology has better prediction performance than the recommendation method considering all existing preference ratings. In addition, the results suggest that can enhance the performance of traditional CF when the information on review usefulness reflects in the personalized recommendation service.

전자상거래 시장이 빠르게 성장하면서 다양한 유형의 제품이 출시되고 있으며, 이로 인해 사용자들은 구매 의사결정과정에 많은 시간이 소요되는 정보 과부하 문제에 직면하고 있다. 따라서 사용자에게 맞춤형 제품 및 서비스를 제공해줄 수 있는 개인화 추천 서비스의 중요성이 대두되고 있다. 대표적으로 Netflix, Amazon, Google 등 세계적 기업은 개인화 추천 서비스를 도입하여 사용자의 구매 의사결정을 지원하고 있다. 이에 따라 사용자의 정보탐색 비용이 감소하는 효과가 나타났고, 기업의 매출 상승에도 긍정적인 영향을 끼치고 있다. 기존 개인화 추천 서비스 관련 연구에서 주로 사용된 협업필터링(Collaborative Filtering, CF) 기법은 정량화된 정보를 활용하여 사용자의 선호도를 예측하였다. 그러나 정량화된 정보만을 활용하면 사용자의 구매 의도는 고려하지 못하므로 추천 성능이 저하될 수 있다는 문제점이 제기되고 있다. 이와 같은 기존 연구의 문제점을 개선하기 위해 최근에는 사용자가 작성한 리뷰를 활용한 개인화 추천 서비스 연구가 활발히 진행되고 있다. 그러나 리뷰에는 광고성 내용, 거짓 후기, 의미를 전혀 파악할 수 없거나 제품과 관련 없는 내용 등 구매의사결정을 저해하는 요소들이 포함되어 있다. 이러한 요소들이 포함된 리뷰를 활용하여 추천 서비스를 제공하게 되면, 추천 성능이 저하되는 문제가 발생할 수 있다. 따라서 본 연구에서는 이러한 문제점을 개선하기 위해 Convolutional Neural Network(CNN) 기반 리뷰 유용성 점수 예측을 통한 새로운 추천 방법론을 제안하였다. 본 연구에서 제안하는 유용한 리뷰를 포함하는 방법론과 기존 모든 선호도 평점을 고려하는 추천 방법론을 비교한 결과, 본 연구에서 제안한 방법론이 더 우수한 예측 성능을 나타내고 있음을 확인할 수 있었다. 또한 본 연구의 결과는 리뷰 유용성에 대한 정보를 개인화 추천 서비스에 반영하면 전통적인 CF의 성능을 향상할 수 있음을 시사한다.

Keywords

Acknowledgement

본 논문은 교육부 및 한국연구재단의 4단계 두뇌한국21 사업(4단계 BK21 사업)으로 지원된 연구임.

References

  1. Abdollahi, B., and O. Nasraoui, "Using explainability for constrained matrix factorization", Proceedings of the Eleventh ACM Conference on Recommender Systems, (2017), 79~83.
  2. Al-Bashiri, H., M. A. Abdulgabber, A. Romli, and H. Kahtan, "An improved memory-based collaborative filtering method based on the TOPSIS technique", PloS one, Vol. 13, No.10(2018), e0204434. https://doi.org/10.1371/journal.pone.0204434
  3. Ar, Y., and E. Bostanci, "A genetic algorithm solution to the collaborative filtering problem", Expert Systems with Applications, Vol.61 (2016), 122~128. https://doi.org/10.1016/j.eswa.2016.05.021
  4. Baek, H., J. Ahn, and Y. Choi, "Helpfulness of online consumer reviews: Readers' objectives and review cues", International Journal of Electronic Commerce, Vol.17, No.2(2012), 99~126. https://doi.org/10.2753/jec1086-4415170204
  5. Bang, H. B., H. W. Lee, and J. H. Lee, "TV Program Recommender System Using Viewing Time Patterns", Journal of Korean Institute of Intelligent Systems, Vol.25, No.5(2015), 431~436. https://doi.org/10.5391/JKIIS.2015.25.5.431
  6. Barragans-Martinez, A. B., E. Costa-Montenegro, J. C. Burguillo, M. Rey-Lopez, F. A. Mikic-Fonte, and A. Peleteiro, "A hybrid content-based and item-based collaborative filtering approach to recommend TV programs enhanced with singular value decomposition", Information Sciences, Vol.180, No.22(2010), 4290~4311. https://doi.org/10.1016/j.ins.2010.07.024
  7. Bennett, J., and S. Lanning, "The netflix prize", Proceedings of KDD Cup and Workshop, Vol.2007, (2007), 35.
  8. Bobadilla, J., F. Ortega, A. Hernando, and J. Alcala, "Improving collaborative filtering recommender system results and performance using genetic algorithms", Knowledge-Based Systems, Vol.24, No.8(2011), 1310~1316. https://doi.org/10.1016/j.knosys.2011.06.005
  9. Bokde, D., S. Girase, and D. Mukhopadhyay, "Matrix factorization model in collaborative filtering algorithms: A survey", Procedia Computer Science, Vol.49, (2015), 136~146. https://doi.org/10.1016/j.procs.2015.04.237
  10. Cao, R., X. Zhang, and H. Wang, "A Review Semantics Based Model for Rating Prediction", IEEE Access, Vol.8, (2019), 4714~4723. https://doi.org/10.1109/access.2019.2962075
  11. Castelli, M., L. Manzoni, L. Vanneschi, and A. Popovic, "An expert system for extracting knowledge from customers' reviews: The case of Amazon. com, Inc", Expert Systems with Applications, Vol.84, (2017), 117~126. https://doi.org/10.1016/j.eswa.2017.05.008
  12. Chai, T., and R. R. Draxler, "Root mean square error (RMSE) or mean absolute error (MAE)?- Arguments against avoiding RMSE in the literature", Geoscientific Model Development, Vol.7, No.3(2014), 1247~1250. https://doi.org/10.5194/gmd-7-1247-2014
  13. Cheng, Z., Y. Ding, L. Zhu, and M. Kankanhalli, "Aspect-aware latent factor model: Rating prediction with ratings and reviews", Proceedings of the World Wide Web Conference, (2018), 639~648.
  14. Choi, I. Y., M. G. Oh, J. K. Kim, and Y. U. Ryu, "Collaborative filtering with facial expressions for online video recommendation", International Journal of Information Management, Vol.35, No.3(2016), 397~402.
  15. Chung, K. Y., D. Lee, and K. J. Kim, "Categorization for grouping associative items using data mining in item-based collaborative filtering", Multimedia Tools and Applications, Vol.71, No.2(2014), 889~904. https://doi.org/10.1007/s11042-011-0885-z
  16. Cui, C., and T. Fearn, "Modern practical convolutional neural networks for multivariate regression: Applications to NIR calibration", Chemometrics and Intelligent Laboratory Systems, Vol.182, (2018), 9~20. https://doi.org/10.1016/j.chemolab.2018.07.008
  17. Das, A. S., M. Datar, A. Garg, and S. Rajaram, "Google news personalization: scalable online collaborative filtering", Proceedings of the 16th International Conference on World Wide Web, (2007), 271~280.
  18. Elahi, M., F. Ricci, and N. Rubens, "A survey of active learning in collaborative filtering recommender systems", Computer Science Review, Vol.20, (2016), 29~50. https://doi.org/10.1016/j.cosrev.2016.05.002
  19. Fu, M., H. Qu, D. Moges, and L. Lu, "Attention based collaborative filtering", Neurocomputing, Vol.311, (2018), 88~98. https://doi.org/10.1016/j.neucom.2018.05.049
  20. Garcia-Cumbreras, M. A., A. Montejo-Raez, and M. C. Diaz-Galiano, "Pessimists and optimists: Improving collaborative filtering through sentiment analysis", Expert Systems with Applications, Vol.40, No.17(2013), 6758~6765. https://doi.org/10.1016/j.eswa.2013.06.049
  21. Ge, S., T. Qi, C. Wu, F. Wu, X. Xie, and Y. Huang, "Helpfulness-aware review based neural recommendation", CCF Transactions on Pervasive Computing and Interaction, Vol.1, No.4(2019), 285~295. https://doi.org/10.1007/s42486-019-00023-0
  22. Goldberg, D., D. Nichols, B. M. Oki, and D. Terry, "Using collaborative filtering to weave an information tapestry", Communications of the ACM, (1992), 61~70.
  23. Guy, I., M. Avihai, A. Nus, and F. Raiber, "Extracting and Ranking Travel Tips from User-Generated Reviews", Proceedings of the 26th International Conference on World Wide Web, (2017), 987~996.
  24. Hammou, B. A., and A. A. Lahcen, "FRAIPA: A fast recommendation approach with improved prediction accuracy", Expert Systems with Applications, Vol.87, (2017), 90~97. https://doi.org/10.1016/j.eswa.2017.06.001
  25. He, X., L. Liao, H. Zhang, L. Nie, X. Hu, and T.S. Chua, "Neural collaborative filtering", Proceedings of the 26th International Conference on World Wide Web, (2017), 173~182.
  26. Herlocker, J. L., J. A. Konstan, L. G. Terveen, and J. T. Riedl, "Evaluating collaborative filtering recommender systems", ACM Transactions on Information Systems, Vol.22, No.1(2004), 5~53. https://doi.org/10.1145/963770.963772
  27. Hu, Y. H., Y. L. Chen, and H. L. Chou, "Opinion mining from online hotel reviews-a text summarization approach", Information Processing & Management, Vol.53, No.2(2017), 436~449. https://doi.org/10.1016/j.ipm.2016.12.002
  28. Hyun, J., S. Ryu, and S. Y. Lee, "How to improve the accuracy of recommendation systems: Combining ratings and review texts sentiment score", Journal of Intelligence and Information Systems, Vol.25, No.1(2019), 219~239. https://doi.org/10.13088/JIIS.2019.25.1.219
  29. Isinkaye, F. O., Y. O. Folajimi, and B. A. Ojokoh, "Recommendation systems: Principles, methods and evaluation", Egyptian Informatics Journal, Vol.16, No.3(2015), 261~273. https://doi.org/10.1016/j.eij.2015.06.005
  30. Janke, J., M. Castelli, and A. Popovic, "Analysis of the proficiency of fully connected neural networks in the process of classifying digital images. Benchmark of different classification algorithms on high-level image features from convolutional layers", Expert Systems with Applications, Vol.135, (2019), 12~38. https://doi.org/10.1016/j.eswa.2019.05.058
  31. Jeon, B. K., and H. Ahn, "A Collaborative Filtering System Combined with Users' Review Mining: Application to the Recommendation of Smartphone Apps", Journal of Intelligence and Information Systems, Vol.21, No.2(2015), 1~18. https://doi.org/10.13088/JIIS.2015.21.2.01
  32. Jeong, B., J. Lee, and H. Cho, "Improving memory-based collaborative filtering via similarity updating and prediction modulation", Information Sciences, Vol.180, No.5(2010), 602~612. https://doi.org/10.1016/j.ins.2009.10.016
  33. Johnson, R., and T. Zhang, "Effective use of word order for text categorization with convolutional neural networks", arXiv preprint arXiv:1412.1058, (2014).
  34. Kaushik, K., R. Mishra, N. P. Rana, and Y. K. Dwivedi, "Exploring reviews and review sequences on e-commerce platform: A study of helpful reviews on Amazon", Journal of Retailing and Consumer Services, Vol.45, (2018), 21~32. https://doi.org/10.1016/j.jretconser.2018.08.002
  35. Khan, Z. Y., and Z. Niu, "CNN with Depthwise Separable Convolutions and Combined Kernels for Rating Prediction", Expert Systems with Applications, (2020), 114528.
  36. Kim, H. K., J. K. Kim., and Y. U. Ryu, "Personalized recommendation over a customer network for ubiquitous shopping", IEEE Transactions on Services Computing, Vol.2, No.2(2009), 140~151. https://doi.org/10.1109/TSC.2009.7
  37. Kim, J. K., H. K. Kim, H. Y. Oh, and Y. U. Ryu, "A group recommendation system for online communities", International Journal of Information Management, Vol.30, No.3(2010), 212~219. https://doi.org/10.1016/j.ijinfomgt.2009.09.006
  38. Knees, P., D. Schnitzer, and A. Flexer, "Improving neighborhood-based collaborative filtering by reducing hubness", Proceedings of International Conference on Multimedia Retrieval, (2014), 161~168.
  39. Koren, Y., and R. Bell, Recommender systems handbook, Springer, New York, USA, 2015.
  40. Krishnamoorthy, S, "Linguistic features for review helpfulness prediction", Expert Systems with Applications, Vol.42, No.7(2015), 3751~3759. https://doi.org/10.1016/j.eswa.2014.12.044
  41. Lee, D., and K. Hosanagar, "How do recommender systems affect sales diversity? A crosscategory investigation via randomized field experiment", Information Systems Research, Vol.30, No.1(2019), 239~259. https://doi.org/10.1287/isre.2018.0800
  42. Lee, Y., H. Won, J. Shim, and H. Ahn, "A Hybrid Collaborative Filtering-based Product Recommender System using Search Keywords", Journal of Intelligence and Information Systems, Vol.26, No.1(2020), 151~166. https://doi.org/10.13088/jiis.2020.26.1.151
  43. Lei, X., X. Qian, and G. Zhao, "Rating prediction based on social sentiment from textual reviews", IEEE Transactions on Multimedia, Vol.18, No.9(2016), 1910~1921. https://doi.org/10.1109/TMM.2016.2575738
  44. Leung, C. W., S. C. Chan, and F. Chung, "Integrating Collaborative Filtering and Sentiment Analysis: A Rating Inference Approach", Proceedings of the ECAI Workshop on Recommender Systems, (2006), 62~68.
  45. Li, X., M. Wang, and T. P. Liang, "A multi-theoretical kernel-based approach to social network-based recommendation", Decision Support Systems, Vol.65, (2014), 95~104. https://doi.org/10.1016/j.dss.2014.05.006
  46. Linden, G., B. Smith, and J. York, "Amazon.com recommendations: Item-to-item collaborative filtering", IEEE Internet Computing, Vol.7, No.1(2003), 76~80. https://doi.org/10.1109/MIC.2003.1167344
  47. Liu, Y., X. Huang, A. An, and X. Yu, "Modeling and predicting the helpfulness of online reviews", 8th IEEE International Conference on Data Mining, (2008), 443~452.
  48. Lu, J., D. Wu, M. Mao, W. Wang, and G. Zhang, "Recommender system application developments: a survey", Decision Support Systems, Vol.74, (2015), 12~32. https://doi.org/10.1016/j.dss.2015.03.008
  49. Mandal, S., and A. Maiti, "Deep collaborative filtering with social promoter score-based user-item interaction: a new perspective in recommendation", Applied Intelligence, (2021), 1~26.
  50. Mishra, R., P. Kumar, and B. Bhasker, "A web recommendation system considering sequential information", Decision Support Systems, Vol.75, (2015), 1~10. https://doi.org/10.1016/j.dss.2015.04.004
  51. Moon, H. S., D. Sung, and J. K. Kim, "An Analytical Approach Using Topic Mining for Improving the Service Quality of Hotels", Journal of Intelligence and Information Systems, Vol.25, No.1(2019), 21~41. https://doi.org/10.13088/JIIS.2019.25.1.021
  52. Moon, H. S., J. H. Yoon, I. Y. Choi, and J. K. Kim, "An Exploratory Study of Collaborative Filtering Techniques to Analyze the Effect of Information Amount", Asia Pacific Journal of Information Systems, Vol.27, No.2(2017), 126~138. https://doi.org/10.14329/apjis.2017.27.2.126
  53. Moore, S. G., "Attitude predictability and helpfulness in online reviews: The role of explained actions and reactions", Journal of Consumer Research, Vol.42, No.1(2015), 30~44. https://doi.org/10.1093/jcr/ucv003
  54. Na, H., and K. Nam, "Application of diversity of recommender system according to user preference change", Journal of Intelligence and Information Systems, Vol.26, No.4(2020), 67~86. https://doi.org/10.13088/JIIS.2020.26.4.067
  55. Nassirtoussi, A. K., S. Aghabozorgi, T. Y. Wah, and D. C. L. Ngo, "Text mining for market prediction: A systematic review", Expert Systems with Applications, Vol.41, No.16(2014), 7653~7670. https://doi.org/10.1016/j.eswa.2014.06.009
  56. Ngo-Ye, T. L., and A. P. Sinha, "The influence of reviewer engagement characteristics on online review helpfulness: A text regression model", Decision Support Systems, Vol.61, (2014), 47~58. https://doi.org/10.1016/j.dss.2014.01.011
  57. Paradarami, T. K., N. D. Bastian, and J. L. Wightman, "A hybrid recommender system using artificial neural networks", Expert Systems with Applications, Vol.83, (2017), 300~313. https://doi.org/10.1016/j.eswa.2017.04.046
  58. Park, D. H., H. K. Kim, I. Y. Choi and J. K. Kim, "A literature review and classification of recommender systems research", Expert Systems with Applications, Vol.39, No.11(2012), 10059~10072. https://doi.org/10.1016/j.eswa.2012.02.038
  59. Polatidis, N., and C. K. Georgiadis "A multi-level collaborative filtering method that improves recommendations", Expert Systems with Applications, Vol.48, (2016), 100~110. https://doi.org/10.1016/j.eswa.2015.11.023
  60. Postmus, S., and S. Bhulai, "Recommender system techniques applied to Netflix movie data", Research Paper Business Analytics, Vrije Universiteit Amsterdam, Netherlands, 2018.
  61. Qiu, L., S. Gao, W. Cheng, and J. Guo, "Aspect-based latent factor model by integrating ratings and reviews for recommender system", Knowledge-Based Systems, Vol.110, (2016), 233~243. https://doi.org/10.1016/j.knosys.2016.07.033
  62. Ricci, F., L. Rokach and B. Shapira, Introduction to recommender systems handbook, Springer, Boston, USA, 2011.
  63. Sanchez-Moreno, D., A. B. G. Gonzalez, M. D. M. Vicente, V. F. L. Batista, and M. N. M. Garcia, "A collaborative filtering method for music recommendation using playing coefficients for artists and users", Expert Systems with Applications, Vol.66, (2016), 234~244. https://doi.org/10.1016/j.eswa.2016.09.019
  64. Sarwar, B., G. Karypis, J. Konstan, and J. Riedl, "Item-based collaborative filtering recommendation algorithms", Proceedings of the 10th International Conference on World Wide Web, (2001), 285~295.
  65. Saumya, S., and J. P. Singh, "Detection of spam reviews: a sentiment analysis approach", CSI Transactions on ICT, Vol.6, No.2(2018), 137~148. https://doi.org/10.1007/s40012-018-0193-0
  66. Siering, M., A. V. Deokar, and C. Janze, "Disentangling consumer recommendations: Explaining and predicting airline recommendations based on online reviews", Decision Support Systems, Vol.107, (2018), 52~63. https://doi.org/10.1016/j.dss.2018.01.002
  67. Song, C., X. K. Wang, P. F. Cheng, J. Q. Wang, and L. Li, "SACPC: A framework based on probabilistic linguistic terms for short text sentiment analysis", Knowledge-Based Systems, Vol.194, (2020), 105572. https://doi.org/10.1016/j.knosys.2020.105572
  68. Srifi, M., A. Oussous, A. A. Lahcen, and S. Mouline, "Recommender Systems Based on Collaborative Filtering Using Review Texts-A Survey", Information, Vol.11, No.6(2020), 317. https://doi.org/10.3390/info11060317
  69. Su, X., and T. M. Khoshgoftaar, "A survey of collaborative filtering techniques", Advances in Artificial Intelligence, (2009).
  70. Ullah, I., M. Hussain, and H. Aboalsamh, "An automated system for epilepsy detection using EEG brain signals based on deep learning approach", Expert Systems with Applications, Vol.107, (2018), 61~71. https://doi.org/10.1016/j.eswa.2018.04.021
  71. Wang, X., X. Lin, and M. K. Spencer, "Exploring the effects of extrinsic motivation on consumer behaviors in social commerce: Revealing consumers' perceptions of social commerce benefits", International Journal of Information Management, Vol.45, (2019), 163~175. https://doi.org/10.1016/j.ijinfomgt.2018.11.010
  72. Wang, X., Z. Dai., H. Li, and J. Yang, "Research on Hybrid Collaborative Filtering Recommendation Algorithm Based on the Time Effect and Sentiment Analysis", Complexity, (2021).
  73. Wei, S., N. Ye, S. Zhang, X. Huang, and J. Zhu, "Item-based collaborative filtering recommendation algorithm combining item category with interestingness measure", International Conference on Computer Science and Service System, (2012), 2038~2041.
  74. Wu, P., X. Li., S. Shen, and D. He, "Social media opinion summarization using emotion cognition and convolutional neural networks", International Journal of Information Management, Vol.51, (2020), 101978. https://doi.org/10.1016/j.ijinfomgt.2019.07.004
  75. Yoo, S., J. Song, and O. Jeong, "Social media contents based sentiment analysis and prediction system", Expert Systems with Applications, Vol.105, (2018), 102~111. https://doi.org/10.1016/j.eswa.2018.03.055
  76. Yun, Y., D, Hooshyar, J. Jo, and H. Lim, "Developing a hybrid collaborative filtering recommendation system with opinion mining on purchase review", Journal of Information Science, Vol.44, No.3(2018), 331~344. https://doi.org/10.1177/0165551517692955
  77. Zafari, F., I. Moser, and T. Sellis, "ReEx: An integrated architecture for preference model representation and explanation", Expert Systems with Applications, Vol.161, (2020), 113706. https://doi.org/10.1016/j.eswa.2020.113706
  78. Zhang, Y., and B. Wallace, "A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification", arXiv preprint arXiv:1510.03820, (2015).
  79. Zhang, Z., and B. Varadarajan, "Utility scoring of product reviews", Proceedings of the 15th ACM International Conference on Information and Knowledge Management, (2006), 51~57.
  80. Zhang, Z., D. Zhang, and J. Lai, "urCF: User Review Enhanced Collaborative Filtering", Proceedings of the 20th Americas Conference on Information Systems, (2014).
  81. Zhang, Z., H. Lin, K. Liu, D. Wu, G. Zhang, and J. Lu, "A hybrid fuzzy-based personalized recommender system for telecom products/services", Information Sciences, Vol.235, (2013), 117~129. https://doi.org/10.1016/j.ins.2013.01.025
  82. Zheng, L., V. Noroozi, and S. Yu, "Joint deep modeling of users and items using reviews for recommendation", Proceedings of the 10th ACM International Conference on Web Search and Data Mining, (2017), 425~434.
  83. Zhou, L., and P. Chaovalit, "Ontology-Supported Polarity Mining", Journal of the American Society for Information Science and Technology, Vol.59, No.1(2008), 98~110. https://doi.org/10.1002/asi.20735
  84. Zhu, T., Y. Ren, W. Zhou, J. Rong, and P. Xiong, "An effective privacy preserving algorithm for neighborhood-based collaborative filtering", Future Generation Computer Systems, Vol.36, (2014), 142~155. https://doi.org/10.1016/j.future.2013.07.019