DOI QR코드

DOI QR Code

A Study on Market Size Estimation Method by Product Group Using Word2Vec Algorithm

Word2Vec을 활용한 제품군별 시장규모 추정 방법에 관한 연구

  • Jung, Ye Lim (Div. of Information Analysis, Korea Institute of Science and Technology Information) ;
  • Kim, Ji Hui (Div. of Information Analysis, Korea Institute of Science and Technology Information) ;
  • Yoo, Hyoung Sun (Div. of Information Analysis, Korea Institute of Science and Technology Information)
  • 정예림 (한국과학기술정보연구원 데이터분석본부) ;
  • 김지희 (한국과학기술정보연구원 데이터분석본부) ;
  • 유형선 (한국과학기술정보연구원 데이터분석본부 과학기술연합대학원대학교 과학기술경영정책학과)
  • Received : 2019.12.09
  • Accepted : 2020.02.28
  • Published : 2020.03.31

Abstract

With the rapid development of artificial intelligence technology, various techniques have been developed to extract meaningful information from unstructured text data which constitutes a large portion of big data. Over the past decades, text mining technologies have been utilized in various industries for practical applications. In the field of business intelligence, it has been employed to discover new market and/or technology opportunities and support rational decision making of business participants. The market information such as market size, market growth rate, and market share is essential for setting companies' business strategies. There has been a continuous demand in various fields for specific product level-market information. However, the information has been generally provided at industry level or broad categories based on classification standards, making it difficult to obtain specific and proper information. In this regard, we propose a new methodology that can estimate the market sizes of product groups at more detailed levels than that of previously offered. We applied Word2Vec algorithm, a neural network based semantic word embedding model, to enable automatic market size estimation from individual companies' product information in a bottom-up manner. The overall process is as follows: First, the data related to product information is collected, refined, and restructured into suitable form for applying Word2Vec model. Next, the preprocessed data is embedded into vector space by Word2Vec and then the product groups are derived by extracting similar products names based on cosine similarity calculation. Finally, the sales data on the extracted products is summated to estimate the market size of the product groups. As an experimental data, text data of product names from Statistics Korea's microdata (345,103 cases) were mapped in multidimensional vector space by Word2Vec training. We performed parameters optimization for training and then applied vector dimension of 300 and window size of 15 as optimized parameters for further experiments. We employed index words of Korean Standard Industry Classification (KSIC) as a product name dataset to more efficiently cluster product groups. The product names which are similar to KSIC indexes were extracted based on cosine similarity. The market size of extracted products as one product category was calculated from individual companies' sales data. The market sizes of 11,654 specific product lines were automatically estimated by the proposed model. For the performance verification, the results were compared with actual market size of some items. The Pearson's correlation coefficient was 0.513. Our approach has several advantages differing from the previous studies. First, text mining and machine learning techniques were applied for the first time on market size estimation, overcoming the limitations of traditional sampling based- or multiple assumption required-methods. In addition, the level of market category can be easily and efficiently adjusted according to the purpose of information use by changing cosine similarity threshold. Furthermore, it has a high potential of practical applications since it can resolve unmet needs for detailed market size information in public and private sectors. Specifically, it can be utilized in technology evaluation and technology commercialization support program conducted by governmental institutions, as well as business strategies consulting and market analysis report publishing by private firms. The limitation of our study is that the presented model needs to be improved in terms of accuracy and reliability. The semantic-based word embedding module can be advanced by giving a proper order in the preprocessed dataset or by combining another algorithm such as Jaccard similarity with Word2Vec. Also, the methods of product group clustering can be changed to other types of unsupervised machine learning algorithm. Our group is currently working on subsequent studies and we expect that it can further improve the performance of the conceptually proposed basic model in this study.

인공지능 기술의 급속한 발전과 함께 빅데이터의 상당 부분을 차지하는 비정형 텍스트 데이터로부터 의미있는 정보를 추출하기 위한 다양한 연구들이 활발히 진행되고 있다. 비즈니스 인텔리전스 분야에서도 새로운 시장기회를 발굴하거나 기술사업화 주체의 합리적 의사결정을 돕기 위한 많은 연구들이 이뤄져 왔다. 본 연구에서는 기업의 성공적인 사업 추진을 위해 핵심적인 정보 중의 하나인 시장규모 정보를 도출함에 있어 기존에 제공되던 범위보다 세부적인 수준의 제품군별 시장규모 추정이 가능하고 자동화된 방법론을 제안하고자 한다. 이를 위해 신경망 기반의 시멘틱 단어 임베딩 모델인 Word2Vec 알고리즘을 적용하여 개별 기업의 생산제품에 대한 텍스트 데이터를 벡터 공간으로 임베딩하고, 제품명 간 코사인 거리(유사도)를 계산함으로써 특정한 제품명과 유사한 제품들을 추출한 뒤, 이들의 매출액 정보를 연산하여 자동으로 해당 제품군의 시장규모를 산출하는 알고리즘을 구현하였다. 실험 데이터로서 통계청의 경제총조사 마이크로데이터(약 34만 5천 건)를 이용하여 제품명 텍스트 데이터를 벡터화 하고, 한국표준산업분류 해설서의 산업분류 색인어를 기준으로 활용하여 코사인 거리 기반으로 유사한 제품명을 추출하였다. 이후 개별 기업의 제품 데이터에 연결된 매출액 정보를 기초로 추출된 제품들의 매출액을 합산함으로써 11,654개의 상세한 제품군별 시장규모를 추정하였다. 성능 검증을 위해 실제 집계된 통계청의 품목별 시장규모 수치와 비교한 결과 피어슨 상관계수가 0.513 수준으로 나타났다. 본 연구에서 제시한 모형은 의미 기반 임베딩 모델의 정확성 향상 및 제품군 추출 방식의 개선이 필요하나, 표본조사 또는 다수의 가정을 기반으로 하는 전통적인 시장규모 추정 방법의 한계를 뛰어넘어 텍스트 마이닝 및 기계학습 기법을 최초로 적용하여 시장규모 추정 방식을 지능화하였다는 점, 시장규모 산출범위를 사용 목적에 따라 쉽고 빠르게 조절할 수 있다는 점, 이를 통해 다양한 분야에서 수요가 높은 세부적인 제품군별 시장정보 도출이 가능하여 실무적인 활용성이 높다는 점에서 의의가 있다.

Keywords

References

  1. An, J., S.-H. Lee, E.-H. An and H.-W. Kim, "Fintech Trends and Mobile Payment Service Anlaysis in Korea: Application of Text Mining Techniques", Informatization Policy, Vol.23, No.3(2016), 26-42. https://doi.org/10.22693/NIAIP.2016.23.3.026
  2. Balachandra, R. and J. H. Friar, "Factors for Success in R&D Projects and New Product Innovation: A Contextual Framework", IEEE Transactions on Engineering Management, Vol.44, No.3(1997), 276-287. https://doi.org/10.1109/17.618169
  3. Chakraborty, G. and M. Krishna, "Analysis of Unstructured Data: Applications of Text Analytics and Sentiment Mining", SAS global forum, (2014), 1288-2014.
  4. Choi, B.-O. and B. S. Kim, "A Forecasting on the Market Size of Korean Solar Salt", Journal of Korea Academia-Industrial cooperation Society, Vol.14, No.10(2013), 4812-4818. https://doi.org/10.5762/KAIS.2013.14.10.4812
  5. Choi, S., J. Seol and S.-g. Lee, "On Word Embedding Models and Parameters Optimized for Korean", Proceedings of the 28th Annual Conference on Human and Cognitive Language Technology, (2016), 252-256.
  6. Chun, S.-Y. and S.-G. Kim, "Estimation for Market Size of the Purchasing Costs for the Textbooks and Reference Books", The Journal of Economics and Finance of Education, Vol.19, (2010), 95-124.
  7. Grbovic, M., V. Radosavljevic, N. Djuric, N. Bhamidipati, J. Savla, V. Bhagwan and D. Sharp, "E-Commerce in Your Inbox: Product recommendations at scale", Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (2015), 1809-1818.
  8. Harris, Z. S., "Distributional Structure", Word, Vol.10, No.2-3(1954), 146-162. https://doi.org/10.1080/00437956.1954.11659520
  9. Heo, C. and S.-Y. Ohn, "A Novel Method for Constructing Sentiment Dictionaries Using Word2vec and Label Propagation", Journal of Korean Institute of Next Generation Computing, Vol.13, No.2(2017), 93-101.
  10. Heu, J.-U., "Korean Language Clustering Using Word2vec", The Journal of The Institute of Internet, Broadcasting and Communication, Vol.18, No.5(2018), 25-30. https://doi.org/10.7236/JIIBC.2018.18.5.25
  11. Jang, E.-S., Y.-H. Baek and S.-W. Lee, "The Survey on the State and Scale of Constitutional Medical Service Market in Korea", Journal of Sasang Constitutional Medicine, Vol.25, No.1(2013), 43-50. https://doi.org/10.7730/JSCM.2013.25.1.43
  12. Jun, S.-P., T.-E. Sung and S. Choi, "A Data-Based Sales Forecasting Support System for New Businesses", Journal of Intelligence and Information Systems, Vol.23, No.1(2017), 1-22. https://doi.org/10.13088/jiis.2017.23.1.001
  13. Jung, Y. L., H. S. Yoo, J. H. Kim, H. G. Kim and E. S. Kim, "Estimating Apparatus for Market Size, and Control Method Thereof", Patent Application Number 10-2019-0112446, Korean Intellectual Property Office, Republic of Korea, 2019.
  14. Kang, B.-S., "A Study on the Accuracy Improvement of Movie Recommender System Using Word2vec and Ensemble Convolutional Neural Networks", Journal of digital convergence, Vol.17, No.1(2019), 123-130. https://doi.org/10.14400/JDC.2019.17.1.123
  15. Kang, H. and J. Yang, "Optimization of Word2vec Models for Korean Word Embeddings", Journal of Digital Contents Society, Vol.20, No.4(2019), 825-833. https://doi.org/10.9728/dcs.2019.20.4.825
  16. Kang, H. and J. Yang, "The Analogy Test Set Suitable to Evaluate Word Embedding Models for Korean", Journal of Digital Contents Society, Vol.19, No.10(2018), 1999-2008. https://doi.org/10.9728/dcs.2018.19.10.1999
  17. Kang, J. and J. Cho, "The Demographic Structure, Firm Age and Economic Performance: A Local Level Analysis", Economic Analysis, Vol.24, No.4(2018), 101-128.
  18. Kim, D. and M.-W. Koo, "Categorization of Korean News Articles Based on Convolutional Neural Network Using Doc2vec and Word2vec", Journal of KIISE, Vol.44, No.7 (2017), 742-747. https://doi.org/10.5626/JOK.2017.44.7.742
  19. Kim, D. J., D. I. Park and J. S. Park, "Study on the Change of Marketing Strategy through Data Mining Technique", Korea Business Review, Vol.22, No.2(2018), 177-194. https://doi.org/10.17287/kbr.2018.22.2.177
  20. Kim, K. and C. Park, "Automatic Ipc Classification of Patent Documents Using Word2vec and Two Layers Bidirectional Long Short Term Memory Network", The Journal of Korean Institute of Next Generation Computing, Vol.15, No.2(2019), 50-60.
  21. Kim, S. and S. Lee, "Automatic Extraction of Alternative Word Candidates Using the Word2vec Model", Journal of Korean Institute of Information Scientists and Engineers, Vol.23, No.12(2015), 769-771.
  22. Le, Q. and T. Mikolov, "Distributed Representations of Sentences and Documents", International conference on machine learning, (2014), 1188-1196.
  23. Lee, D.-Y., J.-C. Jo and H.-S. Lim, "User Sentiment Analysis on Amazon Fashion Product Review Using Word Embedding", Journal of the Korea Convergence Society, Vol.8, No.4(2017), 1-8. https://doi.org/10.15207/JKCS.2017.8.4.001
  24. Lee, K., K.-Y. Kim and Z. Lee, "Factors Affecting Purchase Intention of Smart Mobility : Integrating Text Mining and Mental Accounting Theory", Korea Journal of Business Administration, Vol.31, No.11(2018), 2147-2168.
  25. Lee, M. and H.-J. Kim, "Construction of Event Networks from Large News Data Using Text Mining Techniques", Journal of Intelligence and Information Systems, Vol.24, No.1(2018), 183-203. https://doi.org/10.13088/jiis.2018.24.1.183
  26. Lee, Y.-J., J.-H. Seo and J.-T. Choi, "Fashion Trend Marketing Prediction Analysis Based on Opinion Mining Applying Sns Text Contents", Journal of Korean Institute of Information Technology, Vol.12, No.12(2014), 163-170.
  27. Lee, Y., Y.-J. Lee and H. Kang, "A Study on Estimating the Size of the Fashion Market through Sample Survey", Journal of The Korean Data Analysis Society, Vol.14, No.3(2012), 1281-1290.
  28. Lilleberg, J., Y. Zhu and Y. Zhang, "Support Vector Machines and Word2vec for Text Classification with Semantic Features", 2015 IEEE 14th International Conference on Cognitive Informatics & Cognitive Computing, (2015), 136-140.
  29. Lim, J. and H. Oh, "A Study on the New Product Forecasting Methodology", Journal of the Korean Institute of Industrial Engineers, Vol.18, No.2(1992), 51-63.
  30. Liu, H., "Sentiment Analysis of Citations Using Word2vec", arXiv preprint arXiv:1704.00177, (2017).
  31. Maaten, L. v. d. and G. Hinton, "Visualizing Data Using t-SNE", Journal of machine learning research, Vol.9, No.Nov(2008), 2579-2605.
  32. Mikolov, T., K. Chen, G. Corrado and J. Dean, "Efficient Estimation of Word Representations in Vector Space", arXiv preprint arXiv:1301.3781, (2013).
  33. Nam, Y., "Analysis on the Determinants of Exit of Self-Employed Businesses in Korea", BOK working paper, Vol.5, (2017), 1-37.
  34. Ngo, D. L., N. Yamamoto, V. A. Tran, N. G. Nguyen, D. Phan, F. R. Lumbanraja, M. Kubo and K. Satou, "Application of Word Embedding to Drug Repositioning", Journal of Biomedical Science and Engineering, Vol.9, No.01(2016), 7-16. https://doi.org/10.4236/jbise.2016.91002
  35. Park, H. and J. Ahn, "Demand Forecasting for G2B E-Commerce Using Public Data : A Case Study of Public Procurement Service", The Journal of Korean Institute of Information Technology, Vol.12, No.10(2014), 113-121.
  36. Park, S. S. and K. C. Lee, "Effective Korean Sentiment Classification Method Using Word2vec and Ensemble Classifier", Journal of Digital Contents Society, Vol.19, No.1 (2018), 133-140. https://doi.org/10.9728/dcs.2018.19.1.133
  37. Park, Y.-J., Y.-B. Kim, S.-Y. Jeong, Y. J. Kim and S.-W. Son, "Network Analysis in Korean Presidential Speeches by Using Word2vec", New Physics: Sae Mulli, Vol.67, No.5(2017), 569-574. https://doi.org/10.3938/NPSM.67.569
  38. Shin, M. C., Basic Statistics for Business and Economics, Changmin, Seoul, 2010.
  39. Son, N. S., Y. Lee and H. Chun, "Growth and Failure of Manufacturing Plants in Korea: Single-Unit Versus Multi-Unit Plants", Journal of Market Economy, Vol.47, No.1(2018), 1-27. https://doi.org/10.38162/jome.47.1.1
  40. Statistics Korea, "Report on the Census on Establishments", 2015.
  41. Statistics Korea, "Report on the Economic Census - Whole Country", 2017.
  42. Stein, R. A., P. A. Jaques and J. F. Valiati, "An Analysis of Hierarchical Text Classification Using Word Embeddings", Information Sciences, Vol.471, (2019), 216-232. https://doi.org/10.1016/j.ins.2018.09.001
  43. Vasile, F., E. Smirnova and A. Conneau, "Meta-Prod2vec - Product Embeddings Using Side-Information for Recommendation", Proceedings of the 10th ACM Conference on Recommender Systems, (2016), 225-232.
  44. Weiss, S. M., N. Indurkhya and T. Zhang, Fundamentals of Predictive Text Mining, Springer, 2015.
  45. Xue, B., C. Fu and Z. Shaobin, "A Study on Sentiment Computing and Classification of Sina Weibo with Word2vec", 2014 IEEE International Congress on Big Data, (2014), 358-363.
  46. Yang, H., Y.-I. Lee, H.-j. Lee, S. W. Cho and M.-W. Koo, "A Study on Word Vector Models for Representing Korean Semantic Information", Phonetics and Speech Sciences, Vol.7, No.4(2015), 41-47. https://doi.org/10.13064/KSSS.2015.7.4.041
  47. Yang, Y.-J., B.-H. Lee, J.-S. Kim and K. Y. Lee, "Development of an Automatic Classification System for Game Reviews Based on Word Embedding and Vector Similarity", The Journal of Society for e-Business Studies, Vol.24, No.2(2019), 1-14.
  48. Yoo, H. S., J. H. Seo, S.-P. Jun and J. Seo, "A Study on an Estimation Method of Domestic Market Size by Using the Standard Statistical Classifications", Journal of Korea Technology Innovation Society, Vol.18, No.3(2015), 387-415.
  49. Yoon, Y. S., J.-C. Park, and S. S. Cho, "A Study on the Market Size Distribution of Artificial Information Industry", Journal of Industrial Economics and Business, Vol.29, No.6(2016), 2179-2198. https://doi.org/10.22558/jieb.2016.12.29.6.2179