Journal of the Korean Institute of Intelligent Systems (한국지능시스템학회논문지)

Korean Institute of Intelligent Systems (한국지능시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Design and Implementation of Web Crawler Wrappers to Collect User Reviews on Shopping Mall with Various Hierarchical Tree Structure

다양한 계층 트리 구조를 갖는 쇼핑몰 상에서의 상품평 수집을 위한 웹 크롤러 래퍼의 설계 및 구현

  • 강한훈 (세종대학교 컴퓨터공학과) ;
  • 유성준 (세종대학교 컴퓨터공학과) ;
  • 한동일 (세종대학교 컴퓨터공학과)
  • Received : 2009.12.02
  • Accepted : 2010.05.10
  • Published : 2010.06.25
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, the wrapper database description language and model is suggested to collect product reviews from Korean shopping malls with multi-layer structures and are built in a variety of web languages. Above all, the wrapper based web crawlers have the website structure information to bring the exact desired data. The previously suggested wrapper based web crawler can collect HTML documents and the hierarchical structure of the target documents were only 2-3 layers. However, the Korean shopping malls in the study consist of not only HTML documents but also of various web language (JavaScript, Flash, and AJAX), and have a 5-layer hierarchical structure. A web crawler should have information about the review pages in order to visit the pages without visiting any non-review pages. The proposed wrapper contains the location information of review pages. We also propose a language grammar used in describing the location information.

본 논문에서는 다 계층 구조와 다양한 웹 언어로 구성된 한국내 쇼핑몰로부터 상품평 수집을 위한 래퍼 데이터베이스 기술 언어 및 모델을 제안한다. 기존에 제안된 래퍼 기반 웹 크롤러는 HTML 문서를 수집할 수 있고, 수집 대상으로 하는 문서의 계층 구조는 2~3계층이다. 그러나 한국형 쇼핑몰 사이트는 HTML 문서뿐만 아니라 다양한 웹 언어(JavaScript, Flash, AJAX)로 구성되어 있고, 그 계층 또한 5계층으로 이루어졌다. 웹크롤러가 이 5 계층 사이트에 있는 상품평만을 수집하려고 하면 상품평이 있는 위치를 정확히 알고 있으면 된다. 우리가 제안하는 래퍼에는 이러한 정보를 포함하고 있도록 하였고, 이러한 정보를 기술하기 위해 필요한 래퍼 데이터 기술 문법도 제안한다.

Keywords

References

  1. P.Tuerny, “Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews,” In Proc. of the Meeting of the Association for Computational Linguistics(ACL’02), pp.417-424, 2002
  2. Bo Pang, Lillian Lee and Shivakumar Vaithyanathan, “Thumbs up? Sentiment Classification using Machine Learning Techniques,” In Proc. of the Conference on Empirical Methods in Natural Language Processing(EMNLP'02), pp.79-86, 2002
  3. M. Hu and B. Liu, “Mining and Summarizing Customer Reviews,” In Proc. of ACM SIGKDD ’04, pp.168-177,2004
  4. M. Hu and B. Liu, “Mining Opinion Features in Customer Reviews,” In Proc. of the 19th National Conference on Artificial Intelligence(AAAI’04), pp. 755-760, 2004
  5. Bing Liu, Web Data Mining : Exploring Hyperlinks, Contents, and Usage Data, Springer, pp. 273-289, 2007
  6. S. Chakrabarti,M. van den Berg, and B. Dom, “Focused Crawling : A New Approach to Topic-Specific Web Resource Discovery,” Computer Networks, Vol.31, No. 11-16, pp.1623-1640, 1999 https://doi.org/10.1016/S1389-1286(99)00052-3
  7. Ziyu Guan, Can Wang, Chun Chen, Jiajun Bu, Junfeng Wang, “Guide Focused Crawler Efficiently and Effectively Using On-line Topical Importance Estimation," In Proc. of ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 757-758, 2008
  8. S. Chakrabarti, Mining the Web. Discovering Knowledge from Hypertext Data, Morgan Kaufmamm, pp. 257-287, 2003
  9. J. Cho, H. Garcia-Molina, and L. Page, “Efficient Crawling through URL Ordering,” Computer Networks, Vol.30, No.1-7, pp. 161-172, 1998
  10. Chia-Hui Chang, Mohammed Kayed, Moheb Ramzy Girgis, and Khaled Shaalan, “A Survey of Web Information Extraction Systems,” IEEE Transaction on Knowledge and Data Engineering, Vol.18, No. 10, pp.1411-1428, 2006 https://doi.org/10.1109/TKDE.2006.152
  11. Jaeyoung Yang, Tae-Hyung Kim, and Joongmin Choi, “An Interface Agent for Wrapper-Based Information Extraction,” In Proc. of the International Confenrence on Principles of Practice in Multi-Agent Systems(PRIMA'04), pp.291-302, 2004
  12. Claudio Bertoli, Valter Crescenzi, and Paolo Merialdo, “Crawling programs for wrapper-based applications,” In Proc. of IEEE International Conference on Information Reuse and Integration(IRI'08), pp.160-165, 2008
  13. Stephen Soderland, Claire Cardie, and Raymond Mooney, "Learning information extraction rules for semi-structured and Free Text," Machine Learning, Vol. 34, No.1-3, pp.233-272, 1999 https://doi.org/10.1023/A:1007562322031
  14. Hanhoon Kang, Seong Joon Yoo, Dongil Han, “Modeling Web Crawler Wrappers to Collect User Reviews on Shopping Mall with Various Hierarchical Tree Struture,” In Proc. of the International Conference on Web Information Systems and Mining(WISM ’09), pp. 69-73, 2009
  15. http://autos.yahoo.com/new_cars.html