• Title/Summary/Keyword: Crawler

Search Result 199, Processing Time 0.021 seconds

A Study on Focused Crawling of Web Document for Building of Ontology Instances (온톨로지 인스턴스 구축을 위한 주제 중심 웹문서 수집에 관한 연구)

  • Chang, Moon-Soo
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.18 no.1
    • /
    • pp.86-93
    • /
    • 2008
  • The construction of ontology defines as complicated semantic relations needs precise and expert skills. For the well defined ontology in real applications, plenty of information of instances for ontology classes is very critical. In this study, crawling algorithm which extracts the fittest topic from the Web overflowing over by a great number of documents has been focused and developed. Proposed crawling algorithm made a progress to gather documents at high speed by extracting topic-specific Link using URL patterns. And topic fitness of Link block text has been represented by fuzzy sets which will improve a precision of the focused crawler.

Differences across countries in the impact of developers' collaboration characteristics on performance : Focused on weak tie theory (국가별 오픈소스 소프트웨어 개발자의 네트워크 특성이 개방형 협업 성과에 미치는 영향 : 약한 연결 이론을 중심으로)

  • Lee, Saerom;Baek, Hyunmi;Lee, Uijun
    • The Journal of Information Systems
    • /
    • v.29 no.2
    • /
    • pp.149-171
    • /
    • 2020
  • Purpose With the advent of the 4th Industrial Revolution, related technologies such as IoT, big data, and artificial intelligence technologies are developing through not only specific companies but also a number of unspecified developers called open collaboration. For this reason, it is important to understand the nature of the collaboration that leads to successful open collaboration. Design/methodology/approach We focused the relationship between the collaboration characteristics and collaboration performance of developers who participating in open source software development, which is a representative open collaboration. Specifically, we create the country-specific network and draw the individual developers characteristics from the network such as collaboration scope and collaboration intensity. We compare and analyze the characteristics of developers across countries and explore whether there are differences between indicators. We develop a Web crawler for GitHub, a representative OSSD development site, and collected data of developers who located at China, Japan, Korea, the United States, and Canada. Findings China showed the characteristics of cooperation suitable for the form of weak tie theory, and consistent results were not drawn from other countries. This study confirmed the necessity of exploratory research on collaboration characteristics by country considering that there are differences in open collaboration characteristics or software development environments by country.

Web Crawling and PageRank Calculation for Community-Limited Search (커뮤니티 제한 검색을 위한 웹 크롤링 및 PageRank 계산)

  • Kim Gye-Jeong;Kim Min-Soo;Kim Yi-Reun;Whang Kyu-Young
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2005.07b
    • /
    • pp.1-3
    • /
    • 2005
  • 최근 웹 검색 분야에서는 검색 질을 높이기 위한 기법들이 많이 연구되어 왔으며, 대표적인 연구로는 제한 검색, focused crawling, 웹 클러스터링 등이 있다. 그러나 제한 검색은 검색 범위를 의미적으로 관련된 사이트들로 제한할 수 없으며, focused crawling은 질의 시점에 클러스터링하기 때문에 질의 처리 시간이 오래 걸리고, 웹 클러스터링은 많은 웹 페이지들을 대상으로 클러스터링하기 위한 오버헤드가 크다. 본 논문에서는 검색 범위를 특정 커뮤니티로 제한하여 검색 하는 커뮤니티 제한 검색과 커뮤니티를 구하는 방법으로 cluster crawler를 제안하여 이러한 문제점을 해결한다. 또한, 커뮤니티를 이용하여 PageRank를 2단계로 계산하는 방법을 제안한다. 제안된 방법은 첫 번째 과정에서 커뮤니티 단위로 지역적으로 PageRank를 계산한 후, 두 번째 과정에서 이를 바탕으로 전역적으로 PageRank론 계산한다. 제안된 방법은 Wang에 의해 제안된 방법에 비해 PageRank 근사치의 오차를 $59\%$ 정도로 줄일 수 있다.

  • PDF

Implementation of a Web Robot and Statistics on the Korean Web (웹 로봇 구현 및 한국 웹 통계보고)

  • Kim, Sung-Jin;Lee, Sang-Ho
    • The KIPS Transactions:PartC
    • /
    • v.10C no.4
    • /
    • pp.509-518
    • /
    • 2003
  • A web robot is a program that downloads and stores web pages. Implementation issues for developing web robots have been studied widely and various web statistics are reported in the literature. First, this paper describes the overall architecture of our robot and implementation decisions on several important issues. Second, we show empirical statistics on approximately 74 million Korean web pages. Third, we monitored 1,424 Korean web sites to observe the changes of web pages. We identify what factors of web pages could affect the changes. The factors may be used for the selection of web pages to be updated incrementally.

The Effects of Social Media on Music-induced Tourism: A Case of Korean Pop Music and Inbound Tourism to Korea

  • Oh, Sehwan;Ahn, JoongHo;Baek, Hyunmi
    • Asia pacific journal of information systems
    • /
    • v.25 no.1
    • /
    • pp.119-141
    • /
    • 2015
  • With the rapid spread of social media, video-sharing social media like YouTube has emerged as a consumption and distribution channel for entertainment goods such as music videos and movie trailers. In tourism research, there has been a lot of research of how the visual media such as movies and soap operas induced tourism. However, no studies have attempted to examine the role of social media as a music consumption channel and its impact on tourism. Expanding a body of media-induced tourism, we analysed the impact of video-sharing social media on music-induced tourism with a case of Korean pop music and inbound tourism to Korea. Developing a Web-crawler, we collected YouTube users' comments data on 166 Korean pop music video clips which were released from 2009 to 2012 with over 1 million view counts. Controlling many of the determinants of tourism and analysing country-by-country impact of YouTube comments with the panel data, we found that engagement of Korean pop music video clips on YouTube is a significant predictor for the flow of inbound tourists to Korea.

Semantic Network Analysis about Comments on Internet Articles about Nurse Workplace Bullying (간호사 괴롭힘 관련 인터넷 포털 기사에 대한 댓글의 의미연결망 분석)

  • Kim, Chang Hee;Moon, Seong Mi
    • Journal of Korean Clinical Nursing Research
    • /
    • v.25 no.3
    • /
    • pp.209-220
    • /
    • 2019
  • Purpose: A significant amount of public opinion about nurse bullying is expressed on the internet. The purpose of this study was to analyze the linkage structures among words extracted from comments on internet articles related to nurse workplace bullying using semantic network analysis. Methods: From February 2018 to April 2019, comments made on news articles posted to the Daum and Naver web portal containing keywords such as "nurse", "Taeum", and "bullying" were collected using a web crawler written in Python. A morphological analysis performed with Open Korean Text in KoNLPy generated 54 major nodes. The frequencies, eigenvector centralities, and betweenness centralities of the 54 nodes were calculated and semantic networks were visualized using the UCINET and NetDraw programs. Convergence of iterated correlations (CONCOR) analysis was performed to identify structural equivalence. Results: This paper presents results about March 2018 and January 2019 because these months had highest number of articles. Of the 54 major nodes, "nurse", "hospital", "patient", and "physician" were the most frequent and had the highest eigenvector and betweenness centralities. The CONCOR analysis identified work environment, nurse, gender, and military clusters. Conclusion: This study structurally explored public opinion about nurse bullying through semantic network analysis. It is suggested that various studies on nursing phenomena will be conducted using social network analysis.

Flood monitoring and prediction using online unstructured data (비정형데이터를 활용한 홍수 모니터링 및 예측)

  • Lee, Jeong Ha;Hwang, Seok Hwan
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2019.05a
    • /
    • pp.118-118
    • /
    • 2019
  • 현재 홍수예보는 정형데이터인 유량 및 수위 등을 활용하여 이뤄지고 있다. 하지만 실제 사람들이 체감하는 홍수에 대한 위험도는 홍수예보 발령과는 달라 홍수예보가 이뤄지지 않은 지역에서 인명사고가 발생하기도 한다. 이는 수위 측정이 이뤄지지 않는 소규모 하천이나 사람들의 유동성이 큰 도심지역에서 빈번하게 발생한다. 이를 보완하기 위해서는 사람들의 체감 정도 및 인구의 유동성을 고려한 비정형데이터를 활용해야 한다. 특히 소셜 네트워크 서비스(Social Network Commuinty, SNS)를 사용하는 사람들이 많아지면서 기존에 사용되어 왔던 정형데이터 센서 이외의 데이터를 제공한다. 또한 개개인이 작성하는 글은 실시간으로 활용이 가능하여 인구의 유동성 및 시 공간적 데이터를 얻기에 유용하여 활용성이 매우 높은 비정형데이터이다. 따라서 본 연구에서는 SNS 데이터를 추출하고 이를 분석하여 2018년에 발생했던 강우사상과의 패턴을 비교하여 홍수예보에서의 활용성을 분석하였다. 홍수와 관련한 키워드를 중심으로 시 공간적 정보 및 추출이 가능한 웹 크롤러(Web Crawler) 프로그램을 작성하였으며 이를 토대로 데이터를 수집하였다. 수집한 데이터와 실제 홍수사상을 비교 분석을 한 결과 강우량 및 수위와 해당 지역에 대한 데이터의 양이 유사한 패턴을 보인 것으로 확인되었다. 실시간으로 데이터를 수집하고 이를 분석하여 리드타임을 충분히 확보한다면 홍수예측에 활용 가능할 것이라 생각된다. 본 연구는 한국건설기술연구원 19주요-대4-시드사업인 '커뮤니티 빅데이터 패턴 해석을 통한 수난(水難) 발생 및 규모 예측 기술 개발(20190126-001) '로 수행되었습니다.

  • PDF

Registry Metadata Quality Assessment by the Example of re3data.org Schema

  • Kim, Suntae;Choi, Myung-Seok
    • International Journal of Knowledge Content Development & Technology
    • /
    • v.7 no.2
    • /
    • pp.41-51
    • /
    • 2017
  • Nowadays, research data repositories (RDR) have become progressively widespread all over the world. To expand repository services and build up inbound linking strategy, organizations list their repositories with so called Global Registries. Accordingly, such registries should be carefully described by the related data. In this study, I explore the metadata schema of re3data.org. I collect and analyze descriptions from the listed repositories, and come up with some suggestions concerning possible improvements to the metadata schema. To accomplish this, I develop a crawler program, which collects necessary data from the re3data.org. Based on the analysis results, I have identified two issues that required elements is missing, one issue that required element value is missing when the corresponding property is applied, five inconsistency issues with re3data controlled vocabulary, six issues with undescribed optional elements, and two inconsistency issues between the elements and their attributes which do not pair with. I believe this discussion can facilitate improvements to the existing re3data.org schema and further help researchers who analyze data repository trends.

Design for Recommended System of Movies using Social Network Keyword of Analysis (소셜 네트워크 키워드 분석을 통한 영화 추천 시스템 설계)

  • Yang, Xi-tong;Lee, Jong-Won;Chu, Xun;Pyoun, Do-Kil;Jung, Hoe-Kyung
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2014.10a
    • /
    • pp.609-611
    • /
    • 2014
  • Was developed of the web service in Due to the dissemination for IT skills development and smart appliances. In particular, Social network service for should be able to communicate feel free to a user across without distinguishing between production and consumption information in contrast to the existing web service. And strengthen to the information sharing relationships between existing human relation and new human relation. In this paper, a social network service in providing a social networking from users using their communication and information sharing is used to collect and analyze in the keyword. And a design of recommended system of movies for appropriate keyword.

  • PDF

A Study on Space Consumption Behavior of Contemporary Consumers -Focusing on Analysis of Social Media Big Data- (현대 소비자의 공간소비행동에 관한 연구 -소셜미디어 데이터 분석을 중심으로-)

  • Ahn, Suh Young;Koh, Ae-Ran
    • Journal of the Korean Society of Clothing and Textiles
    • /
    • v.44 no.5
    • /
    • pp.1019-1035
    • /
    • 2020
  • This study examines the millennial generation, who express themselves and share information on social media after experiencing constantly changing 'hot places' (places of interest) in contemporary cities, with the goal of analyzing space consumption behaviors. Data were collected via an Instagram crawler application developed with Python 3.4 administered to 19,262 posts using the term 'hot places' from November 1 and December 15, 2019. Issues were derived from a text mining technique using Textom 2.0; in addition, semantic network analysis using Ucinet6 and the NetDraw program were also conducted. The results are as follows. First, a frequency analysis of keywords for hot places indicated words frequently found in nouns were related to food, local names, SNS and timing. Words related to positive emotions felt in experience, and words related to behavior in hot places appeared in predicate. Based on importance, communication is the most important keyword and influenced all issues. Second, the results of visualization of semantic network analysis revealed four categories in the scope of the definition of "hot place": (1) culinary exploration, (2) atmosphere of cafés, (3) happy daily life of 'me' expressed in images, (4) emotional photos.