• 제목/요약/키워드: web data mining

Search Result 412, Processing Time 0.023 seconds

Frequently Occurred Information Extraction from a Collection of Labeled Trees (라벨 트리 데이터의 빈번하게 발생하는 정보 추출)

  • Paik, Ju-Ryon;Nam, Jung-Hyun;Ahn, Sung-Joon;Kim, Ung-Mo
    • Journal of Internet Computing and Services
    • /
    • v.10 no.5
    • /
    • pp.65-78
    • /
    • 2009
  • The most commonly adopted approach to find valuable information from tree data is to extract frequently occurring subtree patterns from them. Because mining frequent tree patterns has a wide range of applications such as xml mining, web usage mining, bioinformatics, and network multicast routing, many algorithms have been recently proposed to find the patterns. However, existing tree mining algorithms suffer from several serious pitfalls in finding frequent tree patterns from massive tree datasets. Some of the major problems are due to (1) modeling data as hierarchical tree structure, (2) the computationally high cost of the candidate maintenance, (3) the repetitious input dataset scans, and (4) the high memory dependency. These problems stem from that most of these algorithms are based on the well-known apriori algorithm and have used anti-monotone property for candidate generation and frequency counting in their algorithms. To solve the problems, we base a pattern-growth approach rather than the apriori approach, and choose to extract maximal frequent subtree patterns instead of frequent subtree patterns. The proposed method not only gets rid of the process for infrequent subtrees pruning, but also totally eliminates the problem of generating candidate subtrees. Hence, it significantly improves the whole mining process.

  • PDF

Investigating Opinion Mining Performance by Combining Feature Selection Methods with Word Embedding and BOW (Bag-of-Words) (속성선택방법과 워드임베딩 및 BOW (Bag-of-Words)를 결합한 오피니언 마이닝 성과에 관한 연구)

  • Eo, Kyun Sun;Lee, Kun Chang
    • Journal of Digital Convergence
    • /
    • v.17 no.2
    • /
    • pp.163-170
    • /
    • 2019
  • Over the past decade, the development of the Web explosively increased the data. Feature selection step is an important step in extracting valuable data from a large amount of data. This study proposes a novel opinion mining model based on combining feature selection (FS) methods with Word embedding to vector (Word2vec) and BOW (Bag-of-words). FS methods adopted for this study are CFS (Correlation based FS) and IG (Information Gain). To select an optimal FS method, a number of classifiers ranging from LR (logistic regression), NN (neural network), NBN (naive Bayesian network) to RF (random forest), RS (random subspace), ST (stacking). Empirical results with electronics and kitchen datasets showed that LR and ST classifiers combined with IG applied to BOW features yield best performance in opinion mining. Results with laptop and restaurant datasets revealed that the RF classifier using IG applied to Word2vec features represents best performance in opinion mining.

Facilitating Web Service Taxonomy Generation : An Artificial Neural Network based Framework, A Prototype Systems, and Evaluation (인공신경망 기반 웹서비스 분류체계 생성 프레임워크의 실증적 평가)

  • Hwang, You-Sub
    • Journal of Intelligence and Information Systems
    • /
    • v.16 no.2
    • /
    • pp.33-54
    • /
    • 2010
  • The World Wide Web is transitioning from being a mere collection of documents that contain useful information toward providing a collection of services that perform useful tasks. The emerging Web service technology has been envisioned as the next technological wave and is expected to play an important role in this recent transformation of the Web. By providing interoperable interface standards for application-to-application communication, Web services can be combined with component based software development to promote application interaction both within and across enterprises. To make Web services for service-oriented computing operational, it is important that Web service repositories not only be well-structured but also provide efficient tools for developers to find reusable Web service components that meet their needs. As the potential of Web services for service-oriented computing is being widely recognized, the demand for effective Web service discovery mechanisms is concomitantly growing. A number of public Web service repositories have been proposed, but the Web service taxonomy generation has not been satisfactorily addressed. Unfortunately, most existing Web service taxonomies are either too rudimentary to be useful or too hard to be maintained. In this paper, we propose a Web service taxonomy generation framework that combines an artificial neural network based clustering techniques with descriptive label generating and leverages the semantics of the XML-based service specification in WSDL documents. We believe that this is one of the first attempts at applying data mining techniques in the Web service discovery domain. We have developed a prototype system based on the proposed framework using an unsupervised artificial neural network and empirically evaluated the proposed approach and tool using real Web service descriptions drawn from operational Web service repositories. We report on some preliminary results demonstrating the efficacy of the proposed approach.

A Study on the Perception of Fashion Platforms and Fashion Smart Factories using Big Data Analysis (빅데이터 분석을 이용한 패션 플랫폼과 패션 스마트 팩토리에 대한 인식 연구)

  • Song, Eun-young
    • Fashion & Textile Research Journal
    • /
    • v.23 no.6
    • /
    • pp.799-809
    • /
    • 2021
  • This study aimed to grasp the perceptions and trends in fashion platforms and fashion smart factories using big data analysis. As a research method, big data analysis, fashion platform, and smart factory were identified through literature and prior studies, and text mining analysis and network analysis were performed after collecting text from the web environment between April 2019 and April 2021. After data purification with Textom, the words of fashion platform (1,0591 pieces) and fashion smart factory (9750 pieces) were used for analysis. Key words were derived, the frequency of appearance was calculated, and the results were visualized in word cloud and N-gram. The top 70 words by frequency of appearance were used to generate a matrix, structural equivalence analysis was performed, and the results were displayed using network visualization and dendrograms. The collected data revealed that smart factory had high social issues, but consumer interest and academic research were insufficient, and the amount and frequency of related words on the fashion platform were both high. As a result of structural equalization analysis, it was found that fashion platforms with strong connectivity between clusters are creating new competitiveness with service platforms that add sharing, manufacturing, and curation functions, and fashion smart factories can expect future value to grow together, according to digital technology innovation and platforms. This study can serve as a foundation for future research topics related to fashion platforms and smart factories.

The Analysis on the Relationship between Firms' Exposures to SNS and Stock Prices in Korea (기업의 SNS 노출과 주식 수익률간의 관계 분석)

  • Kim, Taehwan;Jung, Woo-Jin;Lee, Sang-Yong Tom
    • Asia pacific journal of information systems
    • /
    • v.24 no.2
    • /
    • pp.233-253
    • /
    • 2014
  • Can the stock market really be predicted? Stock market prediction has attracted much attention from many fields including business, economics, statistics, and mathematics. Early research on stock market prediction was based on random walk theory (RWT) and the efficient market hypothesis (EMH). According to the EMH, stock market are largely driven by new information rather than present and past prices. Since it is unpredictable, stock market will follow a random walk. Even though these theories, Schumaker [2010] asserted that people keep trying to predict the stock market by using artificial intelligence, statistical estimates, and mathematical models. Mathematical approaches include Percolation Methods, Log-Periodic Oscillations and Wavelet Transforms to model future prices. Examples of artificial intelligence approaches that deals with optimization and machine learning are Genetic Algorithms, Support Vector Machines (SVM) and Neural Networks. Statistical approaches typically predicts the future by using past stock market data. Recently, financial engineers have started to predict the stock prices movement pattern by using the SNS data. SNS is the place where peoples opinions and ideas are freely flow and affect others' beliefs on certain things. Through word-of-mouth in SNS, people share product usage experiences, subjective feelings, and commonly accompanying sentiment or mood with others. An increasing number of empirical analyses of sentiment and mood are based on textual collections of public user generated data on the web. The Opinion mining is one domain of the data mining fields extracting public opinions exposed in SNS by utilizing data mining. There have been many studies on the issues of opinion mining from Web sources such as product reviews, forum posts and blogs. In relation to this literatures, we are trying to understand the effects of SNS exposures of firms on stock prices in Korea. Similarly to Bollen et al. [2011], we empirically analyze the impact of SNS exposures on stock return rates. We use Social Metrics by Daum Soft, an SNS big data analysis company in Korea. Social Metrics provides trends and public opinions in Twitter and blogs by using natural language process and analysis tools. It collects the sentences circulated in the Twitter in real time, and breaks down these sentences into the word units and then extracts keywords. In this study, we classify firms' exposures in SNS into two groups: positive and negative. To test the correlation and causation relationship between SNS exposures and stock price returns, we first collect 252 firms' stock prices and KRX100 index in the Korea Stock Exchange (KRX) from May 25, 2012 to September 1, 2012. We also gather the public attitudes (positive, negative) about these firms from Social Metrics over the same period of time. We conduct regression analysis between stock prices and the number of SNS exposures. Having checked the correlation between the two variables, we perform Granger causality test to see the causation direction between the two variables. The research result is that the number of total SNS exposures is positively related with stock market returns. The number of positive mentions of has also positive relationship with stock market returns. Contrarily, the number of negative mentions has negative relationship with stock market returns, but this relationship is statistically not significant. This means that the impact of positive mentions is statistically bigger than the impact of negative mentions. We also investigate whether the impacts are moderated by industry type and firm's size. We find that the SNS exposures impacts are bigger for IT firms than for non-IT firms, and bigger for small sized firms than for large sized firms. The results of Granger causality test shows change of stock price return is caused by SNS exposures, while the causation of the other way round is not significant. Therefore the correlation relationship between SNS exposures and stock prices has uni-direction causality. The more a firm is exposed in SNS, the more is the stock price likely to increase, while stock price changes may not cause more SNS mentions.

Development of an Organism-specific Protein Interaction Database with Supplementary Data from the Web Sources (다양한 웹 데이터를 이용한 특정 유기체의 단백질 상호작용 데이터베이스 개발)

  • Hwang, Doo-Sung
    • The KIPS Transactions:PartD
    • /
    • v.9D no.6
    • /
    • pp.1091-1096
    • /
    • 2002
  • This paper presents the development of a protein interaction database. The developed system is characterized as follows. First, the proposed system not only maintains interaction data collected by an experiment, but also the genomic information of the protein data. Secondly, the system can extract details on interacting proteins through the developed wrappers. Thirdly, the system is based on wrapper-based system in order to extract the biologically meaningful data from various web sources and integrate them into a relational database. The system inherits a layered-modular architecture by introducing a wrapper-mediator approach in order to solve the syntactic and semantic heterogeneity among multiple data sources. Currently the system has wrapped the relevant data for about 40% of about 11,500 proteins on average from various accessible sources. A wrapper-mediator approach makes a protein interaction data comprehensive and useful with support of data interoperability and integration. The developing database will be useful for mining further knowledge and analysis of human life in proteomics studies.

Analysis of Research Trends in Relation to the Yellow Sea using Text Mining (텍스트 마이닝을 활용한 황해 관련 연구동향 분석연구)

  • Kyu Won Hwang;Kim Jinkyung;Kang Seung-Koo;Kang Gil Mo
    • Journal of the Korean Society of Marine Environment & Safety
    • /
    • v.29 no.7
    • /
    • pp.724-739
    • /
    • 2023
  • Located in the sea area between South Korea, North Korea, and China, the Yellow Sea plays an important role from a geopolitical perspective, and recently, as the use of marine space in the Yellow Sea is expanding, its social and economic values have been increasing further. In addition, owing to rapid climate changes, the need for joint response and cooperation between Korea and China is increasing in various fields, including changes in the marine environment and marine ecosystem and generation and movement of air pollutants. Accordingly, in this study, core topics were derived from research papers with the Yellow Sea as a keyword, and research trends to date were explored through author network analysis. As a specific research method, research papers related to the Yellow Sea published between 1984 and 2021 were extracted from the Web of Science database and were classified into four periods to derive core topics using topic modeling, a type of text mining. Furthermore, the influences of major research communities, researchers, and research institutes in the appropriate fields were identified through analyzing the author network, and their implications were presented. The analysis results indicated that the core topics of research papers on the Yellow Sea had changed over time, and differences existed in the influence (centrality) of key researchers. Finally, based on the results of this study, this study aims to identify research trends related to the Yellow Sea, major researchers, and research institutes and contribute to research cooperation between Korea and China regarding the Yellow Sea in the future.

Operational Big Data Analytics platform for Smart Factory (스마트팩토리를 위한 운영빅데이터 분석 플랫폼)

  • Bae, Hyerim;Park, Sanghyuck;Choi, Yulim;Joo, Byeongjun;Sutrisnowati, Riska Asriana;Pulshashi, Iq Reviessay;Putra, Ahmad Dzulfikar Adi;Adi, Taufik Nur;Lee, Sanghwa;Won, Seokrae
    • The Journal of Bigdata
    • /
    • v.1 no.2
    • /
    • pp.9-19
    • /
    • 2016
  • Since ICT convergence became a major issue, German government has carried forward a policy 'Industry 4.0' that triggered ICT convergence with manufacturing. Now this trend gets into our stride. From this facts, we can expect great leap up to quality perfection in low cost. Recently Korean government also enforces policy with 'Manufacturing 3.0' for upgrading Korean manufacturing industry with being accelerated by many related technologies. We, in the paper, developed a custom-made operational big data analysis platform for the implementation of operational intelligence to improve industry capability. Our platform is designed based on spring framework and web. In addition, HDFS and spark architectures helps our system analyze massive data on the field with streamed data processed by process mining algorithm. Extracted knowledge from data will support enhancement of manufacturing performance.

  • PDF

Evaluation of Collaborative Filtering Methods for Developing Online Music Contents Recommendation System (온라인 음악 콘텐츠 추천 시스템 구현을 위한 협업 필터링 기법들의 비교 평가)

  • Yoo, Youngseok;Kim, Jiyeon;Sohn, Bangyong;Jung, Jongjin
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.66 no.7
    • /
    • pp.1083-1091
    • /
    • 2017
  • As big data technologies have been developed and massive data have exploded from users through various channels, CEO of global IT enterprise mentioned core importance of data in next generation business. Therefore various machine learning technologies have been necessary to apply data driven services but especially recommendation has been core technique in viewpoint of directly providing summarized information or exact choice of items to users in information flooding environment. Recently evolved recommendation techniques have been proposed by many researchers and most of service companies with big data tried to apply refined recommendation method on their online business. For example, Amazon used item to item collaborative filtering method on its sales distribution platform. In this paper, we develop a commercial web service for suggesting music contents and implement three representative collaborative filtering methods on the service. We also produce recommendation lists with three methods based on real world sample data and evaluate the usefulness of them by comparison among the produced result. This study is meaningful in terms of suggesting the right direction and practicality when companies and developers want to develop web services by applying big data based recommendation techniques in practical environment.

Performance analysis of volleyball games using the social network and text mining techniques (사회네트워크분석과 텍스트마이닝을 이용한 배구 경기력 분석)

  • Kang, Byounguk;Huh, Mankyu;Choi, Seungbae
    • Journal of the Korean Data and Information Science Society
    • /
    • v.26 no.3
    • /
    • pp.619-630
    • /
    • 2015
  • The purpose of this study is to provide basic information to develop a game strategy plan of a team in a future by identifying the patterns of attack and pass of national men's professional volleyball teams and extracting core key words related with volleyball game performance to evaluate game performance using 'social network analysis' and 'text mining'. As for the analysis result of 'social network analysis' with the whole data, group '0' (6 players) and group '1' (11 players) were partitioned. A point of view the degree centrality and betweenness centrality in 'social network analysis' results, we can know that the group '1' more active game performance than the group '0'. The significant result for two group (win and loss) obtained by 'text mining' according to two groups ('0' and '1') obtained by 'social network analysis' showed significant difference (p-value: 0.001). As for clustering of each network, group '0' had the tendency to score points through set player D and E. In group '1', the player K had the tendency to fail if he attack through 'dig'; players C and D have a good performance through 'set' play.