• Title/Summary/Keyword: 빅데이터 기법

Search Result 796, Processing Time 0.025 seconds

The Efficient Method of Parallel Genetic Algorithm using MapReduce of Big Data (빅 데이터의 MapReduce를 이용한 효율적인 병렬 유전자 알고리즘 기법)

  • Hong, Sung-Sam;Han, Myung-Mook
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.23 no.5
    • /
    • pp.385-391
    • /
    • 2013
  • Big Data is data of big size which is not processed, collected, stored, searched, analyzed by the existing database management system. The parallel genetic algorithm using the Hadoop for BigData technology is easily realized by implementing GA(Genetic Algorithm) using MapReduce in the Hadoop Distribution System. The previous study that the genetic algorithm using MapReduce is proposed suitable transforming for the GA by MapReduce. However, they did not show good performance because of frequently occurring data input and output. In this paper, we proposed the MRPGA(MapReduce Parallel Genetic Algorithm) using improvement Map and Reduce process and the parallel processing characteristic of MapReduce. The optimal solution can be found by using the topology, migration of parallel genetic algorithm and local search algorithm. The convergence speed of the proposal method is 1.5 times faster than that of the existing MapReduce SGA, and is the optimal solution can be found quickly by the number of sub-generation iteration. In addition, the MRPGA is able to improve the processing and analysis performance of Big Data technology.

Big Data Analysis of Busan Civil Affairs Using the LDA Topic Modeling Technique (LDA 토픽모델링 기법을 활용한 부산시 민원 빅데이터 분석)

  • Park, Ju-Seop;Lee, Sae-Mi
    • Informatization Policy
    • /
    • v.27 no.2
    • /
    • pp.66-83
    • /
    • 2020
  • Local issues that occur in cities typically garner great attention from the public. While local governments strive to resolve these issues, it is often difficult to effectively eliminate them all, which leads to complaints. In tackling these issues, it is imperative for local governments to use big data to identify the nature of complaints, and proactively provide solutions. This study applies the LDA topic modeling technique to research and analyze trends and patterns in complaints filed online. To this end, 9,625 cases of online complaints submitted to the city of Busan from 2015 to 2017 were analyzed, and 20 topics were identified. From these topics, key topics were singled out, and through analysis of quarterly weighting trends, four "hot" topics(Bus stops, Taxi drivers, Praises, and Administrative handling) and four "cold" topics(CCTV installation, Bus routes, Park facilities including parking, and Festivities issues) were highlighted. The study conducted big data analysis for the identification of trends and patterns in civil affairs and makes an academic impact by encouraging follow-up research. Moreover, the text mining technique used for complaint analysis can be used for other projects requiring big data processing.

Food Exchange Table Organization Model Based on Decision Tree Using Machine Learning (머신러닝을 이용한 의사결정트리 기반의 식품교환표 구성 모델)

  • Kim, JiYun;Lee, Sangmin;Jeon, Hyeongjun;Kim, Gaeun;Kim, Ji-Hyun;Park, Naeun;Jin, ChangGyun;Kwon, Jin young;Kim Jongwan
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2020.11a
    • /
    • pp.680-684
    • /
    • 2020
  • 최근 국내에서는 식품에 대한 관심도가 높아짐에 따라 먹거리에 건강·환경·미래지향적 가치가 부여되고 있으며 식품 산업에서도 신규 식품 개발이 증가하는 추세이다. 식단을 구성할 때 기준이 되는 식품교환표는 개정과정에서 많은 인력과 시간이 소요되기 때문에 식품 섭취 변화를 신속하게 반영하기 어렵다. 본 논문에서는 식품교환표의 활용도를 높이기 위한 식품교환표 갱신 기법을 제안한다. 제안 기법은 의사결정트리 모델을 학습하여 새롭게 추가된 식품의 정보를 바탕으로 식품군을 분류하여 식품교환표를 갱신한다. 이는 영양 관리가 필요한 당뇨병 환자 등에게 실용적이며 기호성·다양성이 높은 식단을 구성하는 데 도움을 준다.

A Multimodal Profile Ensemble Approach to Development of Recommender Systems Using Big Data (빅데이터 기반 추천시스템 구현을 위한 다중 프로파일 앙상블 기법)

  • Kim, Minjeong;Cho, Yoonho
    • Journal of Intelligence and Information Systems
    • /
    • v.21 no.4
    • /
    • pp.93-110
    • /
    • 2015
  • The recommender system is a system which recommends products to the customers who are likely to be interested in. Based on automated information filtering technology, various recommender systems have been developed. Collaborative filtering (CF), one of the most successful recommendation algorithms, has been applied in a number of different domains such as recommending Web pages, books, movies, music and products. But, it has been known that CF has a critical shortcoming. CF finds neighbors whose preferences are like those of the target customer and recommends products those customers have most liked. Thus, CF works properly only when there's a sufficient number of ratings on common product from customers. When there's a shortage of customer ratings, CF makes the formation of a neighborhood inaccurate, thereby resulting in poor recommendations. To improve the performance of CF based recommender systems, most of the related studies have been focused on the development of novel algorithms under the assumption of using a single profile, which is created from user's rating information for items, purchase transactions, or Web access logs. With the advent of big data, companies got to collect more data and to use a variety of information with big size. So, many companies recognize it very importantly to utilize big data because it makes companies to improve their competitiveness and to create new value. In particular, on the rise is the issue of utilizing personal big data in the recommender system. It is why personal big data facilitate more accurate identification of the preferences or behaviors of users. The proposed recommendation methodology is as follows: First, multimodal user profiles are created from personal big data in order to grasp the preferences and behavior of users from various viewpoints. We derive five user profiles based on the personal information such as rating, site preference, demographic, Internet usage, and topic in text. Next, the similarity between users is calculated based on the profiles and then neighbors of users are found from the results. One of three ensemble approaches is applied to calculate the similarity. Each ensemble approach uses the similarity of combined profile, the average similarity of each profile, and the weighted average similarity of each profile, respectively. Finally, the products that people among the neighborhood prefer most to are recommended to the target users. For the experiments, we used the demographic data and a very large volume of Web log transaction for 5,000 panel users of a company that is specialized to analyzing ranks of Web sites. R and SAS E-miner was used to implement the proposed recommender system and to conduct the topic analysis using the keyword search, respectively. To evaluate the recommendation performance, we used 60% of data for training and 40% of data for test. The 5-fold cross validation was also conducted to enhance the reliability of our experiments. A widely used combination metric called F1 metric that gives equal weight to both recall and precision was employed for our evaluation. As the results of evaluation, the proposed methodology achieved the significant improvement over the single profile based CF algorithm. In particular, the ensemble approach using weighted average similarity shows the highest performance. That is, the rate of improvement in F1 is 16.9 percent for the ensemble approach using weighted average similarity and 8.1 percent for the ensemble approach using average similarity of each profile. From these results, we conclude that the multimodal profile ensemble approach is a viable solution to the problems encountered when there's a shortage of customer ratings. This study has significance in suggesting what kind of information could we use to create profile in the environment of big data and how could we combine and utilize them effectively. However, our methodology should be further studied to consider for its real-world application. We need to compare the differences in recommendation accuracy by applying the proposed method to different recommendation algorithms and then to identify which combination of them would show the best performance.

Image-based Artificial Intelligence Deep Learning to Protect the Big Data from Malware (악성코드로부터 빅데이터를 보호하기 위한 이미지 기반의 인공지능 딥러닝 기법)

  • Kim, Hae Jung;Yoon, Eun Jun
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.54 no.2
    • /
    • pp.76-82
    • /
    • 2017
  • Malware, including ransomware to quickly detect, in this study, to provide an analysis method of malicious code through the image analysis that has been learned in the deep learning of artificial intelligence. First, to analyze the 2,400 malware data, and learning in artificial neural network Convolutional neural network and to image data. Extracts subgraphs to convert the graph of abstracted image, summarizes the set represent malware. The experimentally analyzed the malware is not how similar. Using deep learning of artificial intelligence by classifying malware and It shows the possibility of accurate malware detection.

Process analysis in Supply Chain Management with Process Mining: A Case Study (프로세스 마이닝 기법을 활용한 공급망 분석: 사례 연구)

  • Lee, Yonghyeok;Yi, Hojeong;Song, Minseok;Lee, Sang-Jin;Park, Sera
    • The Journal of Bigdata
    • /
    • v.1 no.2
    • /
    • pp.65-78
    • /
    • 2016
  • In the rapid change of business environment, it is crucial that several companies with core competence cooperate together in order to deliver competitive products to the market faster. Thus a lot of companies are participating in supply chains and SCM (Supply Chain Management) become more important. To efficiently manage supply chains, the analysis of data from SCM systems is required. In this paper, we explain how to analyze SCM related data with process mining techniques. After discussing the data requirement for process mining, several process mining techniques for the data analysis are explained. To show the applicability of the techniques, we have performed a case study with a company in South Korea. The case study shows that process mining is useful tool to analyze SCM data. On specifically, an overall process, several performance measures, and social networks can be easily discovered and analyzed with the techniques.

  • PDF

The Comparison Among Prediction Methods of Water Demand And Analysis of Data on Water Services Using Data Mining Techniques (데이터마이닝 기법을 활용한 상수 이용현황 분석 및 단기 물 수요예측 방법 비교)

  • Ahn, Jihoon;Kim, Jinhwa
    • The Journal of Bigdata
    • /
    • v.1 no.1
    • /
    • pp.9-17
    • /
    • 2016
  • This study identifies major features in water supply and introduces important factors in water services based on the information from data mining analysis of water quantity and water pressure measured from sensors. It also suggests more accurate methods using multiple regression analysis and neural network in predicting short term prediction of water demand in water service. A small block of a county is selected for the data collection and tests. There isa water demand on business such as public offices and hospitalstoo in this area. Real stream data from sensors in this area is collected. Among 2,728 data sets collected, 2,632 sets are used for modelling and 96 sets are used for testing. The shows that neural network is better than multiple regression analysis in their prediction performance.

  • PDF

A proposal on a proactive crawling approach with analysis of state-of-the-art web crawling algorithms (최신 웹 크롤링 알고리즘 분석 및 선제적인 크롤링 기법 제안)

  • Na, Chul-Won;On, Byung-Won
    • Journal of Internet Computing and Services
    • /
    • v.20 no.3
    • /
    • pp.43-59
    • /
    • 2019
  • Today, with the spread of smartphones and the development of social networking services, structured and unstructured big data have stored exponentially. If we analyze them well, we will get useful information to be able to predict data for the future. Large amounts of data need to be collected first in order to analyze big data. The web is repository where these data are most stored. However, because the data size is large, there are also many data that have information that is not needed as much as there are data that have useful information. This has made it important to collect data efficiently, where data with unnecessary information is filtered and only collected data with useful information. Web crawlers cannot download all pages due to some constraints such as network bandwidth, operational time, and data storage. This is why we should avoid visiting many pages that are not relevant to what we want and download only important pages as soon as possible. This paper seeks to help resolve the above issues. First, We introduce basic web-crawling algorithms. For each algorithm, the time-complexity and pros and cons are described, and compared and analyzed. Next, we introduce the state-of-the-art web crawling algorithms that have improved the shortcomings of the basic web crawling algorithms. In addition, recent research trends show that the web crawling algorithms with special purposes such as collecting sentiment words are actively studied. We will one of the introduce Sentiment-aware web crawling techniques that is a proactive web crawling technique as a study of web crawling algorithms with special purpose. The result showed that the larger the data are, the higher the performance is and the more space is saved.

Analysis of Urban Traffic Network Structure based on ITS Big Data (ITS 빅데이터를 활용한 도시 교통네트워크 구조분석)

  • Kim, Yong Yeon;Lee, Kyung-Hee;Cho, Wan-Sup
    • The Journal of Bigdata
    • /
    • v.2 no.2
    • /
    • pp.1-7
    • /
    • 2017
  • Intelligent transportation system (ITS) has been introduced to maximize the efficiency of operation and utilization of the urban traffic facilities and promote the safety and convenience of the users. With the expansion of ITS, various traffic big data such as road traffic situation, traffic volume, public transportation operation status, management situation, and public traffic use status have been increased exponentially. In this paper, we derive structural characteristics of urban traffic according to the vehicle flow by using big data network analysis. DSRC (Dedicated Short Range Communications) data is used to construct the traffic network. The results can help to understand the complex urban traffic characteristics more easily and provide basic research data for urban transportation plan such as road congestion resolution plan, road expansion plan, and bus line/interval plan in a city.

  • PDF

Big Data Model for Analyzing Plant Growth Environment Informations and Biometric Informations (농작물 생육환경정보와 생체정보 분석을 위한 빅데이터 모델)

  • Lee, JongYeol;Moon, ChangBae;Kim, ByeongMan
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.25 no.6
    • /
    • pp.15-23
    • /
    • 2020
  • While research activities in the agricultural field for climate change are being actively carried out, smart agriculture using information and communication technology has become a new trend in line with the Fourth Industrial Revolution. Accordingly, research is being conducted to identify and respond to signs of abnormal growth in advance by monitoring the stress of crops in various outdoor environments and soil conditions. There are also attempts to analyze data collected in real time through various sensors using artificial intelligence techniques or big data technologies. In this paper, we propose a big data model that is effective in analyzing the growth environment informations and biometric information of crops by using the existing relational database for big data analysis. The performance of the model was measured by the response time to a query according to the amount of data. As a result, it was confirmed that there is a maximum time reduction effect of 23.8%.