• Title/Summary/Keyword: Big data processing

Search Result 1,038, Processing Time 0.023 seconds

Distributed Processing of Big Data Analysis based on R using SparkR (SparkR을 이용한 R 기반 빅데이터 분석의 분산 처리)

  • Ryu, Woo-Seok
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.17 no.1
    • /
    • pp.161-166
    • /
    • 2022
  • In this paper, we analyze the problems that occur when performing the big data analysis using R as a data analysis tool, and present the usefulness of the data analysis with SparkR which connects R and Spark to support distributed processing of big data effectively. First, we study the memory allocation problem of R which occurs when loading large amounts of data and performing operations, and the characteristics and programming environment of SparkR. And then, we perform the comparison analysis of the execution performance when linear regression analysis is performed in each environment. As a result of the analysis, it was shown that R can be used for data analysis through SparkR without additional language learning, and the code written in R can be effectively processed distributedly according to the increase in the number of nodes in the cluster.

Semantic Computing for Big Data: Approaches, Tools, and Emerging Directions (2011-2014)

  • Jeong, Seung Ryul;Ghani, Imran
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.8 no.6
    • /
    • pp.2022-2042
    • /
    • 2014
  • The term "big data" has recently gained widespread attention in the field of information technology (IT). One of the key challenges in making use of big data lies in finding ways to uncover relevant and valuable information. The high volume, velocity, and variety of big data hinder the use of solutions that are available for smaller datasets, which involve the manual interpretation of data. Semantic computing technologies have been proposed as a means of dealing with these issues, and with the advent of linked data in recent years, have become central to mainstream semantic computing. This paper attempts to uncover the state-of-the-art semantics-based approaches and tools that can be leveraged to enrich and enhance today's big data. It presents research on the latest literature, including 61 studies from 2011 to 2014. In addition, it highlights the key challenges that semantic approaches need to address in the near future. For instance, this paper presents cutting-edge approaches to ontology engineering, ontology evolution, searching and filtering relevant information, extracting and reasoning, distributed (web-scale) reasoning, and representing big data. It also makes recommendations that may encourage researchers to more deeply explore the applications of semantic technology, which could improve the processing of big data. The findings of this study contribute to the existing body of basic knowledge on semantics and computational issues related to big data, and may trigger further research on the field. Our analysis shows that there is a need to put more effort into proposing new approaches, and that tools must be created that support researchers and practitioners in realizing the true power of semantic computing and solving the crucial issues of big data.

Study on the Big Data Platform Construction of Fisheries (수산업 빅데이터 플랫폼 구축 방안에 대한 연구)

  • Choi, Joowon;Jung, Jaewook;Kim, Youngae;Shin, Yongtae
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.8
    • /
    • pp.181-188
    • /
    • 2020
  • The fisheries industry is rapidly shifting from a traditional fishery to aquaculture paradigm and it faces various problems such as depletion of fishery resources and aging of fishing villages. We need the establishment of a fisheries big data platform that includes both the data of the central and surrounding industries of the fisheries industry for enhancement of establishment of a fisheries, 6th industrialization of fishing villages, establishment of related technical standards, and discovery of the new industries to overcome this. Data center agencies should collect, link, and pre-processing, and the platform organizer should create a water industry data virtuous circle through the establishment, operation, and data market of big data platforms to help overcome the current crisis, secure smart fisheries hegemony, and use it as a key to value transfer. Through this study, I would like to propose a policy and technical big data platform construction plan to successfully promote it.

Aircraft Recognition from Remote Sensing Images Based on Machine Vision

  • Chen, Lu;Zhou, Liming;Liu, Jinming
    • Journal of Information Processing Systems
    • /
    • v.16 no.4
    • /
    • pp.795-808
    • /
    • 2020
  • Due to the poor evaluation indexes such as detection accuracy and recall rate when Yolov3 network detects aircraft in remote sensing images, in this paper, we propose a remote sensing image aircraft detection method based on machine vision. In order to improve the target detection effect, the Inception module was introduced into the Yolov3 network structure, and then the data set was cluster analyzed using the k-means algorithm. In order to obtain the best aircraft detection model, on the basis of our proposed method, we adjusted the network parameters in the pre-training model and improved the resolution of the input image. Finally, our method adopted multi-scale training model. In this paper, we used remote sensing aircraft dataset of RSOD-Dataset to do experiments, and finally proved that our method improved some evaluation indicators. The experiment of this paper proves that our method also has good detection and recognition ability in other ground objects.

Automatic Generation of Issue Analysis Report Based on Social Big Data Mining (소셜 빅데이터 마이닝 기반 이슈 분석보고서 자동 생성)

  • Heo, Jeong;Lee, Chung Hee;Oh, Hyo Jung;Yoon, Yeo Chan;Kim, Hyun Ki;Jo, Yo Han;Ock, Cheol Young
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.3 no.12
    • /
    • pp.553-564
    • /
    • 2014
  • In this paper, we propose the system for automatic generation of issue analysis report based on social big data mining, with the purpose of resolving three problems of the previous technologies in a social media analysis and analytic report generation. Three problems are the isolation of analysis, the subjectivity of experts and the closure of information attributable to a high price. The system is comprised of the natural language query analysis, the issue analysis, the social big data analysis, the social big data correlation analysis and the automatic report generation. For the evaluation of report usefulness, we used a Likert scale and made two experts of big data analysis evaluate. The result shows that the quality of report is comparatively useful and reliable. Because of a low price of the report generation, the correlation analysis of social big data and the objectivity of social big data analysis, the proposed system will lead us to the popularization of social big data analysis.

A Stochastic Model for Virtual Data Generation of Crack Patterns in the Ceramics Manufacturing Process

  • Park, Youngho;Hyun, Sangil;Hong, Youn-Woo
    • Journal of the Korean Ceramic Society
    • /
    • v.56 no.6
    • /
    • pp.596-600
    • /
    • 2019
  • Artificial intelligence with a sufficient amount of realistic big data in certain applications has been demonstrated to play an important role in designing new materials or in manufacturing high-quality products. To reduce cracks in ceramic products using machine learning, it is desirable to utilize big data in recently developed data-driven optimization schemes. However, there is insufficient big data for ceramic processes. Therefore, we developed a numerical algorithm to make "virtual" manufacturing data sets using indirect methods such as computer simulations and image processing. In this study, a numerical algorithm based on the random walk was demonstrated to generate images of cracks by adjusting the conditions of the random walk process such as the number of steps, changes in direction, and the number of cracks.