• Title/Summary/Keyword: Big Data Environment

Search Result 962, Processing Time 0.03 seconds

A Review on the Management of Water Resources Information based on Big Data and Cloud Computing (빅 데이터와 클라우드 컴퓨팅 기반의 수자원 정보 관리 방안에 관한 검토)

  • Kim, Yonsoo;Kang, Narae;Jung, Jaewon;Kim, Hung Soo
    • Journal of Wetlands Research
    • /
    • v.18 no.1
    • /
    • pp.100-112
    • /
    • 2016
  • In recent, the direction of water resources policy is changing from the typical plan for water use and flood control to the sustainable water resources management to improve the quality of life. This change makes the information related to water resources such as data collection, management, and supply is becoming an important concern for decision making of water resources policy. We had analyzed the structured data according to the purpose of providing information on water resources. However, the recent trend is big data and cloud computing which can create new values by linking unstructured data with structured data. Therefore, the trend for the management of water resources information is also changing. According to the paradigm change of information management, this study tried to suggest an application of big data and cloud computing in water resources field for efficient management and use of water. We examined the current state and direction of policy related to water resources information in Korea and an other country. Then we connected volume, velocity and variety which are the three basic components of big data with veracity and value which are additionally mentioned recently. And we discussed the rapid and flexible countermeasures about changes of consumer and increasing big data related to water resources via cloud computing. In the future, the management of water resources information should go to the direction which can enhance the value(Value) of water resources information by big data and cloud computing based on the amount of data(Volume), the speed of data processing(Velocity), the number of types of data(Variety). Also it should enhance the value(Value) of water resources information by the fusion of water and other areas and by the production of accurate information(Veracity) required for water management and prevention of disaster and for protection of life and property.

A Study on the Intergrated Voice/Data transmission Algorithm characteristics on Local Area Network (유선 LAN상의 음성/데이타 혼합전송 알고리즘 특성에 관한 연구)

  • 김동일
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.1 no.2
    • /
    • pp.137-143
    • /
    • 1997
  • From now on, the network is being developed into PSTN(public switched telephone network) and PDN(public data network), that is depend on the form of data. The former one pursues sending voice, and the latter one pursues sending data. But it causes big loss of the economy and efficiency. So, ISDN, processing voice and data at same time, gives a big profit to user. To enlarge the ISDN at the narrow area, it is necessary that study to send the mixture form of voice and data in LAN environment. So, this paper proposes the algorithm about the mixture form of voice and data in ethernet and token-ring. that is widely used in these days.

  • PDF

Big data distributed processing system using RHadoop (RHadoop을 이용한 빅데이터 분산처리 시스템)

  • Shin, Ji Eun;Jung, Byung Ho;Lim, Dong Hoon
    • Journal of the Korean Data and Information Science Society
    • /
    • v.26 no.5
    • /
    • pp.1155-1166
    • /
    • 2015
  • It is almost impossible to store or analyze big data increasing exponentially with traditional technologies, so Hadoop is a new technology to make that possible. In recent R is using as an engine for big data analysis based on distributed processing with Hadoop technology. With RHadoop that integrates R and Hadoop environment, we implemented parallel multiple regression analysis with various data sizes of actual data and simulated data. Experimental results showed our RHadoop system was faster as the number of data nodes increases. We also compared the performance of our RHadoop with lm function and biglm packages available on bigmemory. The results showed that our RHadoop was faster than other packages owing to paralleling processing with increasing the number of map tasks as the size of data increases.

KISTI-ML Platform: A Community-based Rapid AI Model Development Tool for Scientific Data (KISTI-ML 플랫폼: 과학기술 데이터를 위한 커뮤니티 기반 AI 모델 개발 도구)

  • Lee, Jeongcheol;Ahn, Sunil
    • Journal of Internet Computing and Services
    • /
    • v.20 no.6
    • /
    • pp.73-84
    • /
    • 2019
  • Machine learning as a service, the so-called MLaaS, has recently attracted much attention in almost all industries and research groups. The main reason for this is that you do not need network servers, storage, or even data scientists, except for the data itself, to build a productive service model. However, machine learning is often very difficult for most developers, especially in traditional science due to the lack of well-structured big data for scientific data. For experiment or application researchers, the results of an experiment are rarely shared with other researchers, so creating big data in specific research areas is also a big challenge. In this paper, we introduce the KISTI-ML platform, a community-based rapid AI model development for scientific data. It is a place where machine learning beginners use their own data to automatically generate code by providing a user-friendly online development environment. Users can share datasets and their Jupyter interactive notebooks among authorized community members, including know-how such as data preprocessing to extract features, hidden network design, and other engineering techniques.

Performance Comparison of Spatial Split Algorithms for Spatial Data Analysis on Spark (Spark 기반 공간 분석에서 공간 분할의 성능 비교)

  • Yang, Pyoung Woo;Yoo, Ki Hyun;Nam, Kwang Woo
    • Journal of Korean Society for Geospatial Information Science
    • /
    • v.25 no.1
    • /
    • pp.29-36
    • /
    • 2017
  • In this paper, we implement a spatial big data analysis prototype based on Spark which is an in-memory system and compares the performance by the spatial split algorithm on this basis. In cluster computing environments, big data is divided into blocks of a certain size order to balance the computing load of big data. Existing research showed that in the case of the Hadoop based spatial big data system, the split method by spatial is more effective than the general sequential split method. Hadoop based spatial data system stores raw data as it is in spatial-divided blocks. However, in the proposed Spark-based spatial analysis system, there is a difference that spatial data is converted into a memory data structure and stored in a spatial block for search efficiency. Therefore, in this paper, we propose an in-memory spatial big data prototype and a spatial split block storage method. Also, we compare the performance of existing spatial split algorithms in the proposed prototype. We presented an appropriate spatial split strategy with the Spark based big data system. In the experiment, we compared the query execution time of the spatial split algorithm, and confirmed that the BSP algorithm shows the best performance.

Scalable Big Data Pipeline for Video Stream Analytics Over Commodity Hardware

  • Ayub, Umer;Ahsan, Syed M.;Qureshi, Shavez M.
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.4
    • /
    • pp.1146-1165
    • /
    • 2022
  • A huge amount of data in the form of videos and images is being produced owning to advancements in sensor technology. Use of low performance commodity hardware coupled with resource heavy image processing and analyzing approaches to infer and extract actionable insights from this data poses a bottleneck for timely decision making. Current approach of GPU assisted and cloud-based architecture video analysis techniques give significant performance gain, but its usage is constrained by financial considerations and extremely complex architecture level details. In this paper we propose a data pipeline system that uses open-source tools such as Apache Spark, Kafka and OpenCV running over commodity hardware for video stream processing and image processing in a distributed environment. Experimental results show that our proposed approach eliminates the need of GPU based hardware and cloud computing infrastructure to achieve efficient video steam processing for face detection with increased throughput, scalability and better performance.

A Study on the Technological Priorities of Manufacturing and Service Companies for Response to the 4th Industrial Revolution and Transformation into a Smart Company (4차 산업혁명 대응과 스마트 기업으로의 변화를 위한 제조 및 서비스 기업의 기술적용 우선순위에 대한 연구)

  • Park, Chan-Kwon;Seo, Yeong-Bok
    • Journal of Convergence for Information Technology
    • /
    • v.11 no.4
    • /
    • pp.83-101
    • /
    • 2021
  • This study is to investigate, using AHP, what technologies should be applied first to Korean SMEs in order to respond to the 4th industrial revolution and change to a smart enterprise. To this end, technologies related to the 4th industrial revolution and smart factory are synthesized, and the classification criteria of Dae-Hoon Kim et al. (2019) are applied, but additional opinions of experts are collected and related technologies are converted to artificial intelligence (AI), Big Data, and Cloud Computing. As a base technology, mobile, Internet of Things (IoT), block chain as hyper-connected technology, unmanned transportation (autonomous driving), robot, 3D printing, drone as a convergence technology, smart manufacturing and logistics, smart healthcare, smart transportation and smart finance were classified as smart industrial technologies. As a result of confirming the priorities for technical use by AHP analysis and calculating the total weight, manufacturing companies have a high ranking in mobile, artificial intelligence (AI), big data, and robots, while service companies are in big data and robots, artificial intelligence (AI), and smart healthcare are ranked high, and in all companies, it is in the order of big data, artificial intelligence (AI), robot, and mobile. Through this study, it was clearly identified which technologies should be applied first in order to respond to the 4th industrial revolution and change to a smart company.

A Study on Big Data Based Investment Strategy Using Internet Search Trends (인터넷 검색추세를 활용한 빅데이터 기반의 주식투자전략에 대한 연구)

  • Kim, Minsoo;Koo, Pyunghoi
    • Journal of the Korean Operations Research and Management Science Society
    • /
    • v.38 no.4
    • /
    • pp.53-63
    • /
    • 2013
  • Together with soaring interest on Big Data, now there are vigorous reports that unearth various social values lying underneath those data from a number of application areas. Among those reports many are using such data as Internet search histories from Google site, social relationships from Facebook, and transactional or locational traces collected from various ubiquitous devices. Many of those researches, however, are conducted based on the data sets that are accumulated over the North American and European areas, which means that direct interpretation and application of social values exhibited by those researches to the other areas like Korea can be a disturbing task. This research has started from a validation study against Korean environment of the former paper which says an investment strategy that exploits up and down of Google search volume on a carefully selected set of terms shows high market performance. A huge difference between North American and Korean environment can be eye witnessed via the distinction in profit rates that are exhibited by the corresponding set of search terms. Two sets of search terms actually presented low correlation in their profit rates over two financial markets. Even in an experiment which compares the profit rates with two different investment periods with the same set of search terms showed no such meaningful result that outperforms the market average. With all these results, we cautiously conclude that establishing an investment strategy that exploits Internet search volume over a specified word set needs more conscious approach.

Development of Customized 3D Characters for Growth Management and Prediction of Adolescents Using Big Data (빅데이터를 활용한 청소년 성장관리와 예측을 위한 맞춤형 3D 캐릭터 개발 연구)

  • Choo, Hye-Jin;Ha, Seo-Ho
    • The Journal of the Korea Contents Association
    • /
    • v.18 no.1
    • /
    • pp.250-257
    • /
    • 2018
  • Today, the integration of the rapid development of ICT and the smart devices moves our lives quickly into an online community environment through not only quick and easy information search but also various social media. Accordingly, individual activities in the smart media environment are pouring out vast quantities of data in many fields, accumulating a tremendous amount of data. The everyday data of individuals is reproducing different values from the previous ones, while suggesting new customized services that utilize them in various fields. Recently, big data utilization has attracted a great attention in the field of healthcare. Especially, development of healthcare service linked with mobile is expected to bring a new paradigm in this field. In this study, creation of a 3D avatar character model as a means to transfer information to individuals more efficiently is proposed in the development of mobile customized service for health promotion and growth prediction of children and adolescents, at the same time, an effective visual expression method to have a sense of immersion and unity is searched.

Mobile-based Big Data Processing and Monitoring Technology in IoT Environment (IoT 환경에서 모바일 기반 빅데이터 처리 및 모니터링 기술)

  • Lee, Seung-Hae;Kim, Ju-Ho;Shin, Dong-Youn;Shin, Dong-Jin;Park, Jeong-Min;Kim, Jeong-Joon
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.18 no.6
    • /
    • pp.1-9
    • /
    • 2018
  • In the fourth industrial revolution, which has become an issue now, we have been able to receive instant analysis results faster than the existing slow speed through various Big Data technologies, and to conduct real-time monitoring on mobile and web. First, various irregular sensor Data is generated using IoT device, Raspberry Pi. Sensor Data is collected in real time, and the collected data is distributed and stored using several nodes. Then, the stored Sensor Data is processed and refined. Visualize and output the analysis result after analysis. By using these methods, we can train the human resources required for Big Data and mobile related fields using IoT, and process data efficiently and quickly. We also provide information that can confirm the reliability of research results through real time monitoring.