• Title/Summary/Keyword: Big data storage

Search Result 206, Processing Time 0.031 seconds

A biometric information collecting system for biomedical big data analysis (생체 의학 빅 데이터 분석을 위한 생체 정보 수집 시스템)

  • Lim, Damsub;Hong, Sunhag;Ku, Mino;Min, Dugki
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2013.10a
    • /
    • pp.513-516
    • /
    • 2013
  • In this paper, we present an information collecting system in medical information management domain. Our proposed system performs a systemized process, consisting of collection, transmission, and management, to develop intelligent medical information system and medical big data processing system. Our information collecting system consists of low-power biomedical sensors, biomedical information collecting devices, and storage systems. Currently, almost biomedical information of patients is collected manually by employees like nurses and medical doctors. Therefore, collected biometric data can be error-pronoun data. Since there is a lack to make big data of medical information, it is difficult to enhance the quality of medical services and researches. Accordingly, through our proposed system, we can overcome the problems like error-pronoun biometric data. In addition, we can extremely extend the area of collectable biometric data. Furthermore, using this system, we are able to make a real-time biomedical analysis system, like a real-time patient diagnosis system, and establish a strategy to against future medical markets changing rapidly.

  • PDF

Structuring of unstructured big data and visual interpretation (부산지역 교통관련 기사를 이용한 비정형 빅데이터의 정형화와 시각적 해석)

  • Lee, Kyeongjun;Noh, Yunhwan;Yoon, Sanggyeong;Cho, Youngseuk
    • Journal of the Korean Data and Information Science Society
    • /
    • v.25 no.6
    • /
    • pp.1431-1438
    • /
    • 2014
  • We analyzed the articles from "Kukje Shinmun" and "Busan Ilbo", which are two local newpapers of Busan Metropolitan City. The articles cover from January 1, 2013 to December 31, 2013. Meaningful pattern inherent in 2889 articles of which the title includes "Busan" and "Traffic" and related data was analyzed. Textmining method, which is a part of datamining, was used for the social network analysis (SNA). HDFS and MapReduce (from Hadoop ecosystem), which is open-source framework based on JAVA, were used with Linux environment (Uubntu-12.04LTS) for the construction of unstructured data and the storage, process and the analysis of big data. We implemented new algorithm that shows better visualization compared with the default one from R package, by providing the color and thickness based on the weight from each node and line connecting the nodes.

Dynamic Cluster Management of Hadoop Distributed Filesystem (하둡 분산 파일시스템의 동적 클러스터 관리 기법)

  • Ryu, Wooseok
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2016.10a
    • /
    • pp.435-437
    • /
    • 2016
  • Hadoop Distributed File System(HDFS) is a file system for distributed processing of big data by replicating data to distributed data nodes. HDFS cluster shows a great scalability up to thousands of nodes, but it assumes a exclusive node cluster with numerous nodes for the big data processing. Various operational-purpose worker systems used by office are hardly considered as a part of cluster. This paper discusses this problem and proposes a dynamic cluster management technique to increase storage capability and analytic performance of hadoop cluster. The propsed technique can add legacy systems to the cluster and can remove them from the cluster dynamically depending on their availability.

  • PDF

Design and Implementation of Efficient Storage and Retrieval Technology of Traffic Big Data (교통 빅데이터의 효율적 저장 및 검색 기술의 설계와 구현)

  • Kim, Ki-su;Yi, Jae-Jin;Kim, Hong-Hoi;Jang, Yo-lim;Hahm, Yu-Kun
    • The Journal of Bigdata
    • /
    • v.4 no.2
    • /
    • pp.207-220
    • /
    • 2019
  • Recent developments in information and communication technology has enabled the deployment of sensor based data to provide real-time services. In Korea, The Korea Transportation Safety Authority is collecting driving information of all commercial vehicles through a fitted digital tachograph (DTG). This information gathered using DTG can be utilized in various ways in the field of transportation. Notably in autonomous driving, the real-time analysis of this information can be used to prevent or respond to dangerous driving behavior. However, there is a limit to processing a large amount of data at a level suitable for real-time services using a traditional database system. In particular, due to a such technical problem, the processing of large quantity of traffic big data for real-time commercial vehicle operation information analysis has never been attempted in Korea. In order to solve this problem, this study optimized the new database server system and confirmed that a real-time service is possible. It is expected that the constructed database system will be used to secure base data needed to establish digital twin and autonomous driving environments.

  • PDF

Secure Authentication Protocol in Hadoop Distributed File System based on Hash Chain (해쉬 체인 기반의 안전한 하둡 분산 파일 시스템 인증 프로토콜)

  • Jeong, So Won;Kim, Kee Sung;Jeong, Ik Rae
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.23 no.5
    • /
    • pp.831-847
    • /
    • 2013
  • The various types of data are being created in large quantities resulting from the spread of social media and the mobile popularization. Many companies want to obtain valuable business information through the analysis of these large data. As a result, it is a trend to integrate the big data technologies into the company work. Especially, Hadoop is regarded as the most representative big data technology due to its terabytes of storage capacity, inexpensive construction cost, and fast data processing speed. However, the authentication token system of Hadoop Distributed File System(HDFS) for the user authentication is currently vulnerable to the replay attack and the datanode hacking attack. This can cause that the company secrets or the personal information of customers on HDFS are exposed. In this paper, we analyze the possible security threats to HDFS when tokens or datanodes are exposed to the attackers. Finally, we propose the secure authentication protocol in HDFS based on hash chain.

Smart Device based ECG Sensing IoT Applications (스마트 디바이스 기반 ECG 감지 IoT 응용 서비스에 관한 연구)

  • Mariappan, Vinayagam;Lee, Seungyoun;Lee, Junghoon;Lee, Juyoung;Cha, Jaesang
    • Journal of Satellite, Information and Communications
    • /
    • v.11 no.3
    • /
    • pp.18-23
    • /
    • 2016
  • Internet of things (IoT) is revolutionizing in the patient-Centered medical monitoring and management by authorizing the Smartphone application and data analysis with medical centers. The network connectivity is basic requirement to collect the observed human beings' health information from Smartphone to monitor the health from IoT medical devices in personal healthcare. The IoT environment built in Smartphone is very effective and does not demand infrastructure. This paper presents the smart phone deployed personal IoT architecture for Non-Invasive ECG Capturing. The adaptable IoT medical device cum Gateway is used for personal healthcare with big data storage on cloud configuration. In this approach, the Smartphone camera based imaging technique used to extract the personal ECG waveform and forward it to the cloud based big data storage connectivity using IoT architecture. Elaborated algorithm allows for efficient ECG registration directly from face image captured from Smartphone or Tablet camera. The profound technique may have an exceptional value in monitoring personal healthcare after adequate enhancements are introduced.

Sustainable Smart City Building-energy Management Based on Reinforcement Learning and Sales of ESS Power

  • Dae-Kug Lee;Seok-Ho Yoon;Jae-Hyeok Kwak;Choong-Ho Cho;Dong-Hoon Lee
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.17 no.4
    • /
    • pp.1123-1146
    • /
    • 2023
  • In South Korea, there have been many studies on efficient building-energy management using renewable energy facilities in single zero-energy houses or buildings. However, such management was limited due to spatial and economic problems. To realize a smart zero-energy city, studying efficient energy integration for the entire city, not just for a single house or building, is necessary. Therefore, this study was conducted in the eco-friendly energy town of Chungbuk Innovation City. Chungbuk successfully realized energy independence by converging new and renewable energy facilities for the first time in South Korea. This study analyzes energy data collected from public buildings in that town every minute for a year. We propose a smart city building-energy management model based on the results that combine various renewable energy sources with grid power. Supervised learning can determine when it is best to sell surplus electricity, or unsupervised learning can be used if there is a particular pattern or rule for energy use. However, it is more appropriate to use reinforcement learning to maximize rewards in an environment with numerous variables that change every moment. Therefore, we propose a power distribution algorithm based on reinforcement learning that considers the sales of Energy Storage System power from surplus renewable energy. Finally, we confirm through economic analysis that a 10% saving is possible from this efficiency.

A Study on the Analysis of Aviation Safety Data Structure and Standard Classification (항공안전데이터 구조 분석 및 표준 분류체계에 관한 연구)

  • Kim, Jun Hwan;Lim, Jae Jin;Lee, Jang Ryong
    • Journal of the Korean Society for Aviation and Aeronautics
    • /
    • v.28 no.4
    • /
    • pp.89-101
    • /
    • 2020
  • In order to enhance the safety of the international aviation industry, the International Civil Aviation Organization has recommended establishing an operational foundation for systematic and integrated collection, storage, analysis and sharing of aviation safety data. Accordingly, the Korea aviation industry also needs to comprehensively manage the safety data which generated and collected by various stakeholders related to aviation safety, and through this, it is necessary to previously identify and remove hazards that may cause accident. For more effective data management and utilization, a standard structure should be established to enable integrated management and sharing of safety data. Therefore, this study aims to propose the framework about how to manage and integrate the aviation safety data for big data-based aviation safety management and shared platform.

The Model of Network Packet Analysis based on Big Data (빅 데이터 기반의 네트워크 패킷 분석 모델)

  • Choi, Bomin;Kong, Jong-Hwan;Han, Myung-Mook
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.23 no.5
    • /
    • pp.392-399
    • /
    • 2013
  • Due to the development of IT technology and the information age, a dependency of the network over the most of our lives have grown to a greater extent. Although it provides us to get various useful information and service, it also has negative effectiveness that can provide network intruder with vulnerable roots. In other words, we need to urgently cope with theses serious security problem causing service disableness or system connected to network obstacle with exploiting various packet information. Many experts in a field of security are making an effort to develop the various security solutions to respond against these threats, but existing solutions have a lot of problems such as lack of storage capacity and performance degradation along with the massive increase of packet data volume. Therefore we propose the packet analysis model to apply issuing Big Data technology in the field of security. That is, we used NoSQL which is technology of massive data storage to collect the packet data growing massive and implemented the packet analysis model based on K-means clustering using MapReudce which is distributed programming framework, and then we have shown its high performance by experimenting.

Design and Implementation of Kernel-Level Split and Merge Operations for Efficient File Transfer in Cyber-Physical System (사이버 물리 시스템에서 효율적인 파일 전송을 위한 커널 레벨 분할 및 결합 연산의 설계와 구현)

  • Park, Hyunchan;Jang, Jun-Hee;Lee, Junseok
    • IEMEK Journal of Embedded Systems and Applications
    • /
    • v.14 no.5
    • /
    • pp.249-258
    • /
    • 2019
  • In the cyber-physical system, big data collected from numerous sensors and IoT devices is transferred to the Cloud for processing and analysis. When transferring data to the Cloud, merging data into one single file is more efficient than using the data in the form of split files. However, current merging and splitting operations are performed at the user-level and require many I / O requests to memory and storage devices, which is very inefficient and time-consuming. To solve this problem, this paper proposes kernel-level partitioning and combining operations. At the kernel level, splitting and merging files can be done with very little overhead by modifying the file system metadata. We have designed the proposed algorithm in detail and implemented it in the Linux Ext4 file system. In our experiments with the real Cloud storage system, our technique has achieved a transfer time of up to only 17% compared to the case of transferring split files. It also confirmed that the time required can be reduced by up to 0.5% compared to the existing user-level method.