• Title/Summary/Keyword: Map reduce

Search Result 852, Processing Time 0.027 seconds

Implementation of a Raspberry-Pi-Sensor Network (라즈베리파이 센서 네트워크 구현)

  • Moon, Sangook
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2014.10a
    • /
    • pp.915-916
    • /
    • 2014
  • With the upcoming era of internet of things, the study of sensor network has been paid attention. Raspberry pi is a tiny versatile computer system which is able to act as a sensor node in hadoop cluster network. In this paper, we deployed 5 Raspberry pi's to construct an experimental testbed of hadoop sensor network with 5-node map-reduce hadoop software framework. We compared and analyzed the network architecture in terms of efficiency, resource management, and throughput using various parameters. We used a learning machine with support vector machine as test workload. In our experiments, Raspberry pi fulfilled the role of distributed computing sensor node in the sensor network.

  • PDF

Distributed Support Vector Machines for Localization on a Sensor Newtork (센서 네트워크에서 위치 측정을 위한 분산 지지 벡터 머신)

  • Moon, Sangook
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2014.10a
    • /
    • pp.944-946
    • /
    • 2014
  • Localization of a sensor network node using machine learning has been recently studied. It is easy for Support vector machines algorithm to implement in high level language enabling parallelism. In this paper, we realized Support vector machine using python language and built a sensor network cluster with 5 Pi's. We also established a Hadoop software framework to employ MapReduce mechanism. We modified the existing Support vector machine algorithm to fit into the distributed hadoop architecture system for localization of a sensor node. In our experiment, we implemented the test sensor network with a variety of parameters and examined based on proficiency, resource evaluation, and processing time.

  • PDF

A Study on the Estimation of Smartphone Movement Distance using Optical Flow Technology on a Limited Screen (제한된 화면에 광류 기술을 적용한 스마트폰 이동 거리 추정에 관한 연구)

  • Jung, Keunyoung;Oh, Jongtaek
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.19 no.4
    • /
    • pp.71-76
    • /
    • 2019
  • Research on indoor location tracking technology using smartphone is actively being carried out. Especially, the movement distance of the smartphone should be accurately measured and the movement route of the user should be displayed on the map. Location tracking technology using sensors mounted on smart phones has been used for a long time, but accuracy is not good enough to measure the moving distance of the user using only the sensor. Therefore, when the user moves the smartphone in a certain posture, it must research and develop an appropriate algorithm to measure the distance accurately. In this paper, we propose a method to reduce moving distance estimation error by removing user 's foot shape by limiting the screen of smartphone in pyramid - based optical flow estimation method.

Implementaion of Video Processing Framework using Hadoop-based cloud computing (Hadoop 기반 클라우드 컴퓨팅을 이용한 영상 처리 프레임워크 구현)

  • Ryu, Chungmo;Lee, Daecheol;Jang, Minwook;Kim, Cheolgi
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.11a
    • /
    • pp.139-142
    • /
    • 2013
  • 최근 대용량 영상데이터로부터 정보 수집, 영상 처리를 위한 클라우드 관련 연구들이 활발하다. 그러나 공개 소프트웨어를 이용한 클라우드 연구의 대부분은 라이브러리 수준이 아닌 단순히 프로그램 수준의 조합으로 작동한다. 이런 이유로 단순 조합에 따른 비효율성에 의한 성능문제는 크게 다루어지지 않는다. 본 논문에서는 이 비효율성을 해결하는데 중점을 두고 FFmpeg과 Hadoop을 라이브러리 수준으로 결합하여 기존보다 더 나은 성능의 영상클라우드 환경을 구축하였다. C기반의 영상처리 라이브러리인 FFmpeg와 JAVA기반의 클라우드 환경 Hadoop의 결합을 위해 JNI(Java Native Interface)를 이용하였다. 상세구현으로는 HDFS(Hadoop Distributed File System)을 확장하여 Hadoop MapReduce가 직접 FFmpeg을 통한 영상파일 접근이 가능하게 하였다. 이로써 FFmpeg과 Hadoop간 상이한 파일 접근 방식에서 발생하는 불필요한 작업에 의한 시스템의 성능저하를 막았다. 또한 응용의 확장성을 위해 영상작업시 작업영상을 영상처리의 최소단위인 GOP(Group of Pictures)단위로 잘라 클라우드의 노드들에게 분산시켰다. 결과적으로 기존에 존재하는 Hadoop과 FFmpeg을 프로그램적으로 결합한 영상처리 클라우드보다 총 처리시간을 앞당겼고, GOP 단위의 영상 처리는 영상기반 작업에 안정성과 응용의 확장성을 보장해주었다.

Big IoT Healthcare Data Analytics Framework Based on Fog and Cloud Computing

  • Alshammari, Hamoud;El-Ghany, Sameh Abd;Shehab, Abdulaziz
    • Journal of Information Processing Systems
    • /
    • v.16 no.6
    • /
    • pp.1238-1249
    • /
    • 2020
  • Throughout the world, aging populations and doctor shortages have helped drive the increasing demand for smart healthcare systems. Recently, these systems have benefited from the evolution of the Internet of Things (IoT), big data, and machine learning. However, these advances result in the generation of large amounts of data, making healthcare data analysis a major issue. These data have a number of complex properties such as high-dimensionality, irregularity, and sparsity, which makes efficient processing difficult to implement. These challenges are met by big data analytics. In this paper, we propose an innovative analytic framework for big healthcare data that are collected either from IoT wearable devices or from archived patient medical images. The proposed method would efficiently address the data heterogeneity problem using middleware between heterogeneous data sources and MapReduce Hadoop clusters. Furthermore, the proposed framework enables the use of both fog computing and cloud platforms to handle the problems faced through online and offline data processing, data storage, and data classification. Additionally, it guarantees robust and secure knowledge of patient medical data.

DEMO: Deep MR Parametric Mapping with Unsupervised Multi-Tasking Framework

  • Cheng, Jing;Liu, Yuanyuan;Zhu, Yanjie;Liang, Dong
    • Investigative Magnetic Resonance Imaging
    • /
    • v.25 no.4
    • /
    • pp.300-312
    • /
    • 2021
  • Compressed sensing (CS) has been investigated in magnetic resonance (MR) parametric mapping to reduce scan time. However, the relatively long reconstruction time restricts its widespread applications in the clinic. Recently, deep learning-based methods have shown great potential in accelerating reconstruction time and improving imaging quality in fast MR imaging, although their adaptation to parametric mapping is still in an early stage. In this paper, we proposed a novel deep learning-based framework DEMO for fast and robust MR parametric mapping. Different from current deep learning-based methods, DEMO trains the network in an unsupervised way, which is more practical given that it is difficult to acquire large fully sampled training data of parametric-weighted images. Specifically, a CS-based loss function is used in DEMO to avoid the necessity of using fully sampled k-space data as the label, thus making it an unsupervised learning approach. DEMO reconstructs parametric weighted images and generates a parametric map simultaneously by unrolling an interaction approach in conventional fast MR parametric mapping, which enables multi-tasking learning. Experimental results showed promising performance of the proposed DEMO framework in quantitative MR T1ρ mapping.

A parallel SNP detection algorithm for RNA-Seq data (RNA 시퀀싱 데이터를 이용한 병렬 SNP 추출 알고리즘)

  • Kim, Deok-Keun;Lee, Deok-Hae;Kong, Jin-Hwa;Lee, Un-Joo;Yoon, Jee-Hee
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2011.04a
    • /
    • pp.1260-1263
    • /
    • 2011
  • 최근 차세대 시퀀싱 (Next Generation Sequencing, NGS) 기술이 발전하면서 DNA, RNA 등의 시퀀싱 데이터를 이용한 유전체 분석 방식에 관한 연구가 활발히 이루어지고 있다. 차세대 시퀀싱 데이터를 이용한 유전체 분석 방식은 마이크로어레이 혹은 EST/cDNA 데이터를 이용한 기존의 분석 방식에 비하여 비용이 적게 들고 정확한 결과를 얻을 수 있다는 장점이 있다. 그러나 이 들 DNA, RNA 시퀀싱 데이터는 각 시퀀스의 길이가 짧고 전체 용량은 매우 커서 이 들 데이터로부터 정확한 분석 결과를 추출하는 데에 많은 어려움이 있다. 본 연구에서는 클라우드 컴퓨팅 기술을 기반으로 하여 대용량의 RNA 시퀀싱 데이터를 고속으로 처리하는 병렬 SNP 추출 알고리즘을 제안한다. 전체 게놈 데이터 중 유전자 영역만을 high coverage로 시퀀싱하여 얻어지는 RNA 시퀀싱 데이터는 유전자 변이 추출을 목적으로 분석되며, SNP(Single Nucleotide Polymorphism)와 같은 유전자 변이는 질병의 원인 규명 및 치료법 개발에 직접 이용된다. 제안된 알고리즘은 동시에 실행되는 다수의 Map/Reduce 함수에 의해서 대규모 RNA 시퀀스를 병렬로 처리하며, 레퍼런스 시퀀스에 매핑된 각 염기의 출현 빈도와 품질점수를 이용하여 SNP를 추출한다. 또한 이 들 SNP 추출 결과에 대한 시각적 분석 도구를 제공하여 SNP 추출 과정 및 근거를 시각적으로 확인/검증할 수 있도록 지원한다.

Image-based Extraction of Histogram Index for Concrete Crack Analysis

  • Kim, Bubryur;Lee, Dong-Eun
    • International conference on construction engineering and project management
    • /
    • 2022.06a
    • /
    • pp.912-919
    • /
    • 2022
  • The study is an image-based assessment that uses image processing techniques to determine the condition of concrete with surface cracks. The preparations of the dataset include resizing and image filtering to ensure statistical homogeneity and noise reduction. The image dataset is then segmented, making it more suited for extracting important features and easier to evaluate. The image is transformed into grayscale which removes the hue and saturation but retains the luminance. To create a clean edge map, the edge detection process is utilized to extract the major edge features of the image. The Otsu method is used to minimize intraclass variation between black and white pixels. Additionally, the median filter was employed to reduce noise while keeping the borders of the image. Image processing techniques are used to enhance the significant features of the concrete image, especially the defects. In this study, the tonal zones of the histogram and its properties are used to analyze the condition of the concrete. By examining the histogram, the viewer will be able to determine the information on the image through the number of pixels associated and each tonal characteristic on a graph. The features of the five tonal zones of the histogram which implies the qualities of the concrete image may be evaluated based on the quality of the contrast, brightness, highlights, shadow spikes, or the condition of the shadow region that corresponds to the foreground.

  • PDF

Rapid construction delivery of COVID-19 special hospital: Case study on Wuhan Huoshenshan hospital

  • Wang, Chen;Yu, Liangcheng;Kassem, Mukhtar A.;Li, Heng;Wang, Ziming
    • Advances in Computational Design
    • /
    • v.7 no.4
    • /
    • pp.345-369
    • /
    • 2022
  • Infectious disease emergency hospitals are usually temporarily built during the pneumonia epidemic with higher requirements regarding diagnosis and treatment efficiency, hygiene and safety, and infection control.This study aims to identify how the Building Information Modeling (BIM) + Industrialized Building System (IBS) approach could rapidly deliver an infectious disease hospital and develop site epidemic spreading algorithms. Coronavirus-19 pneumonia construction site spreading algorithm model mind map and block diagram of the construction site epidemic spreading algorithm model were developed. BIM+IBS approach could maximize the repetition of reinforced components and reduce the number of particular components. Huoshenshan Hospital adopted IBS and BIM in the construction, which reduced the workload of on-site operations and avoided later rectification. BIM+IBS integrated information on building materials, building planning, building participants, and construction machinery, and realized construction visualization control and parametric design. The delivery of Huoshenshan Hospital was during the most critical period of the Coronavirus-19 pneumonia epidemic. The development of a construction site epidemic spreading algorithm provided theoretical and numerical support for prevention. The agent-based analysis on hospital evacuation observed "arched" congestion formed at the evacuation exit, indicating behavioral blindness caused by fear in emergencies.

Comparison of the Performance of Clustering Analysis using Data Reduction Techniques to Identify Energy Use Patterns

  • Song, Kwonsik;Park, Moonseo;Lee, Hyun-Soo;Ahn, Joseph
    • International conference on construction engineering and project management
    • /
    • 2015.10a
    • /
    • pp.559-563
    • /
    • 2015
  • Identification of energy use patterns in buildings has a great opportunity for energy saving. To find what energy use patterns exist, clustering analysis has been commonly used such as K-means and hierarchical clustering method. In case of high dimensional data such as energy use time-series, data reduction should be considered to avoid the curse of dimensionality. Principle Component Analysis, Autocorrelation Function, Discrete Fourier Transform and Discrete Wavelet Transform have been widely used to map the original data into the lower dimensional spaces. However, there still remains an ongoing issue since the performance of clustering analysis is dependent on data type, purpose and application. Therefore, we need to understand which data reduction techniques are suitable for energy use management. This research aims find the best clustering method using energy use data obtained from Seoul National University campus. The results of this research show that most experiments with data reduction techniques have a better performance. Also, the results obtained helps facility managers optimally control energy systems such as HVAC to reduce energy use in buildings.

  • PDF