• ETRI Journal
• /
• v.42 no.5
• /
• pp.748-760
• /
• 2020

MapReduce is a framework that can process huge datasets in parallel and distributed computing environments. However, a single machine failure during the runtime of MapReduce tasks can increase completion time by 50%. MapReduce handles task failures by restarting the failed task and re-computing all input data from scratch, regardless of how much data had already been processed. To solve this issue, we need the computed key-value pairs to persist in a storage system to avoid re-computing them during the restarting process. In this paper, the task failure resilience (TFR) technique is proposed, which allows the execution of a failed task to continue from the point it was interrupted without having to redo all the work. Amazon ElastiCache for Redis is used as a non-volatile cache for the key-value pairs. We measured the performance of TFR by running different Hadoop benchmarking suites. TFR was implemented using the Hadoop software framework, and the experimental results showed significant performance improvements when compared with the performance of the default Hadoop implementation.

• International Journal of Internet, Broadcasting and Communication
• /
• v.12 no.4
• /
• pp.109-121
• /
• 2020

As a decrease in labor became a serious issue in the manufacturing industry, smart factory technology, which combines IT and the manufacturing business, began to attract attention as a solution. In this study, we have designed and implemented a real-time remote management system for smart factories, which is connected to an IoT sensor and gateway, for plastic manufacturing plants. By implementing the REST API in which an IoT sensor and smart gateway can communicate, the system enabled the data measured from the IoT sensor and equipment status data to the real-time monitoring system through the gateway. Also, a web-based management dashboard enabled remote monitoring and control of the equipment and raw material processing status. A comparative analysis experiment was conducted on the suggested system for the difference in processing speed based on equipment and measurement data number change. The experiment confirmed that saving equipment measurement data using cache mechanisim offered faster processing speed. Through the result our works can provide the basic framework to factory which need implement remote management system.

• KIPS Transactions on Computer and Communication Systems
• /
• v.10 no.10
• /
• pp.277-284
• /
• 2021

Recently, the smart home environment is expected to be a platform that collects, integrates, and utilizes various data through convergence with wireless information and communication technology. In fact, the number of smart devices with various sensors is increasing inside smart homes. The amount of data that needs to be processed by the increased number of smart devices is also increasing, and big data processing systems are actively being introduced to handle it effectively. However, traditional big data processing systems have all requests directed to cluster drivers before they are allocated to distributed nodes, leading to reduced cluster-wide performance sharing as cluster drivers managing segmentation tasks become bottlenecks. In particular, there is a greater delay rate on smart home devices that constantly request small data processing. Thus, in this paper, we design a Apriori-based big data system for effective data processing in smart home environments where frequent requests occur at the same time. According to the performance evaluation results of the proposed system, the data processing time was reduced by up to 38.6% from at least 19.2% compared to the existing system. The reason for this result is related to the type of data being measured. Because the amount of data collected in a smart home environment is large, the use of cache servers plays a major role in data processing, and association analysis with Apriori algorithms stores highly relevant sensor data in the cache.