• Title/Summary/Keyword: Big Data Environment

Search Result 976, Processing Time 0.029 seconds

Big Data Management System for Biomedical Images to Improve Short-term and Long-term Storage

  • Qamar, Shamweel;Kim, Eun Sung;Park, Peom
    • Journal of the Korean Society of Systems Engineering
    • /
    • v.15 no.2
    • /
    • pp.66-71
    • /
    • 2019
  • In digital pathology, an electronic system in the biomedical domain storage of the files is a big constrain and because all the analysis and annotation takes place at every user-end manually, it becomes even harder to manage the data that is being shared inside an enterprise. Therefore, we need such a storage system which is not only big enough to store all the data but also manage it and making communication of that data much easier without losing its true from. A virtual server setup is one of those techniques which can solve this issue. We set a main server which is the main storage for all the virtual machines(that are being used at user-end) and that main server is controlled through a hypervisor so that if we want to make changes in storage overall or the main server in itself, it could be reached remotely from anywhere by just using the server's IP address. The server in our case includes XML-RPC based API which are transmitted between computers using HTTP protocol. JAVA API connects to HTTP/HTTPS protocol through JAVA Runtime Environment and exists on top of other SDK web services for the productivity boost of the running application. To manage the server easily, we use Tkinter library to develop the GUI and pmw magawidgets library which is also utilized through Tkinter. For managing, monitoring and performing operations on virtual machines, we use Python binding to XML-RPC based API. After all these settings, we approach to make the system user friendly by making GUI of the main server. Using that GUI, user can perform administrative functions like restart, suspend or resume a virtual machine. They can also logon to the slave host of the pool in case of emergency and if needed, they can also filter virtual machine by the host. Network monitoring can be performed on multiple virtual machines at same time in order to detect any loss of network connectivity.

Comparison of Distributed and Parallel NGS Data Analysis Methods based on Cloud Computing

  • Kang, Hyungil;Kim, Sangsoo
    • International Journal of Contents
    • /
    • v.14 no.1
    • /
    • pp.34-38
    • /
    • 2018
  • With the rapid growth of genomic data, new requirements have emerged that are difficult to handle with big data storage and analysis techniques. Regardless of the size of an organization performing genomic data analysis, it is becoming increasingly difficult for an institution to build a computing environment for storing and analyzing genomic data. Recently, cloud computing has emerged as a computing environment that meets these new requirements. In this paper, we analyze and compare existing distributed and parallel NGS (Next Generation Sequencing) analysis based on cloud computing environment for future research.

Study on Web Services Middleware for Real-Time Monitoring in the IoT Environment

  • Shin, Seung-Hyeok
    • Journal of the Korea Society of Computer and Information
    • /
    • v.20 no.9
    • /
    • pp.97-104
    • /
    • 2015
  • Recently, the need for real-time systems which are providing various types of information that occur in large quantities in IoT environment is increasing. In this paper, we propose a middleware system which can monitor in real time on a web environment. The proposed system is designed to be integrated by using communication functions provided by a network operating system and external sensors. The proposed system is compared with an existing system and analysed by the server performance testing tool.

A Study on the Link Server Development Using B-Tree Structure in the Big Data Environment (빅데이터 환경에서의 B-tree 구조 기반 링크정보 관리서버의 개발)

  • Park, Sungbum;Hwang, Jong Sung;Lee, Sangwon
    • Journal of Internet Computing and Services
    • /
    • v.16 no.1
    • /
    • pp.75-82
    • /
    • 2015
  • Major corporations and portals have implemented a link server that connects Content Management Systems (CMS) to the physical address of content in a database (DB) to support efficient content use in web-based environments. In particular, a link server automatically connects the physical address of content in a DB to the content URL shown through a web browser screen, and re-connects the URL and the physical address when either is modified. In recent years, the number of users of digital content over the web has increased significantly because of the advent of the Big Data environment, which has also increased the number of link validity checks that should be performed in a CMS and a link server. If the link validity check is performed through an existing URL-based sequential method instead of petabyte or even etabyte environments, the identification rate of dead links decreases because of the degradation of validity check performance; moreover, frequent link checks add a large amount of workload to the DB. Hence, this study is aimed at providing a link server that can recognize URL link deletion or addition through analysis on the B-tree-based Information Identifier count per interval based on a large amount of URLs in order to resolve the existing problems. Through this study, the dead link check that is faster and adds lower loads than the existing method can be performed.

Development of Data Profiling Software Supporting a Microservice Architecture (마이크로 서비스 아키텍처를 지원하는 데이터 프로파일링 소프트웨어의 개발)

  • Chang, Jae-Young;Kim, Jihoon;Jee, Seowoo
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.21 no.5
    • /
    • pp.127-134
    • /
    • 2021
  • Recently, acquisition of high quality data has become an important issue as the expansion of the big data industry. In order to acquiring high quality data, accurate evaluation of data quality should be preceded first. The quality of data can be evaluated through meta-information such as statistics on data, and the task to extract such meta-information is called data profiling. Until now, data profiling software has typically been provided as a component or an additional service of traditional data quality or visualization tools. Hence, it was not suitable for utilizing directly in various environments. To address this problem, this paper presents the development result of data profiling software based on a microservice architecture that can be serviced in various environments. The presented data profiler provides an easy-to-use interface that requests of meta-information can be serviced through the restful API. Also, a proposed data profiler is independent of a specific environment, thus can be integrated efficiently with the various big data platforms or data analysis tools.

Analysis of Component Technology for Smart City Platform

  • Park, Chulsu;Cha, Jaesang
    • International Journal of Advanced Culture Technology
    • /
    • v.7 no.3
    • /
    • pp.143-148
    • /
    • 2019
  • In order to solve the urban problems caused by the increase of the urban population, the construction of smart city applying the latest technology is being carried out all over the world. In particular, we will create a smart city platform that utilizes data generated in the city to collect and store and analyze, thereby enhancing the city's continuous competitiveness and resilience and enhancing the quality of life of citizens. However, existing smart city platforms are not enough to construct a platform for smart city as a platform for solution elements such as IoT platform, big data platform, and AI platform. To complement this, we will reanalyze the existing overseas smart city platform and IoT platform in a comprehensive manner, combine the technical elements applied to it, and apply it to the future Korean smart city platform. This paper aims to investigate the trends of smart city platforms used in domestic and foreign countries and analyze the technology applied to smart city to study smart city platforms that solve various problems of the city such as environment, energy, safety, traffic, environment.

A Study on Anomaly Signal Detection and Management Model using Big Data (빅데이터를 활용한 이상 징후 탐지 및 관리 모델 연구)

  • Kwon, Young-baek;Kim, In-seok
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.16 no.6
    • /
    • pp.287-294
    • /
    • 2016
  • APT attack aimed at the interruption of information and communication facilities and important information leakage of companies. it performs an attack using zero-day vulnerabilities, social engineering base on collected information, such as IT infra, business environment, information of employee, for a long period of time. Fragmentary response to cyber threats such as malware signature detection methods can not respond to sophisticated cyber-attacks, such as APT attacks. In this paper, we propose a cyber intrusion detection model for countermeasure of APT attack by utilizing heterogeneous system log into big-data. And it also utilizes that merging pattern-based detection methods and abnormality detection method.

Application Plan of Graph Databases in the Big Data Environment (빅데이터환경에서의 그래프데이터베이스 활용방안)

  • Park, Sungbum;Lee, Sangwon;Ahn, Hyunsup;Jung, In-Hwan
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2013.10a
    • /
    • pp.247-249
    • /
    • 2013
  • Even though Relational Databases have been widely used in many enterprises, the relations among entities are not managed effectively and efficiently. In order to analyze Big Data, it is absolutely needed to express various relations among entities in a graphical form. In this paper, we define Graph Databases and its structure. And then, we check out their characteristics such as transaction, consistency, availability, retrieval function, and expandability. Also, we appropriate or inappropriate subjects for application of Graph Databases.

  • PDF

Compression-Friendly Low Power Test Application Based on Scan Slices Reusing

  • Wang, Weizheng;Wang, JinCheng;Cai, Shuo;Su, Wei;Xiang, Lingyun
    • JSTS:Journal of Semiconductor Technology and Science
    • /
    • v.16 no.4
    • /
    • pp.463-469
    • /
    • 2016
  • This paper presents a compression-friendly low power test scheme in EDT environment. The proposed approach exploits scan slices reusing to reduce the switching activity during shifting for test scheme based on linear decompressor. To avoid the impact on encoding efficiency from resulting control data, a counter is utilized to generate control signals. Experimental results obtained for some larger ISCAS'89 and ITC'99 benchmark circuits illustrate that the proposed test application scheme can improve significantly the encoding efficiency of linear decompressor.

A Study on the Calculation and Provision of Accruals-Quality by Big Data Real-Time Predictive Analysis Program

  • Shin, YeounOuk
    • International journal of advanced smart convergence
    • /
    • v.8 no.3
    • /
    • pp.193-200
    • /
    • 2019
  • Accruals-Quality(AQ) is an important proxy for evaluating the quality of accounting information disclosures. High-quality accounting information will provide high predictability and precision in the disclosure of earnings and will increase the response to stock prices. And high Accruals-Quality, such as mitigating heterogeneity in accounting information interpretation, provides information usefulness in capital markets. The purpose of this study is to suggest how AQ, which represents the quality of accounting information disclosure, is transformed into digitized data in real-time in combination with IT information technology and provided to financial analyst's information environment in real-time. And AQ is a framework for predictive analysis through big data log analysis system. This real-time information from AQ will help financial analysts to increase their activity and reduce information asymmetry. In addition, AQ, which is provided in real time through IT information technology, can be used as an important basis for decision-making by users of capital market information, and is expected to contribute in providing companies with incentives to voluntarily improve the quality of accounting information disclosure.