• Title/Summary/Keyword: data scalability

Search Result 574, Processing Time 0.028 seconds

Yet Another BGP Archive Forensic Analysis Tool Using Hadoop and Hive (하둡과 하이브를 이용한 BGP 아카이브 데이터의 포렌직 분석 툴)

  • Lee, Yeonhee;Lee, YoungSeok
    • Journal of KIISE
    • /
    • v.42 no.4
    • /
    • pp.541-549
    • /
    • 2015
  • A large volume of continuously growing BGP data files can raise two technical challenges regarding scalability and manageability. Due to the recent development of the open-source distributed computing infrastructure, Hadoop, it becomes feasible to handle a large amount of data in a scalable manner. In this paper, we present a new Hadoop-based BGP tool (BGPdoop) that provides the scale-out performance as well as the extensible and agile analysis capability. In particular, BGPdoop realizes a query-based BGP record exploration function using Hive on the partitioned BGP data structure, which enables flexible and versatile analytics of BGP archive files. From the experiments for the scalability with a Hadoop cluster of 20 nodes, we demonstrate that BGPdoop achieves 5 times higher performance and the user-defined analysis capability by expressing diverse BGP routing analytics in Hive queries.

A Study of Multipath Routing based on Software-Defined Networking for Data Center Networking in Cloud Computing Environments (클라우드 컴퓨팅 환경에서 데이터 센터 네트워킹을 위한 소프트웨어 정의 네트워킹 기반 다중 경로 라우팅 연구)

  • Kang, Yong-Hyeog
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2017.10a
    • /
    • pp.563-564
    • /
    • 2017
  • The core of the cloud computing technology is the data center in that the networking technology is important. Cloud data centers are comprised of tens or even hundreds of thousands of physical servers, so networking technology is required for high-speed data transfer. These networking technologies also require scalability, fault tolerance, and agility. For these requirements, many multi-path based schemes have been proposed. However, it was mainly used for load balancing of traffic and select a path randomly. In this paper, a scheme that can construct a multipath using software defined networking technology and transmit the traffic in parallel by using the multipath to achieve a fast transmission speed, solve the scalability problem and fault tolerance is proposed.

  • PDF

Scalable Blockchain Storage Model Based on DHT and IPFS

  • Chen, Lu;Zhang, Xin;Sun, Zhixin
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.7
    • /
    • pp.2286-2304
    • /
    • 2022
  • Blockchain is a distributed ledger that combines technologies such as cryptography, consensus mechanism, peer-to-peer transmission, and time stamping. The rapid development of blockchain has attracted attention from all walks of life, but storage scalability issues have hindered the application of blockchain. In this paper, a scalable blockchain storage model based on Distributed Hash Table (DHT) and the InterPlanetary File System (IPFS) was proposed. This paper introduces the current research status of the scalable blockchain storage model, as well as the basic principles of DHT and the InterPlanetary File System. The model construction and workflow are explained in detail. At the same time, the DHT network construction mechanism, block heat identification mechanism, new node initialization mechanism, and block data read and write mechanism in the model are described in detail. Experimental results show that this model can reduce the storage burden of nodes, and at the same time, the blockchain network can accommodate more local blocks under the same block height.

Motion-Compensated Layered Video Coding for Dynamic Adaptation (동적 적응을 위한 움직임 보상 계층형 동영상 부호화)

  • 이재용;박희라;고성제
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.24 no.10B
    • /
    • pp.1912-1920
    • /
    • 1999
  • In this paper, we propose a layered video coding scheme which can generate multi-layered bitstream for heterogeneous environments. A new motion prediction structure with temporal hierarchy of frames is developed to afford temporal resolution scalability and the wavelet decomposition is adopted to offer spatial acalability. The proposed scheme can have a higher compression ratio than replenishment schemes by using motion estimation and compensation which can further reduce the temporal redundancy, and it effectively works with dynamic adaption or errors using dispersive intra-subband update (DISU). Moreover, data rate scalability can be attained by employing embeded zerotree wavelet (EZW) technique which can produce embeded bitstream. Therefore, the proposed scheme is expected to be effectively used in heterogeneous environments such as the Internet, ATM, and mobile networks where interoperability are required.

  • PDF

Scalable P2P Botnet Detection with Threshold Setting in Hadoop Framework (하둡 프레임워크에서 한계점 가변으로 확장성이 가능한 P2P 봇넷 탐지 기법)

  • Huseynov, Khalid;Yoo, Paul D.;Kim, Kwangjo
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.25 no.4
    • /
    • pp.807-816
    • /
    • 2015
  • During the last decade most of coordinated security breaches are performed by the means of botnets, which is a large overlay network of compromised computers being controlled by remote botmaster. Due to high volumes of traffic to be analyzed, the challenge is posed by managing tradeoff between system scalability and accuracy. We propose a novel Hadoop-based P2P botnet detection method solving the problem of scalability and having high accuracy. Moreover, our approach is characterized not to require labeled data and applicable to encrypted traffic as well.

A linear systolic array based architecture for full-search block matching motion estimator (선형 시스토릭 어레이를 이용한 완전탐색 블럭정합 이동 예측기의 구조)

  • 김기현;이기철
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.21 no.2
    • /
    • pp.313-325
    • /
    • 1996
  • This paper presents a new architecture for full-search block-matching motion estimation. The architecture is based on linear systolic arrays. High speed operation is obtained by feeding reference data, search data, and control signals into the linear systolic array in a pipelined fashion. Input data are fed into the linear systolic array at a half of the processor speed, reducing the required data bandwidth to half. The proposed architecture has a good scalability with respect to the number of processors and input bandwidth when the size of reference block and search range change.

  • PDF

The Effect of the Perception of ICT Technical Characteristics in Agricultural Industry on the Intention to Start Smart Farm: Focusing on the Mediating Effects of Effort Expectation and Acceptance Intention of Smart Farm (농산업 ICT 기술적특성에 대한 인식이 스마트팜 창업의도에 미치는 영향: 스마트팜의 노력기대와 수용의도의 매개효과 중심으로)

  • Park, Sung Geun;Heo, Chul-Moo
    • Asia-Pacific Journal of Business Venturing and Entrepreneurship
    • /
    • v.15 no.3
    • /
    • pp.19-32
    • /
    • 2020
  • This study analyzed the effects of ICT technical characteristics of agricultural industry on smart farm entrepreneurial intention by using smart farm effort expectation and smart farm acceptance intention as mediators for smart farm pre-founders. Sub-variables of the technical characteristics of agricultural industry ICT were classified into availability, economics, data convergence and scalability. 349 questionnaires collected from pre-founders living in the country were used for empirical analysis. SPSS v22.0 and Process macro v3.4 were used to analyze the data based on serial multiple mediation model. First, economics and scalability had a positive (+) effect on start-up intention. Second, availability, economics and scalability had a significant effect on effort expectation. Third, effort expectation had a significant positive effect on acceptance intention. Fourth, acceptance intention had a significant positive effect on start-up intention. Fifth, the indirect effects of economics on start-up intention were all significant through effort expectation, through acceptance intention and through both effort expectation and acceptance intention. Sixth, the indirect effect of data convergence on start-up intention was significant through acceptance intention. Seventh, the indirect effect of scalability on start-up intention was significant through effort expectation and through both effort expectation and acceptance intention. As a follow-up study, it is necessary to study for the mediating variables other than mediators introduced in the study or the moderated mediation analysis through the conditional process model in which the moderating variable is introduced.

A Scalable Data Integrity Mechanism Based on Provable Data Possession and JARs

  • Zafar, Faheem;Khan, Abid;Ahmed, Mansoor;Khan, Majid Iqbal;Jabeen, Farhana;Hamid, Zara;Ahmed, Naveed;Bashir, Faisal
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.6
    • /
    • pp.2851-2873
    • /
    • 2016
  • Cloud storage as a service provides high scalability and availability as per need of user, without large investment on infrastructure. However, data security risks, such as confidentiality, privacy, and integrity of the outsourced data are associated with the cloud-computing model. Over the year's techniques such as, remote data checking (RDC), data integrity protection (DIP), provable data possession (PDP), proof of storage (POS), and proof of retrievability (POR) have been devised to frequently and securely check the integrity of outsourced data. In this paper, we improve the efficiency of PDP scheme, in terms of computation, storage, and communication cost for large data archives. By utilizing the capabilities of JAR and ZIP technology, the cost of searching the metadata in proof generation process is reduced from O(n) to O(1). Moreover, due to direct access to metadata, disk I/O cost is reduced and resulting in 50 to 60 time faster proof generation for large datasets. Furthermore, our proposed scheme achieved 50% reduction in storage size of data and respective metadata that result in providing storage and communication efficiency.

The Scalability and the Strategy for EMR Database Encryption Techniques

  • Shin, David;Sahama, Tony;Kim, Steve Jung-Tae;Kim, Ji-Hong
    • Journal of information and communication convergence engineering
    • /
    • v.9 no.5
    • /
    • pp.577-582
    • /
    • 2011
  • EMR(Electronic Medical Record) is an emerging technology that is highly-blended between non-IT and IT area. One of methodology to link non-IT and IT area is to construct databases. Nowadays, it supports before and after-treatment for patients and should satisfy all stakeholders such as practitioners, nurses, researchers, administrators and financial department and so on. In accordance with the database maintenance, DAS (Data as Service) model is one solution for outsourcing. However, there are some scalability and strategy issues when we need to plan to use DAS model properly. We constructed three kinds of databases such as plain-text, MS built-in encryption which is in-house model and custom AES (Advanced Encryption Standard) - DAS model scaling from 5K to 2560K records. To perform custom AES-DAS better, we also devised Bucket Index using Bloom Filter. The simulation showed the response times arithmetically increased in the beginning but after a certain threshold, exponentially increased in the end. In conclusion, if the database model is close to in-house model, then vendor technology is a good way to perform and get query response times in a consistent manner. If the model is DAS model, it is easy to outsource the database, however, some technique like Bucket Index enhances its utilization. To get faster query response times, designing database such as consideration of the field type is also important. This study suggests cloud computing would be a next DAS model to satisfy the scalability and the security issues.

A Real-time Service Recommendation System using Context Information in Pure P2P Environment (Pure P2P 환경에서 컨텍스트 정보를 이용한 실시간 서비스 추천 시스템)

  • Lee Se-Il;Lee Sang-Yong
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.15 no.7
    • /
    • pp.887-892
    • /
    • 2005
  • Under pure P2P environments, collaborative filtering must be provided with only a few service items by real time information without accumulated data. However, in case of collaborative filtering with only a few service items collected locally, quality of recommended service becomes low. Therefore, it is necessary to research a method to improve quality of recommended service by users' context information. But because a great volume of users' context information can be recognized in a moment, there can be a scalability problem and there are limitations in supporting differentiated services according to fields and items. In this paper, we solved the scalability problem by clustering context information Per each service field and classifying il per each user, using SOM. In addition, we could recommend proper services for users by measuring the context information of the users belonging to the similar classification to the service requester among classified data and then using collaborative filtering.