• Title/Summary/Keyword: Large Size data Processing

Search Result 246, Processing Time 0.03 seconds

Deep learning approach to generate 3D civil infrastructure models using drone images

  • Kwon, Ji-Hye;Khudoyarov, Shekhroz;Kim, Namgyu;Heo, Jun-Haeng
    • Smart Structures and Systems
    • /
    • v.30 no.5
    • /
    • pp.501-511
    • /
    • 2022
  • Three-dimensional (3D) models have become crucial for improving civil infrastructure analysis, and they can be used for various purposes such as damage detection, risk estimation, resolving potential safety issues, alarm detection, and structural health monitoring. 3D point cloud data is used not only to make visual models but also to analyze the states of structures and to monitor them using semantic data. This study proposes automating the generation of high-quality 3D point cloud data and removing noise using deep learning algorithms. In this study, large-format aerial images of civilian infrastructure, such as cut slopes and dams, which were captured by drones, were used to develop a workflow for automatically generating a 3D point cloud model. Through image cropping, downscaling/upscaling, semantic segmentation, generation of segmentation masks, and implementation of region extraction algorithms, the generation of the point cloud was automated. Compared with the method wherein the point cloud model is generated from raw images, our method could effectively improve the quality of the model, remove noise, and reduce the processing time. The results showed that the size of the 3D point cloud model created using the proposed method was significantly reduced; the number of points was reduced by 20-50%, and distant points were recognized as noise. This method can be applied to the automatic generation of high-quality 3D point cloud models of civil infrastructures using aerial imagery.

High Efficiency Life Prediction and Exception Processing Method of NAND Flash Memory-based Storage using Gradient Descent Method (경사하강법을 이용한 낸드 플래시 메모리기반 저장 장치의 고효율 수명 예측 및 예외처리 방법)

  • Lee, Hyun-Seob
    • Journal of Convergence for Information Technology
    • /
    • v.11 no.11
    • /
    • pp.44-50
    • /
    • 2021
  • Recently, enterprise storage systems that require large-capacity storage devices to accommodate big data have used large-capacity flash memory-based storage devices with high density compared to cost and size. This paper proposes a high-efficiency life prediction method with slope descent to maximize the life of flash memory media that directly affects the reliability and usability of large enterprise storage devices. To this end, this paper proposes the structure of a matrix for storing metadata for learning the frequency of defects and proposes a cost model using metadata. It also proposes a life expectancy prediction policy in exceptional situations when defects outside the learned range occur. Lastly, it was verified through simulation that a method proposed by this paper can maximize its life compared to a life prediction method based on the fixed number of times and the life prediction method based on the remaining ratio of spare blocks, which has been used to predict the life of flash memory.

Fabrication Technology of the Focusing Grating Coupler using Single-step Electron Beam Lithography

  • Kim, Tae-Youb;Kim, Yark-Yeon;Han, Gee-Pyeong;Paek, Mun-Cheol;Kim, Hae-Sung;Lim, Byeong-Ok;Kim, Sung-Chan;Shin, Dong-Hoon;Rhee, Jin-Koo
    • Transactions on Electrical and Electronic Materials
    • /
    • v.3 no.1
    • /
    • pp.30-37
    • /
    • 2002
  • A focusing grating coupler (FGC) was not fabricated by the 'Continuous Path Control'writing strategy but by an electron-beam lithography system of more general exposure mode, which matches not only the address grid with the grating period but also an integer multiple of the address grid resolution (5 nm). To more simplify the fabrication, we are able to reduce a process step without large decrease of pattern quality by excluding a conducting material or layer such as metal (Al, Cr, Au), which are deposited on top or bottom of an e-beam resist to prevent charge build-up during e-beam exposure. A grating pitch period and an aperture feature size of the FGC designed and fabricated by e-beam lithography and reactive ion etching were ranged over 384.3 nm to 448.2 nm, and 0.5 $\times$ 0.5 mm$^2$area, respectively. This fabrication method presented will reduce processing time and improve the grating quality by means of a consideration of the address grid resolution, grating direction, pitch size and shapes when exposing. Here our investigations concentrate on the design and efficient fabrication results of the FGC for coupling from slab waveguide to a spot in free space.

A Pattern Matching Method of Large-Size Text Log Data using In-Memory Relational Database System (인메모리 관계형 데이터베이스 시스템을 이용한 대용량 텍스트 로그 데이터의 패턴 매칭 방법)

  • Han, Hyeok;Choi, Jae-Yong;Jin, Sung-Il
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2017.04a
    • /
    • pp.837-840
    • /
    • 2017
  • 각종 사이버 범죄가 증가함에 따라 실시간 모니터링을 통한 사전 탐지 기술뿐만 아니라, 사후 원인 분석을 통한 사고 재발 방지 기술의 중요성이 증가하고 있다. 사후 분석은 시스템에서 생산된 다양한 유형의 대용량 로그를 기반으로 분석가가 보안 위협 과정을 규명하는 것으로 이를 지원하는 다양한 상용 및 오픈 소스 SW 존재하나, 대부분 단일 분석가 PC에서 운용되는 파일 기반 SW로 대용량 데이터에 대한 분석 성능 저하, 다수 분석가 간의 데이터 공유 불가, 통계 연관 분석 한계 및 대화형 점진적 내용 분석 불가 등의 문제점을 해결하지 못하고 있다. 이러한 문제점을 해결하기 위하여 고성능 인메모리 관계형 데이터베이스 시스템을 로그 스토리지로 활용하는 대용량 로그 분석 SW 개발하였다. 특히, 기 확보된 공격자 프로파일을 활용하여 공격의 유무를 확인하는 텍스트 패턴 매칭 연산은 전통적인 관계형 데이터베이스 시스템의 FTS(Full-Text Search) 기능 활용이 가능하나, 대용량 전용 색인 생성에 따른 비현실적인 DB 구축 소요 시간과 최소 3배 이상의 DB 용량 증가로 인한 시스템 리소스 추가 요구 등의 단점이 있다. 본 논문에서는 인메모리 관계형 데이터베이스 시스템 기반 효율적인 텍스트 패턴 매칭 연산을 위하여, 고성능의 대용량 로그 DB 적재 방법과 새로운 유형의 패턴 매칭 방법을 제안하였다.

Design of a Vido Storage Server that Maximizes Concurrent Streams and Minimizes Initial Latency (사용자 수 증대와 초기 대기시간 감소를 위한 비디오 저장 서버의 설계)

  • Ma, Pyeong-Su;Jo, Chang-Sik;Jin, Yun-Suk;Sin, Gyu-Sang
    • The Transactions of the Korea Information Processing Society
    • /
    • v.6 no.10
    • /
    • pp.2608-2617
    • /
    • 1999
  • One of the most important functionality that commercial video storage servers should provide is to maximize the number of concurrent streams and to minimize the initial latency of new requests. In this paper, we propose a data placement scheme whose disk read unit size can be twice large than that of conventional striping methods. The proposed scheme can significantly increase the number of concurrent streams, since the ratio of rotational latency time is decreased and the disks are effectively utilized. The disk scheduling scheme we propose guarantees constant initial latency time. We also propose a procedural design method for a storage server by introducing the concept of allowed initial latency. The comparison with previous research shows that the proposed scheme provides better performance.

  • PDF

A Comparative Analysis of Music Similarity Measures in Music Information Retrieval Systems

  • Gurjar, Kuldeep;Moon, Yang-Sae
    • Journal of Information Processing Systems
    • /
    • v.14 no.1
    • /
    • pp.32-55
    • /
    • 2018
  • The digitization of music has seen a considerable increase in audience size from a few localized listeners to a wider range of global listeners. At the same time, the digitization brings the challenge of smoothly retrieving music from large databases. To deal with this challenge, many systems which support the smooth retrieval of musical data have been developed. At the computational level, a query music piece is compared with the rest of the music pieces in the database. These systems, music information retrieval (MIR systems), work for various applications such as general music retrieval, plagiarism detection, music recommendation, and musicology. This paper mainly addresses two parts of the MIR research area. First, it presents a general overview of MIR, which will examine the history of MIR, the functionality of MIR, application areas of MIR, and the components of MIR. Second, we will investigate music similarity measurement methods, where we provide a comparative analysis of state of the art methods. The scope of this paper focuses on comparative analysis of the accuracy and efficiency of a few key MIR systems. These analyses help in understanding the current and future challenges associated with the field of MIR systems and music similarity measures.

On-Demand Remote Software Code Execution Unit Using On-Chip Flash Memory Cloudification for IoT Environment Acceleration

  • Lee, Dongkyu;Seok, Moon Gi;Park, Daejin
    • Journal of Information Processing Systems
    • /
    • v.17 no.1
    • /
    • pp.191-202
    • /
    • 2021
  • In an Internet of Things (IoT)-configured system, each device executes on-chip software. Recent IoT devices require fast execution time of complex services, such as analyzing a large amount of data, while maintaining low-power computation. As service complexity increases, the service requires high-performance computing and more space for embedded space. However, the low performance of IoT edge devices and their small memory size can hinder the complex and diverse operations of IoT services. In this paper, we propose a remote on-demand software code execution unit using the cloudification of on-chip code memory to accelerate the program execution of an IoT edge device with a low-performance processor. We propose a simulation approach to distribute remote code executed on the server side and on the edge side according to the program's computational and communicational needs. Our on-demand remote code execution unit simulation platform, which includes an instruction set simulator based on 16-bit ARM Thumb instruction set architecture, successfully emulates the architectural behavior of on-chip flash memory, enabling embedded devices to accelerate and execute software using remote execution code in the IoT environment.

A Cell-based Indexing for Managing Current Location Information of Moving Objects (이동객체의 현재 위치정보 관리를 위한 셀 기반 색인 기법)

  • Lee, Eung-Jae;Lee, Yang-Koo;Ryu, Keun-Ho
    • The KIPS Transactions:PartD
    • /
    • v.11D no.6
    • /
    • pp.1221-1230
    • /
    • 2004
  • In mobile environments, the locations of moving objects such as vehicles, airplanes and users of wireless devices continuously change over time. For efficiently processing moving object information, the database system should be able to deal with large volume of data, and manage indexing efficiently. However, previous research on indexing method mainly focused on query performance, and did not pay attention to update operation for moving objects. In this paper, we propose a novel moving object indexing method, named ACAR-Tree. For processing efficiently frequently updating of moving object location information as well as query performance, the proposed method is based on fixed grid structure with auxiliary R-Tree. This hybrid structure is able to overcome the poor update performance of R-Tree which is caused by reorganizing of R-Tree. Also, the proposed method is able to efficiently deal with skewed-. or gaussian distribution of data using auxiliary R-Tree. The experimental results using various data size and distribution of data show that the proposed method has reduced the size of index and improve the update and query performance compared with R-Tree indexing method.

A Kinematic Approach to Answering Similarity Queries on Complex Human Motion Data (운동학적 접근 방법을 사용한 복잡한 인간 동작 질의 시스템)

  • Han, Hyuck;Kim, Shin-Gyu;Jung, Hyung-Soo;Yeom, Heon-Y.
    • Journal of Internet Computing and Services
    • /
    • v.10 no.4
    • /
    • pp.1-11
    • /
    • 2009
  • Recently there has arisen concern in both the database community and the graphics society about data retrieval from large motion databases because the high dimensionality of motion data implies high costs. In this circumstance, finding an effective distance measure and an efficient query processing method for such data is a challenging problem. This paper presents an elaborate motion query processing system, SMoFinder (Similar Motion Finder), which incorporates a novel kinematic distance measure and an efficient indexing strategy via adaptive frame segmentation. To this end, we regard human motions as multi-linkage kinematics and propose the weighted Minkowski distance metric. For efficient indexing, we devise a new adaptive segmentation method that chooses representative frames among similar frames and stores chosen frames instead of all frames. For efficient search, we propose a new search method that processes k-nearest neighbors queries over only representative frames. Our experimental results show that the size of motion databases is reduced greatly (${\times}1/25$) but the search capability of SMoFinder is equal to or superior to that of other systems.

  • PDF

Classification of Raccoon dog and Raccoon with Transfer Learning and Data Augmentation (전이 학습과 데이터 증강을 이용한 너구리와 라쿤 분류)

  • Dong-Min Park;Yeong-Seok Jo;Seokwon Yeom
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.24 no.1
    • /
    • pp.34-41
    • /
    • 2023
  • In recent years, as the range of human activities has increased, the introduction of alien species has become frequent. Among them, raccoons have been designated as harmful animals since 2020. Raccoons are similar in size and shape to raccoon dogs, so they generally need to be distinguished in capturing them. To solve this problem, we use VGG19, ResNet152V2, InceptionV3, InceptionResNet and NASNet, which are CNN deep learning models specialized for image classification. The parameters to be used for learning are pre-trained with a large amount of data, ImageNet. In order to classify the raccoon and raccoon dog datasets as outward features of animals, the image was converted to grayscale and brightness was normalized. Augmentation methods were applied using left and right inversion, rotation, scaling, and shift to create sufficient data for transfer learning. The FCL consists of 1 layer for the non-augmented dataset while 4 layers for the augmented dataset. Comparing the accuracy of various augmented datasets, the performance increased as more augmentation methods were applied.