• 한국빅데이터학회지
• /
• 제8권1호
• /
• pp.61-71
• /
• 2023

데이터 기반 의사결정 방법론, 고도화된 빅데이터 처리 기법의 발달로 데이터를 처리하는 방법에 대한 정보의 수요가 늘어나고 있다. 데이터를 활용하는 거의 모든 작업과 연구에서 데이터 전처리 과정이 포함되나, 이러한 과정은 주장하고자 하는 내용이나 결과물을 도출하기 위한 수단으로써 언급될 뿐 실질적인 과정에 대해서 자세하게 설명하고 있는 연구는 부족하였다. 실질적인 분석 기법을 활용하기 이전의 단계로 간단하게 언급되는 경우가 많아 데이터 처리에 대한 인사이트를 획득하기 어려운 경우가 많았다. 따라서 이 연구에서는, raw data에서부터 데이터를 처리하는 과정, 즉 데이터 처리 파이프라인에 대해서 자세하게 작성하고자 하였다. 특히 수입식품 수입 절차에 대한 설명을 구체화함으로써 해당 상황에서 데이터의 필드들이 어떻게 해석될 수 있고 어떠한 필드들을 왜 활용하게 되었는지에 대한 상황과 관련 도메인 지식을 공유하면서 흐름을 기술하고자 하였다.

• 한국산업융합학회 논문집
• /
• 제26권5호
• /
• pp.785-792
• /
• 2023

Because hyperspectral imaging provides detailed spectral information across a broad range of wavelengths, it can be utilized in numerous applications, including environmental monitoring, food quality inspection, medical diagnosis, material identification, art authentication, and crime scene analysis. However, hyperspectral images often contain various types of distortions due to the environmental conditions during image acquisition, which necessitates the proper removal of these distortions through a data preprocessing process. In this study, a preprocessing method was investigated to effectively correct the distortion caused by artificial light sources used in indoor hyperspectral imaging. For this purpose, a halogen-tungsten artificial light source was installed indoors, and hyperspectral images were acquired. The acquired images were then corrected for distortion using a preprocessing that does not require complex auxiliary equipment. After the corrections were made, the results were analyzed. According to the analysis, a statistical transformation technique using mean and standard deviation with reference to a reference signal was found to be the most effective in correcting distortions caused by artificial light sources.

• 비파괴검사학회지
• /
• 제22권4호
• /
• pp.361-368
• /
• 2002

본 연구는 초당 2개의 속도로 이송되는 사과를 대상으로 측정된 투과 에너지 스팩트럼 데이터를 이용하여 사과의 당도예측 모델을 개발하기 위해 각종 전처리가 당도 예측 모델의 정밀도에 미치는 영향을 구명하고, 신뢰성이 높은 당도 예측 회귀 모델을 개발하기 위해 수행되었다. 스펙트럼의 산란 보정, 노이즈 감소 등을 위해 1차미분, MSC, SNV, OSC 및 이들 조합으로 구성된 전처리 알고리즘을 프로그래밍하고, 이들 전처리를 스펙트럼데이터에 적용한 결과 특히 MSC SNV에 의해 각 파장에서의 투과에너지와 당도와의 상관관계가 전처리를 하지 않은 경우에 비해 현저히 증가하였다. 각종 전처리를 수행한 후 당도 예측 회귀 모델을 개발하고, 검정한 결과, 전처리 방법에 따라 예측모델의 SEP가 최대 1.265%brix 에서 최소 0.507%brix로 큰 차이를 나타내었다. 이는 SEP를 최소화하기 위해 주어진 스펙트럼 데이터의 특성에 알맞는 전처리 방법이 개발 또는 선택되어야 함을 의미한다. MSC 와 SNV는 예측 정밀도와 밀접한 관계가 있으며, OSC는 PLS의 factor 수와 관계되는 것으로 판단되었다. 1차미분은 오히려 모델의 예측 성능을 저하시키는 것으로 나타났다. 이는 실시간으로 측정된 투과스펙트럼에 상대적으로 노이즈 성분이 많이 포함되어 이들 성분이 미분에 의해 강조된 것으로 판단되었다. 본 연구에 사용된 스펙트럼 데이터의 경우 MSC와 OSC 전처리를 수행한 당도예측모델이 $R^2=0.8823$, SEP=0.5071%brix, bias=0.0327로 가장 우수하였다.

• 한국전자통신학회논문지
• /
• 제14권3호
• /
• pp.547-552
• /
• 2019

최근, 빅데이터 분야에서는 빅 데이터의 양적 팽창이 주요 이슈로 떠오르고 있다. 더군다나 이러한 빅데이터는 기계학습의 입력값으로 사용되어지고 있으며 이들의 성능을 향상시키기 위해 정규화 전처리가 필요하다. 이러한 성능은 빅데이터 컬럼의 범위나 정규화 전처리 방식에 따라 크게 좌우된다. 본 논문에서는 다양한 종류의 정규화 전처리 방식과 빅데이터 컬럼의 범위를 조절하면서 서포트벡터머신(SVM)의 기계학습방식에 적용함으로써 더욱 효과적인 정규화 전처리 방식을 파악하고자 하였다. 이를 위하여 파이썬언어와 주피터 노트북 환경에서 기계학습을 수행하고 분석하였다.

• 제어로봇시스템학회논문지
• /
• 제3권6호
• /
• pp.626-631
• /
• 1997

This paper represents the design of a real-time image preprocessing system. The preprocessing system performs hardware-wise mask operations and thresholding operations at the speed of camera output single rate. The preprocessing system consists of the preprocessing board and the main processing board. The preprocessing board includes preprocessing unit that includes a $5\times5$ mask processor and LUT, and can perform mask and threshold operations in real-time. To achieve high-resolution image input data($20485\timesn$), the preprocessing board has a linescan camera interface. The main processing board includes the image processor unit and main processor unit. The image processor unit is equipped with TI's TMS320C32 DSP and can perform image processing algorithms at high speed. The main processor unit controls the operation of total system. The proposed system is faster than the conventional CPU based system.

• 전기학회논문지
• /
• 제58권1호
• /
• pp.173-180
• /
• 2009

It is difficult to predict non-stationary or chaotic time series which includes the drift and/or the non-linearity as well as uncertainty. To solve it, we propose an effective prediction method which adopts data preprocessing and multiple model TS fuzzy predictors combined with model selection mechanism. In data preprocessing procedure, the candidates of the optimal difference interval are determined based on the correlation analysis, and corresponding difference data sets are generated in order to use them as predictor input instead of the original ones because the difference data can stabilize the statistical characteristics of those time series and better reveals their implicit properties. Then, TS fuzzy predictors are constructed for multiple model bank, where k-means clustering algorithm is used for fuzzy partition of input space, and the least squares method is applied to parameter identification of fuzzy rules. Among the predictors in the model bank, the one which best minimizes the performance index is selected, and it is used for prediction thereafter. Finally, the error compensation procedure based on correlation analysis is added to improve the prediction accuracy. Some computer simulations are performed to verify the effectiveness of the proposed method.

• 한국정보통신학회논문지
• /
• 제24권4호
• /
• pp.536-545
• /
• 2020

모바일 스마트 장치 배터리의 잔여 시간을 예측하기 위해 범용 통계적 회귀 기법을 적용한 경우, 배터리 잔량별 배터리 사용 시간의 편차가 커질수록 범용 통계적 회귀 기법의 예측 정확도가 낮아진다. 따라서 범용 통계적 회귀 기법의 예측 정확도를 향상시키기 위해서는 배터리 잔량별 배터리 사용 시간의 편차가 큰 원 측정 데이터를 가공 처리하여 정제된 데이터로 변환시키는 작업이 필요하다. 이에 본 논문에서는 원 측정 데이터를 정제된 데이터로 가공 처리하는 데이터 전처리 프레임워크를 제안하였다. 제안한 프레임워크를 통해 가공 처리하여 정제된 데이터를 범용 통계적 회귀 기법에 적용한 결과, 범용 통계적 회귀 기법의 예측 정확도가 향상됨을 확인하였다.

• 한국경영과학회지
• /
• 제24권1호
• /
• pp.1-11
• /
• 1999

We classified preprocessing methods into (1) analytic methods, (2) methods for removing implied free variables, (3) methods using pivot or elementary row operations, (4) methods for removing linearly dependent rows and columns and (5) methods for dense columns. We noted some considerations to which should be paid attention when preprocessing methods are applied to interior point methods for linear programming. We proposed an efficient order of preprocessing methods and data structures. We also noted the recovery process for dual solutions. We implemented the proposed preprocessing methods. and tested it with 28 large scale problems of NETLIB. We compared the results of it with those of preprocessing routines of HOPDM, BPDPM and CPLEX.

• Journal of Electrical Engineering and Technology
• /
• 제12권2호
• /
• pp.911-917
• /
• 2017

There are generally three folds when developing neural network classifiers. They are as follows: 1) discriminant function; 2) lots of parameters in the design of classifier; and 3) high dimensional training data. Along with this viewpoint, we propose space search optimized polynomial neural network classifier (PNNC) with the aid of data preprocessing technique and simultaneous tuning strategy, which is a balance optimization strategy used in the design of PNNC when running space search optimization. Unlike the conventional probabilistic neural network classifier, the proposed neural network classifier adopts two type of polynomials for developing discriminant functions. The overall optimization of PNNC is realized with the aid of so-called structure optimization and parameter optimization with the use of simultaneous tuning strategy. Space search optimization algorithm is considered as a optimize vehicle to help the implement both structure and parameter optimization in the construction of PNNC. Furthermore, principal component analysis and linear discriminate analysis are selected as the data preprocessing techniques for PNNC. Experimental results show that the proposed neural network classifier obtains better performance in comparison with some other well-known classifiers in terms of accuracy classification rate.

• 유통과학연구
• /
• 제16권12호
• /
• pp.95-104
• /
• 2018

Purpose - In order to gain the upper hand in competition between container terminals, efforts to improve container terminal productivity continue. Export containers arrive randomly in the container terminal and are carried in the container terminal yard according to the arrival order. On the other hand, containers are carried out of the container terminal yard in order based on container weight, not in order of arrival. Because the carry-in order and the carry-out order are different, rehandling may occur, which reduces the performance of the container terminals. In order to reduce rehandling number, containers can be moved in advance when they arrive, which is called preprocessing. This paper proposes an effective preprocessing algorithm and analyzes the factors that affect the productivity of the container terminals. It also provides a way to choose the best factors for preprocessing for a variety of situations. Research design, data, and methodology - To analyze the impact of factors affecting the performance of preprocessing algorithms presented in this paper, simulations are performed. The simulations are performed for two types of bays, 12 stacks with 8 tiers, and 8 stacks with 6 tiers. Results - The results of the factor analysis that affects the performance of the preprocessing algorithm were as follows. (1) As the LMF increased, preprocessing number increases and rehandling number decreased. (2) The LML effect was greatest when the LML changed from 0 to 1, and that the effect decreased when it changed above 1. (3) The sum of preprocessing number and rehandling number was then shown to be increased after decrease, as the LMF increased. (4) In the case of NCI, a decrease in NCI showed that the containers would become more grouped and thus the performance was improved. (5) There was a positive effect in the case of EFS. Conclusion - In this paper, preprocessing algorithm was proposed and it was possible to choose the best factors for preprocessing for a variety of situations through simulations. Further research related to this study needs to be carried out in the following topic : a study on the improvement of container performance by connecting the preprocessing with remarshalling.