• Title/Summary/Keyword: imbalanced data

Search Result 151, Processing Time 0.021 seconds

Development of Prediction Model of Financial Distress and Improvement of Prediction Performance Using Data Mining Techniques (데이터마이닝 기법을 이용한 기업부실화 예측 모델 개발과 예측 성능 향상에 관한 연구)

  • Kim, Raynghyung;Yoo, Donghee;Kim, Gunwoo
    • Information Systems Review
    • /
    • v.18 no.2
    • /
    • pp.173-198
    • /
    • 2016
  • Financial distress can damage stakeholders and even lead to significant social costs. Thus, financial distress prediction is an important issue in macroeconomics. However, most existing studies on building a financial distress prediction model have only considered idiosyncratic risk factors without considering systematic risk factors. In this study, we propose a prediction model that considers both the idiosyncratic risk based on a financial ratio and the systematic risk based on a business cycle. Ultimately, we build several IT artifacts associated with financial ratio and add them to the idiosyncratic risk factors as well as address the imbalanced data problem by using an oversampling technique and synthetic minority oversampling technique (SMOTE) to ensure good performance. When considering systematic risk, our study ensures that each data set consists of both financially distressed companies and financially sound companies in each business cycle phase. We conducted several experiments that change the initial imbalanced sample ratio between the two company groups into a 1:1 sample ratio using SMOTE and compared the prediction results from the individual data set. We also predicted data sets from the subsequent business cycle phase as a test set through a built prediction model that used business contraction phase data sets, and then we compared previous prediction performance and subsequent prediction performance. Thus, our findings can provide insights into making rational decisions for stakeholders that are experiencing an economic crisis.

Decision Tree Induction with Imbalanced Data Set: A Case of Health Insurance Bill Audit in a General Hospital (불균형 데이터 집합에서의 의사결정나무 추론: 종합 병원의 건강 보험료 청구 심사 사례)

  • Hur, Joon;Kim, Jong-Woo
    • Information Systems Review
    • /
    • v.9 no.1
    • /
    • pp.45-65
    • /
    • 2007
  • In medical industry, health insurance bill audit is unique and essential process in general hospitals. The health insurance bill audit process is very important because not only for hospital's profit but also hospital's reputation. Particularly, at the large general hospitals many related workers including analysts, nurses, and etc. have engaged in the health insurance bill audit process. This paper introduces a case of health insurance bill audit for finding reducible health insurance bill cases using decision tree induction techniques at a large general hospital in Korea. When supervised learning methods had been tried to be applied, one of major problems was data imbalance problem in the health insurance bill audit data. In other words, there were many normal(passing) cases and relatively small number of reduction cases in a bill audit dataset. To resolve the problem, in this study, well-known methods for imbalanced data sets including over sampling of rare cases, under sampling of major cases, and adjusting the misclassification cost are combined in several ways to find appropriate decision trees that satisfy required conditions in health insurance bill audit situation.

Comparison of resampling methods for dealing with imbalanced data in binary classification problem (이분형 자료의 분류문제에서 불균형을 다루기 위한 표본재추출 방법 비교)

  • Park, Geun U;Jung, Inkyung
    • The Korean Journal of Applied Statistics
    • /
    • v.32 no.3
    • /
    • pp.349-374
    • /
    • 2019
  • A class imbalance problem arises when one class outnumbers the other class by a large proportion in binary data. Studies such as transforming the learning data have been conducted to solve this imbalance problem. In this study, we compared resampling methods among methods to deal with an imbalance in the classification problem. We sought to find a way to more effectively detect the minority class in the data. Through simulation, a total of 20 methods of over-sampling, under-sampling, and combined method of over- and under-sampling were compared. The logistic regression, support vector machine, and random forest models, which are commonly used in classification problems, were used as classifiers. The simulation results showed that the random under sampling (RUS) method had the highest sensitivity with an accuracy over 0.5. The next most sensitive method was an over-sampling adaptive synthetic sampling approach. This revealed that the RUS method was suitable for finding minority class values. The results of applying to some real data sets were similar to those of the simulation.

A Study on the Characteristics in Dietary Behavior and Dish Preference of Elementary School Children in Seoul and Kangwha Area

  • Lee, Sim-Yeol;Kang, In-Soo
    • Journal of Community Nutrition
    • /
    • v.3 no.2
    • /
    • pp.69-76
    • /
    • 2001
  • This study was conducted to provide current information on dietary behaviors and dish preferences of elementary school children and to suggest guidelines for proper dietary behaviors. To accomplish study objectives survey was executed using the questionnaire for 420 fifth and sixth grade school children, chosen from schools in Kangwha-gun and East River District of Seoul. A questionnaire largely consists of categories including general characteristics, dietary behavior and preferences of the subjects for some dishes. Results showed 30% of the subjects had breakfast irregularly. A majority of the subjects took Korean style dishes of cooked rice and soup for breakfast. fifty-eight percent of subjects had a regular meal time. Imbalanced diet(avoiding specific flood group thereby causing unbalance in nutrient intake) habit group was estimated to be 47.3%. Twenty one percent had a habit of overeating. In choosing the snack, taste was considered to be a more important factor than nutrition. Advertisement of the snack was shown to be one of the Important factors in selecting the snack. The girls were more concerned about weight control than the boys. Also the girls were likely to rely on the diet to control weight since they exercised only in the physical c1ass while the boys exercised regularly. Generally, children liked animal protein containing foods and the preference for vegetables was low. In order to improve overall dietary behavior, systematic nutrition education programs reflecting sex difference should be developed. Dish preference data would be very useful in selecting substitutive dish for the s[hoof lunch menu to improve imbalanced diet. (J Community Nutrition3(2) : 69∼76, 2001)

  • PDF

An Analytical Study on Automatic Classification of Domestic Journal articles Using Random Forest (랜덤포레스트를 이용한 국내 학술지 논문의 자동분류에 관한 연구)

  • Kim, Pan Jun
    • Journal of the Korean Society for information Management
    • /
    • v.36 no.2
    • /
    • pp.57-77
    • /
    • 2019
  • Random Forest (RF), a representative ensemble technique, was applied to automatic classification of journal articles in the field of library and information science. Especially, I performed various experiments on the main factors such as tree number, feature selection, and learning set size in terms of classification performance that automatically assigns class labels to domestic journals. Through this, I explored ways to optimize the performance of random forests (RF) for imbalanced datasets in real environments. Consequently, for the automatic classification of domestic journal articles, Random Forest (RF) can be expected to have the best classification performance when using tree number interval 100~1000(C), small feature set (10%) based on chi-square statistic (CHI), and most learning sets (9-10 years).

Abnormal signal detection based on parallel autoencoders (병렬 오토인코더 기반의 비정상 신호 탐지)

  • Lee, Kibae;Lee, Chong Hyun
    • The Journal of the Acoustical Society of Korea
    • /
    • v.40 no.4
    • /
    • pp.337-346
    • /
    • 2021
  • Detection of abnormal signal generally can be done by using features of normal signals as main information because of data imbalance. This paper propose an efficient method for abnormal signal detection using parallel AutoEncoder (AE) which can use features of abnormal signals as well. The proposed Parallel AE (PAE) is composed of a normal and an abnormal reconstructors having identical AE structure and train features of normal and abnormal signals, respectively. The PAE can effectively solve the imbalanced data problem by sequentially training normal and abnormal data. For further detection performance improvement, additional binary classifier can be added to the PAE. Through experiments using public acoustic data, we obtain that the proposed PAE shows Area Under Curve (AUC) improvement of minimum 22 % at the expenses of training time increased by 1.31 ~ 1.61 times to the single AE. Furthermore, the PAE shows 93 % AUC improvement in detecting abnormal underwater acoustic signal when pre-trained PAE is transferred to train open underwater acoustic data.

Image Classification using Class-Balanced Loss (Class-Balanced Loss를 이용한 이미지 분류)

  • Jihee Park;Wonjun Hwang
    • Proceedings of the Korean Society of Broadcast Engineers Conference
    • /
    • 2022.11a
    • /
    • pp.164-166
    • /
    • 2022
  • Long-tail problem은 class 별로 sample의 개수에 차이가 있어 성능에 안 좋은 영향을 미치는 것을 말한다. 본 논문에서는 cost-sensitive learning 중 Class-Balanced Loss를 이용해 성능을 개선하여 Long-tail problem을 해결하려고 한다. 먼저, balanced data set과 imbalanced data set의 성능 차이를 살펴보도록 할 것이다. 그 후, Class-Balanced Loss를 3가지 버전으로 이용해 그 성능을 측정하고 분석해 볼 것이다.

  • PDF

Study on Detection Technique for Cochlodinium polykrikoides Red tide using Logistic Regression Model under Imbalanced Data (불균형 데이터 환경에서 로지스틱 회귀모형을 이용한 Cochlodinium polykrikoides 적조 탐지 기법 연구)

  • Bak, Su-Ho;Kim, Heung-Min;Kim, Bum-Kyu;Hwang, Do-Hyun;Enkhjargal, Unuzaya;Yoon, Hong-Joo
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.13 no.6
    • /
    • pp.1353-1364
    • /
    • 2018
  • This study proposed a method to detect Cochlodinium polykrikoides red tide pixels in satellite images using a logistic regression model of machine learning technique under Imbalanced data. The spectral profiles extracted from red tide, clear water, and turbid water were used as training dataset. 70% of the entire data set was extracted and used for as model training, and the classification accuracy of the model was evaluated using the remaining 30%. At this time, the white noise was added to the spectral profile of the red tide, which has a relatively small number of data compared to the clear water and the turbid water, and over-sampling was performed to solve the unbalanced data problem. As a result of the accuracy evaluation, the proposed algorithm showed about 94% classification accuracy.

Comparison of Machine Learning Methodology in COPD Cohort Data (COPD 코호트 자료에서의 Machine Learning 방법론 비교)

  • Jeong, Hyeon-Myeong;Park, Heon-Jin;Rhee, Chin-Kook;Lee, Jong-min
    • The Journal of Bigdata
    • /
    • v.2 no.2
    • /
    • pp.115-128
    • /
    • 2017
  • Recently, Machine Learning Methods are widely used with high prediction performance. But if the limit of the data is solved by the statistical technique, It can, lead to higher prediction performance than the existing one. In this study, the SMOTE method is used to solve the imbalance problem in the longitudinal and imbalanced data. As a result, It, was confirmed that the prediction performance increases. Additionally, Although, studies on COPD have been actively conducted, only studies that are related to acute exacerbation have been conducted. So there are no studies on the prediction of acute exacerbation through multiple perspectives and predictive models for various factors. In this study, We examined the factors related to acute exacerbation of COPD and constructed a personalized specific disease prediction model.

  • PDF

Extraction Method of Significant Clinical Tests Based on Data Discretization and Rough Set Approximation Techniques: Application to Differential Diagnosis of Cholecystitis and Cholelithiasis Diseases (데이터 이산화와 러프 근사화 기술에 기반한 중요 임상검사항목의 추출방법: 담낭 및 담석증 질환의 감별진단에의 응용)

  • Son, Chang-Sik;Kim, Min-Soo;Seo, Suk-Tae;Cho, Yun-Kyeong;Kim, Yoon-Nyun
    • Journal of Biomedical Engineering Research
    • /
    • v.32 no.2
    • /
    • pp.134-143
    • /
    • 2011
  • The selection of meaningful clinical tests and its reference values from a high-dimensional clinical data with imbalanced class distribution, one class is represented by a large number of examples while the other is represented by only a few, is an important issue for differential diagnosis between similar diseases, but difficult. For this purpose, this study introduces methods based on the concepts of both discernibility matrix and function in rough set theory (RST) with two discretization approaches, equal width and frequency discretization. Here these discretization approaches are used to define the reference values for clinical tests, and the discernibility matrix and function are used to extract a subset of significant clinical tests from the translated nominal attribute values. To show its applicability in the differential diagnosis problem, we have applied it to extract the significant clinical tests and its reference values between normal (N = 351) and abnormal group (N = 101) with either cholecystitis or cholelithiasis disease. In addition, we investigated not only the selected significant clinical tests and the variations of its reference values, but also the average predictive accuracies on four evaluation criteria, i.e., accuracy, sensitivity, specificity, and geometric mean, during l0-fold cross validation. From the experimental results, we confirmed that two discretization approaches based rough set approximation methods with relative frequency give better results than those with absolute frequency, in the evaluation criteria (i.e., average geometric mean). Thus it shows that the prediction model using relative frequency can be used effectively in classification and prediction problems of the clinical data with imbalanced class distribution.