• Title/Summary/Keyword: Data quality analysis

Search Result 9,029, Processing Time 0.041 seconds

Saliency Score-Based Visualization for Data Quality Evaluation

  • Kim, Yong Ki;Lee, Keon Myung
    • International Journal of Fuzzy Logic and Intelligent Systems
    • /
    • v.15 no.4
    • /
    • pp.289-294
    • /
    • 2015
  • Data analysts explore collections of data to search for valuable information using various techniques and tricks. Garbage in, garbage out is a well-recognized idiom that emphasizes the importance of the quality of data in data analysis. It is therefore crucial to validate the data quality in the early stage of data analysis, and an effective method of evaluating the quality of data is hence required. In this paper, a method to visually characterize the quality of data using the notion of a saliency score is introduced. The saliency score is a measure comprising five indexes that captures certain aspects of data quality. Some experiment results are presented to show the applicability of proposed method.

An Analysis Method of Superlarge Manufacturing Process Data Using Data Cleaning and Graphical Analysis (데이터 정제와 그래프 분석을 이용한 대용량 공정데이터 분석 방법)

  • 박재홍;변재현
    • Journal of Korean Society for Quality Management
    • /
    • v.30 no.2
    • /
    • pp.72-85
    • /
    • 2002
  • Advances in computer and sensor technology have made it possible to obtain superlarge manufacturing process data in real time, letting us extract meaningful information from these superlarge data sets. We propose a systematic data analysis procedure which field engineers can apply easily to manufacture quality products. The procedure consists of data cleaning and data analysis stages. Data cleaning stage is to construct a database suitable for statistical analysis from the original superlarge manufacturing process data. In the data analysis stage, we suggest a graphical easy-to-implement approach to extract practical information from the cleaned database. This study will help manufacturing companies to achieve six sigma quality.

IMPROVING SOCIAL MEDIA DATA QUALITY FOR EFFECTIVE ANALYTICS: AN EMPIRICAL INVESTIGATION BASED ON E-BDMS

  • B. KARTHICK;T. MEYYAPPAN
    • Journal of applied mathematics & informatics
    • /
    • v.41 no.5
    • /
    • pp.1129-1143
    • /
    • 2023
  • Social media platforms have become an integral part of our daily lives, and they generate vast amounts of data that can be analyzed for various purposes. However, the quality of the data obtained from social media is often questionable due to factors such as noise, bias, and incompleteness. Enhancing data quality is crucial to ensure the reliability and validity of the results obtained from such data. This paper proposes an enhanced decision-making framework based on Business Decision Management Systems (BDMS) that addresses these challenges by incorporating a data quality enhancement component. The framework includes a backtracking method to improve plan failures and risk-taking abilities and a steep optimized strategy to enhance training plan and resource management, all of which contribute to improving the quality of the data. We examine the efficacy of the proposed framework through research data, which provides evidence of its ability to increase the level of effectiveness and performance by enhancing data quality. Additionally, we demonstrate the reliability of the proposed framework through simulation analysis, which includes true positive analysis, performance analysis, error analysis, and accuracy analysis. This research contributes to the field of business intelligence by providing a framework that addresses critical data quality challenges faced by organizations in decision-making environments.

A Study on the Big Data Analysis and Predictive Models for Quality Issues in Defense C5ISR (국방 C5ISR 분야 품질문제의 빅데이터 분석 및 예측 모델에 대한 연구)

  • Hyoung Jo Huh;Sujin Ko;Seung Hyun Baek
    • Journal of Korean Society for Quality Management
    • /
    • v.51 no.4
    • /
    • pp.551-571
    • /
    • 2023
  • Purpose: The purpose of this study is to propose useful suggestions by analyzing the causal effect relationship between the failure rate of quality and the process variables in the C5ISR domain of the defense industry. Methods: The collected data through the in house Systems were analyzed using Big data analysis. Data analysis between quality data and A/S history data was conducted using the CRISP-DM(Cross-Industry Standard Process for Data Mining) analysis process. Results: The results of this study are as follows: After evaluating the performance of candidate models for the influence of inspection data and A/S history data, logistic regression was selected as the final model because it performed relatively well compared to the decision tree with an accuracy of 82%/67% and an AUC of 0.66/0.57. Based on this model, we estimated the coefficients using 'R', a data analysis tool, and found that a specific variable(continuous maximum discharge current time) had a statistically significant effect on the A/S quality failure rate and it was analysed that 82% of the failure rate could be predicted. Conclusion: As the first case of applying big data analysis to quality issues in the defense industry, this study confirms that it is possible to improve the market failure rates of defense products by focusing on the measured values of the main causes of failures derived through the big data analysis process, and identifies improvements, such as the number of data samples and data collection limitations, to be addressed in subsequent studies for a more reliable analysis model.

Study on Correlation-based Feature Selection in an Automatic Quality Inspection System using Support Vector Machine (SVM) (SVM 기반 자동 품질검사 시스템에서 상관분석 기반 데이터 선정 연구)

  • Song, Donghwan;Oh, Yeong Gwang;Kim, Namhun
    • Journal of Korean Institute of Industrial Engineers
    • /
    • v.42 no.6
    • /
    • pp.370-376
    • /
    • 2016
  • Manufacturing data analysis and its applications are getting a huge popularity in various industries. In spite of the fast advancement in the big data analysis technology, however, the manufacturing quality data monitored from the automated inspection system sometimes is not reliable enough due to the complex patterns of product quality. In this study, thus, we aim to define the level of trusty of an automated quality inspection system and improve the reliability of the quality inspection data. By correlation analysis and feature selection, this paper presents a method of improving the inspection accuracy and efficiency in an SVM-based automatic product quality inspection system using thermal image data in an auto part manufacturing case. The proposed method is implemented in the sealer dispensing process of the automobile manufacturing and verified by the analysis of the optimal feature selection from the quality analysis results.

Pattern Analysis of Nonconforming Farmers in Residual Pesticides using Exploratory Data Analysis and Association Rule Analysis (탐색적 자료 분석 및 연관규칙 분석을 활용한 잔류농약 부적합 농업인 유형 분석)

  • Kim, Sangung;Park, Eunsoo;Cho, Hyunjeong;Hong, Sunghie;Sohn, Byungchul;Hong, Jeehwa
    • Journal of Korean Society for Quality Management
    • /
    • v.49 no.1
    • /
    • pp.81-95
    • /
    • 2021
  • Purpose: The purpose of this study was to analysis pattern of nonconforming farmers who is one of the factors of unconformity in residual pesticides. Methods: Pattern analysis of nonconforming farmers were analyzed through convergence of safety data and farmer's DB data. Exploratory data analysis and association rule analysis were used for extracting factors related to unconformity. Results: The results of this study are as follows; regarding the exploratory data analysis, it was found that factors of farmers influencing unconformity in residual pesticides by total 9 factors; sampling time, gender, age, cultivation region, farming career, agricultural start form, type of agriculture, cultivation area, classification of agricultural products. Regarding the association rule analysis, non-conformity association rules were found over the past three years. There was a difference in the pattern of nonconforming farmers depending on the cultivation period. Conclusion: Exploratory data analysis and association rule analysis will be useful tools to establish more efficient and economical safety management plan for agricultural products.

Evaluation of Water Quality Using Multivariate Statistic Analysis with Optimal Scaling

  • Kim, Sang-Soo;Jin, Hyun-Guk;Park, Jong-Soo;Cho, Jang-Sik
    • Journal of the Korean Data and Information Science Society
    • /
    • v.16 no.2
    • /
    • pp.349-357
    • /
    • 2005
  • Principal component analysis(PCA) was carried out to evaluate the water quality with the monitering data collected from 1997 to 2003 along the coastal area of Ulsan, Korea. To enhance evaluation and to complement descriptive power of traditional PCA, optimal scaling was applied to transform the original data into optimally scaled data. Cluster analysis was also applied to classify the monitering stations according to their characteristics of water quality.

  • PDF

Trend of Research and Industry-Related Analysis in Data Quality Using Time Series Network Analysis (시계열 네트워크분석을 통한 데이터품질 연구경향 및 산업연관 분석)

  • Jang, Kyoung-Ae;Lee, Kwang-Suk;Kim, Woo-Je
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.6
    • /
    • pp.295-306
    • /
    • 2016
  • The purpose of this paper is both to analyze research trends and to predict industrial flows using the meta-data from the previous studies on data quality. There have been many attempts to analyze the research trends in various fields till lately. However, analysis of previous studies on data quality has produced poor results because of its vast scope and data. Therefore, in this paper, we used a text mining, social network analysis for time series network analysis to analyze the vast scope and data of data quality collected from a Web of Science index database of papers published in the international data quality-field journals for 10 years. The analysis results are as follows: Decreases in Mathematical & Computational Biology, Chemistry, Health Care Sciences & Services, Biochemistry & Molecular Biology, Biochemistry & Molecular Biology, and Medical Information Science. Increases, on the contrary, in Environmental Sciences, Water Resources, Geology, and Instruments & Instrumentation. In addition, the social network analysis results show that the subjects which have the high centrality are analysis, algorithm, and network, and also, image, model, sensor, and optimization are increasing subjects in the data quality field. Furthermore, the industrial connection analysis result on data quality shows that there is high correlation between technique, industry, health, infrastructure, and customer service. And it predicted that the Environmental Sciences, Biotechnology, and Health Industry will be continuously developed. This paper will be useful for people, not only who are in the data quality industry field, but also the researchers who analyze research patterns and find out the industry connection on data quality.

Analysis of IT Service Quality Elements Using Text Sentiment Analysis (텍스트 감정분석을 이용한 IT 서비스 품질요소 분석)

  • Kim, Hong Sam;Kim, Chong Su
    • Journal of Korean Society of Industrial and Systems Engineering
    • /
    • v.43 no.4
    • /
    • pp.33-40
    • /
    • 2020
  • In order to satisfy customers, it is important to identify the quality elements that affect customers' satisfaction. The Kano model has been widely used in identifying multi-dimensional quality attributes in this purpose. However, the model suffers from various shortcomings and limitations, especially those related to survey practices such as the data amount, reply attitude and cost. In this research, a model based on the text sentiment analysis is proposed, which aims to substitute the survey-based data gathering process of Kano models with sentiment analysis. In this model, from the set of opinion text, quality elements for the research are extracted using the morpheme analysis. The opinions' polarity attributes are evaluated using text sentiment analysis, and those polarity text items are transformed into equivalent Kano survey questions. Replies for the transformed survey questions are generated based on the total score of the original data. Then, the question-reply set is analyzed using both the original Kano evaluation method and the satisfaction index method. The proposed research model has been tested using a large amount of data of public IT service project evaluations. The result shows that it can replace the existing practice and it promises advantages in terms of quality and cost of data gathering. The authors hope that the proposed model of this research may serve as a new quality analysis model for a wide range of areas.

A study on Convergent & Adaptive Quality Analysis using DQnA model (데이터 품질 분석 모델(DQnA)을 이용한 융합적·적응적 품질 분석에 관한 연구)

  • Kim, Yong-Won
    • Journal of the Korea Convergence Society
    • /
    • v.5 no.4
    • /
    • pp.21-25
    • /
    • 2014
  • Now, almost enterprise is applying data analysis method using the information systems on based information technology. The data analysis is focusing on the Quality of the data affecting the decision-making of various companies. This is the result of the data quality is due to the important role in the various parts as well as the effective operation of the enterprise. In this study, we describe about the data quality assessment models that are currently being studied. Based on this, we describe about the adaptive DQnA model being utilized for data quality analysis, and discuss about the quality analysis using this method.