• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.12 no.11
• /
• pp.5287-5303
• /
• 2018

Handling vast amount of data found in large transactional datasets is an obvious challenge for the conventional data mining algorithms. Addressing this challenge, our paper proposes a parallel approach for proper decomposition of mining problem into sub-problems in order to find frequent patterns from these datasets. The proposed, Pattern Mining for Large Distributed Dataset (PMLDD) approach, ensures minimum dependencies as well as minimum communications among sub-problems. It establishes a linear aggregation of the intermediate results so that it can be adapted to large-scale programming models like MapReduce. In this context, an algorithmic structure for MapReduce programming model is presented. PMLDD guarantees an efficient load balancing among the sub-problems by a specific selection criterion. Further, it optimizes the number of required iterations over the dataset for mining frequent patterns as compared to the existing approaches. Finally, we believe that our approach is scalable enough to handle larger datasets in terms of performance evaluation, and the result analysis justifies all these mentioned concerns.

• Journal of Environmental Health Sciences
• /
• v.45 no.2
• /
• pp.192-202
• /
• 2019

Objectives: According to the central limit theorem, the samples in population might be considered to follow normal distribution if a large number of samples are available. Once we assume that toxicity dataset follow normal distribution, we can treat and process data statistically to calculate genus or species mean value with standard deviation. However, little is known and only limited studies are conducted to investigate whether toxicity dataset follows normal distribution or not. Therefore, the purpose of study is to evaluate the generally accepted normality hypothesis of aquatic toxicity dataset Methods: We selected the 8 chemicals, which consist of 4 organic and 4 inorganic chemical compounds considering data availability for the development of species sensitivity distribution. Toxicity data were collected at the US EPA ECOTOX Knowledgebase by simple search with target chemicals. Toxicity data were re-arranged to a proper format based on the endpoint and test duration, where we conducted normality test according to the Shapiro-Wilk test. Also we investigated the degree of normality by simple log transformation of toxicity data Results: Despite of the central limit theorem, only one large dataset (n>25) follow normal distribution out of 25 large dataset. By log transforming, more 7 large dataset show normality. As a result of normality test on small dataset (n<25), log transformation of toxicity value generally increases normality. Both organic and inorganic chemicals show normality growth for 26 species and 30 species, respectively. Those 56 species shows normality growth by log transformation in the taxonomic groups such as amphibian (1), crustacean (21), fish (22), insect (5), rotifer (2), and worm (5). In contrast, mollusca shows normality decrease at 1 species out of 23 that originally show normality. Conclusions: The normality of large toxicity dataset was not always satisfactory to the central limit theorem. Normality of those data could be improved through log transformation. Therefore, care should be taken when using toxicity data to induce, for example, mean value for risk assessment.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.17 no.5
• /
• pp.273-280
• /
• 2022

Along with the advancement of deep learning technology, securing high-quality dataset for verification of developed technology is emerging as an important issue, and developing robust deep learning models to the domestic road environment is focused by many research groups. Especially, unlike expressways and automobile-only roads, in the complex city driving environment, various dynamic objects such as motorbikes, electric kickboards, large buses/truck, freight cars, pedestrians, and traffic lights are mixed in city road. In this paper, we built our dataset through multi camera-based processing (collection, refinement, and annotation) including the various objects in the city road and estimated quality and validity of our dataset by using YOLO-based model in object detection. Then, quantitative evaluation of our dataset is performed by comparing with the public dataset and qualitative evaluation of it is performed by comparing with experiment results using open platform. We generated our 2D dataset based on annotation rules of KITTI/COCO dataset, and compared the performance with the public dataset using the evaluation rules of KITTI/COCO dataset. As a result of comparison with public dataset, our dataset shows about 3 to 53% higher performance and thus the effectiveness of our dataset was validated.

• The Transactions of the Korea Information Processing Society
• /
• v.13 no.4
• /
• pp.181-188
• /
• 2024

Traditional profanity detection methods have limitations in identifying intentionally altered profanities. This paper introduces a new method based on Named Entity Recognition, a subfield of Natural Language Processing. We developed a profanity detection technique using sequence labeling, for which we constructed a dataset by labeling some profanities in Korean malicious comments and conducted experiments. Additionally, to enhance the model's performance, we augmented the dataset by labeling parts of a Korean hate speech dataset using one of the large language models, ChatGPT, and conducted training. During this process, we confirmed that filtering the dataset created by the large language model by humans alone could improve performance. This suggests that human oversight is still necessary in the dataset augmentation process.

• KIPS Transactions on Software and Data Engineering
• /
• v.12 no.11
• /
• pp.481-492
• /
• 2023

This study explores how to build a Korean dataset to extract information from text using generative large language models. In modern society, mixed information circulates rapidly, and effectively categorizing and extracting it is crucial to the decision-making process. However, there is still a lack of Korean datasets for training. To overcome this, this study attempts to extract information using text-based zero-shot learning using a generative large language model to build a purposeful Korean dataset. In this study, the language model is instructed to output the desired result through prompt engineering in the form of "system"-"instruction"-"source input"-"output format", and the dataset is built by utilizing the in-context learning characteristics of the language model through input sentences. We validate our approach by comparing the generated dataset with the existing benchmark dataset, and achieve 25.47% higher performance compared to the KLUE-RoBERTa-large model for the relation information extraction task. The results of this study are expected to contribute to AI research by showing the feasibility of extracting knowledge elements from Korean text. Furthermore, this methodology can be utilized for various fields and purposes, and has potential for building various Korean datasets.

• Journal of Broadcast Engineering
• /
• v.28 no.3
• /
• pp.302-313
• /
• 2023

This paper aims to develop a framework that can fully automate the quality management of training data used in large-scale Artificial Intelligence (AI) models built by the Ministry of Science and ICT (MSIT) in the 'AI Hub Data Dam' project, which has invested more than 1 trillion won since 2017. Autonomous driving technology using AI has achieved excellent performance through many studies, but it requires a large amount of high-quality data to train the model. Moreover, it is still difficult for humans to directly inspect the processed data and prove it is valid, and a model trained with erroneous data can cause fatal problems in real life. This paper presents a dataset reconstruction framework that removes abnormal data from the constructed dataset and introduces strategies to improve the performance of AI models by reconstructing them into a reliable dataset to increase the efficiency of model training. The framework's validity was verified through an experiment on the autonomous driving dataset published through the AI Hub of the National Information Society Agency (NIA). As a result, it was confirmed that it could be rebuilt as a reliable dataset from which abnormal data has been removed.

• Phonetics and Speech Sciences
• /
• v.15 no.3
• /
• pp.83-88
• /
• 2023

This study investigates the fine-tuning of large-scale Automatic Speech Recognition (ASR) models, specifically OpenAI's Whisper model, for domain-specific applications using the KsponSpeech dataset. The primary research questions address the effectiveness of targeted lexical item emphasis during fine-tuning, its impact on domain-specific performance, and whether the fine-tuned model can maintain generalization capabilities across different languages and environments. Experiments were conducted using two fine-tuning datasets: Set A, a small subset emphasizing specific lexical items, and Set B, consisting of the entire KsponSpeech dataset. Results showed that fine-tuning with targeted lexical items increased recognition accuracy and improved domain-specific performance, with generalization capabilities maintained when fine-tuned with a smaller dataset. For noisier environments, a trade-off between specificity and generalization capabilities was observed. This study highlights the potential of fine-tuning using minimal domain-specific data to achieve satisfactory results, emphasizing the importance of balancing specialization and generalization for ASR models. Future research could explore different fine-tuning strategies and novel technologies such as prompting to further enhance large-scale ASR models' domain-specific performance.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.18 no.5
• /
• pp.203-208
• /
• 2023

Recently, with the growing interest in the field of autonomous driving, many researchers have been focusing on developing autonomous driving software platforms. In particular, we have concentrated on developing 3D object detection models that can improve real-time performance. In this paper, we introduce a self-constructed 3D LiDAR dataset specific to domestic environments and propose a VariFocal-based CenterPoint for the 3D object detection model, with improved performance over the previous models. Furthermore, we present experimental results comparing the performance of the 3D object detection modules using our self-built and public dataset. As the results show, our model, which was trained on a large amount of self-constructed dataset, successfully solves the issue of failing to detect large vehicles and small objects such as motorcycles and pedestrians, which the previous models had difficulty detecting. Consequently, the proposed model shows a performance improvement of about 1.0 mAP over the previous model.

• Journal of Information Processing Systems
• /
• v.19 no.5
• /
• pp.602-613
• /
• 2023

High-quality image datasets are in high demand for various applications. With many online sources providing manually collected datasets, a persisting challenge is to fully automate the dataset collection process. In this study, we surveyed an automatic image dataset generation field through analyzing a collection of existing studies. Moreover, we examined fields that are closely related to automated dataset generation, such as query expansion, web scraping, and dataset quality. We assess how both noise and regional search engine differences can be addressed using an automated search query expansion focused on hypernyms, allowing for user-specific manual query expansion. Combining these aspects provides an outline of how a modern web scraping application can produce large-scale image datasets.

• International Journal of Internet, Broadcasting and Communication
• /
• v.16 no.3
• /
• pp.27-32
• /
• 2024

Reliable and fine-grained musical metadata are required for efficient search of rapidly increasing music files. In particular, since the primary motive for listening to music is its emotional effect, diversion, and the memories it awakens, emotion classification along with genre classification of music is crucial. In this paper, as an initial approach towards a "ground-truth" dataset for music emotion and genre classification, we elaborately generated a music corpus through labeling of a large number of ordinary people. In order to verify the suitability of the dataset through the classification results, we extracted features according to MPEG-7 audio standard and applied different machine learning models based on statistics and deep neural network to automatically classify the dataset. By using standard hyperparameter setting, we reached an accuracy of 93% for genre classification and 80% for emotion classification, and believe that our dataset can be used as a meaningful comparative dataset in this research field.