• Journal of Korean Society of Water and Wastewater
• /
• v.35 no.4
• /
• pp.259-275
• /
• 2021

This study explored the usefulness and implications of the Bayesian hyperparameter optimization in developing species distribution models (SDMs). A variety of machine learning (ML) algorithms, namely, support vector machine (SVM), random forest (RF), boosted regression tree (BRT), XGBoost (XGB), and Multilayer perceptron (MLP) were used for predicting the occurrence of four benthic macroinvertebrate species. The Bayesian optimization method successfully tuned model hyperparameters, with all ML models resulting an area under the curve (AUC) > 0.7. Also, hyperparameter search ranges that generally clustered around the optimal values suggest the efficiency of the Bayesian optimization in finding optimal sets of hyperparameters. Tree based ensemble algorithms (BRT, RF, and XGB) tended to show higher performances than SVM and MLP. Important hyperparameters and optimal values differed by species and ML model, indicating the necessity of hyperparameter tuning for improving individual model performances. The optimization results demonstrate that for all macroinvertebrate species SVM and RF required fewer numbers of trials until obtaining optimal hyperparameter sets, leading to reduced computational cost compared to other ML algorithms. The results of this study suggest that the Bayesian optimization is an efficient method for hyperparameter optimization of machine learning algorithms.

• Phonetics and Speech Sciences
• /
• v.13 no.1
• /
• pp.45-51
• /
• 2021

End-to-end (E2E) automatic speech recognition (ASR) has achieved promising performance gains with the introduced self-attention network, Transformer. However, due to training time and the number of hyperparameters, finding the optimal hyperparameter set is computationally expensive. This paper investigates the impact of hyperparameters in the Transformer network to answer two questions: which hyperparameter plays a critical role in the task performance and training speed. The Transformer network for training has two encoder and decoder networks combined with Connectionist Temporal Classification (CTC). We have trained the model with Wall Street Journal (WSJ) SI-284 and tested on devl93 and eval92. Seventeen hyperparameters were selected from the ESPnet training configuration, and varying ranges of values were used for experiments. The result shows that "num blocks" and "linear units" hyperparameters in the encoder and decoder networks reduce Word Error Rate (WER) significantly. However, performance gain is more prominent when they are altered in the encoder network. Training duration also linearly increased as "num blocks" and "linear units" hyperparameters' values grow. Based on the experimental results, we collected the optimal values from each hyperparameter and reduced the WER up to 2.9/1.9 from dev93 and eval93 respectively.

• International Journal of Advanced Culture Technology
• /
• v.7 no.4
• /
• pp.56-62
• /
• 2019

Recently, research using artificial neural networks has further expanded the field of neural network optimization and automatic structuring from improving inference accuracy. The performance of the machine learning algorithm depends on how the hyperparameters are configured. Open-source hyperparameter optimization software can be an important step forward in improving the performance of machine learning algorithms. In this paper, we review open-source hyperparameter optimization softwares.

• The Journal of the Acoustical Society of Korea
• /
• v.42 no.1
• /
• pp.32-41
• /
• 2023

Recently, studies on reverberation suppression using Non-negative Matrix Factorization (NMF) have been actively conducted. The NMF method uses a cost function based on the Kullback-Leibler divergence for optimization. And some constraints are added such as temporal continuity, pulse length, and energy ratio between reverberation and target. The tendency of constraints are controlled by hyperparameters. Therefore, in order to effectively suppress reverberation, hyperparameters need to be optimized. However, related studies are insufficient so far. In this paper, the reverberation suppression performance according to the three hyperparameters of the NMF was analyzed by using sea experimental data. As a result of analysis, when the value of hyperparameters for time continuity and pulse length were high, the energy ratio between the reverberation and the target showed better performance at less than 0.4, but it was confirmed that there was variability depending on the ocean environment. It is expected that the analysis results in this paper will be utilized as a useful guideline for planning precise experiments for optimizing hyperparameters of NMF in the future.

• Communications for Statistical Applications and Methods
• /
• v.14 no.3
• /
• pp.507-516
• /
• 2007

Multinomial logistic regression is probably the most popular representative of probabilistic discriminative classifiers for multiclass classification problems. In this paper, a kernel variant of multinomial logistic regression is proposed by combining a Newton's method with a bound optimization approach. This formulation allows us to apply highly efficient approximation methods that effectively overcomes conceptual and numerical problems of standard multiclass kernel classifiers. We also provide the approximate cross validation (ACV) method for choosing the hyperparameters which affect the performance of the proposed approach. Experimental results are then presented to indicate the performance of the proposed procedure.

• Electronics and Telecommunications Trends
• /
• v.34 no.4
• /
• pp.32-42
• /
• 2019

The performance of machine learning algorithms significantly depends on how a configuration of hyperparameters is identified and how a neural network architecture is designed. However, this requires expert knowledge of relevant task domains and a prohibitive computation time. To optimize these two processes using minimal effort, many studies have investigated automated machine learning in recent years. This paper reviews the conventional random, grid, and Bayesian methods for hyperparameter optimization (HPO) and addresses its recent approaches, which speeds up the identification of the best set of hyperparameters. We further investigate existing neural architecture search (NAS) techniques based on evolutionary algorithms, reinforcement learning, and gradient derivatives and analyze their theoretical characteristics and performance results. Moreover, future research directions and challenges in HPO and NAS are described.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.18 no.3
• /
• pp.755-778
• /
• 2024

In recent years, the number of devices being connected to the internet has grown enormously, as has the intrusive behavior in the network. Thus, it is important for intrusion detection systems to report all intrusive behavior. Using deep learning and machine learning algorithms, intrusion detection systems are able to perform well in identifying attacks. However, the concern with these deep learning algorithms is their inability to identify a suitable network based on traffic volume, which requires manual changing of hyperparameters, which consumes a lot of time and effort. So, to address this, this paper offers a solution using the extended compact genetic algorithm for the automatic tuning of the hyperparameters. The novelty in this work comes in the form of modeling the problem of identifying attacks as a multi-objective optimization problem and the usage of linkage learning for solving the optimization problem. The solution is obtained using the feature map-based Convolutional Neural Network that gets encoded into genes, and using the extended compact genetic algorithm the model is optimized for the detection accuracy and latency. The CIC-IDS-2017 and 2018 datasets are used to verify the hypothesis, and the most recent analysis yielded a substantial F1 score of 99.23%. Response time, CPU, and memory consumption evaluations are done to demonstrate the suitability of this model in a fog environment.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.16 no.4
• /
• pp.1128-1145
• /
• 2022

Owing to the rapid development of information science, data analysis based on machine learning has become an interdisciplinary and strategic area. Marine predators algorithm (MPA) is a novel metaheuristic algorithm inspired by the foraging strategies of marine organisms. Considering the randomness of these strategies, an improved algorithm called co-evolutionary cultural mechanism-based marine predators algorithm (CECMPA) is proposed. Through this mechanism, search agents in different spaces can share knowledge and experience to improve the performance of the native algorithm. More specifically, CECMPA has a higher probability of avoiding local optimum and can search the global optimum quickly. In this paper, it is the first to use CECMPA to perform feature subset selection and optimize hyperparameters in support vector machine (SVM) simultaneously. For performance evaluation the proposed method, it is tested on twelve datasets from the university of California Irvine (UCI) repository. Moreover, the coronavirus disease 2019 (COVID-19) can be a real-world application and is spreading in many countries. CECMPA is also applied to a COVID-19 dataset. The experimental results and statistical analysis demonstrate that CECMPA is superior to other compared methods in the literature in terms of several evaluation metrics. The proposed method has strong competitive abilities and promising prospects.

• Journal of Korea Multimedia Society
• /
• v.24 no.4
• /
• pp.519-527
• /
• 2021

Alzheimer's disease is one of the challenges to tackle in the coming aging era and is attempting to diagnose and predict through various biomarkers. While the application of various deep learning-based technologies as powerful imaging technologies has recently expanded across the medical industry, empirical design is not easy because there are various deep earning neural networks architecture and categorical hyperparameters that rely on problems and data to solve. In this paper, we show the possibility of optimizing a deep learning neural network structure and hyperparameters for Alzheimer's disease classification in amyloid brain images in a representative deep earning neural networks architecture using genetic algorithms. It was observed that the optimal deep learning neural network structure and hyperparameter were chosen as the values of the experiment were converging.

• Journal of the Institute of Convergence Signal Processing
• /
• v.21 no.3
• /
• pp.148-153
• /
• 2020

In order to obtain high accuracy with an convolutional neural network(CNN), it is necessary to set the optimal hyperparameters. However, the exact value of the hyperparameter that can make high performance is not known, and the optimal hyperparameter value is different based on the type of the dataset, therefore, it is necessary to find it through various experiments. In addition, since the range of hyperparameter values is wide and the number of combinations is large, it is necessary to find the optimal values of the hyperparameters after the experimental design in order to save time and computational costs. In this paper, we suggest an algorithm that use the design of experiments and grid search algorithm to determine the optimal hyperparameters for a classification problem. This algorithm determines the optima values of the hyperparameters that yields high performance using the factorial design of experiments. It is shown that the amount of computational time can be efficiently reduced and the accuracy can be improved by performing a grid search after reducing the search range of each hyperparameter through the experimental design. Moreover, Based on the experimental results, it was shown that the learning rate is the only hyperparameter that has the greatest effect on the performance of the model.