• Proceedings of the IEEK Conference
• /
• 2008.06a
• /
• pp.667-668
• /
• 2008

This paper demonstrates that the recurrent neural networks can be used successfully for the identification of finite automata (FAs). A new type of recurrent neural network (RNN) is proposed and the offline training algorithm, regulated Levenberg-Marquadt (LM) algorithm, for the network is developed. Simulation result shows that the identification and the extraction of FAs are practically achievable.

• Journal of Intelligence and Information Systems
• /
• v.22 no.2
• /
• pp.127-142
• /
• 2016

Deep learning model is a kind of neural networks that allows multiple hidden layers. There are various deep learning architectures such as convolutional neural networks, deep belief networks and recurrent neural networks. Those have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. Among those architectures, convolutional neural networks and recurrent neural networks are classified as the supervised learning model. And in recent years, those supervised learning models have gained more popularity than unsupervised learning models such as deep belief networks, because supervised learning models have shown fashionable applications in such fields mentioned above. Deep learning models can be trained with backpropagation algorithm. Backpropagation is an abbreviation for "backward propagation of errors" and a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent. The method calculates the gradient of an error function with respect to all the weights in the network. The gradient is fed to the optimization method which in turn uses it to update the weights, in an attempt to minimize the error function. Convolutional neural networks use a special architecture which is particularly well-adapted to classify images. Using this architecture makes convolutional networks fast to train. This, in turn, helps us train deep, muti-layer networks, which are very good at classifying images. These days, deep convolutional networks are used in most neural networks for image recognition. Convolutional neural networks use three basic ideas: local receptive fields, shared weights, and pooling. By local receptive fields, we mean that each neuron in the first(or any) hidden layer will be connected to a small region of the input(or previous layer's) neurons. Shared weights mean that we're going to use the same weights and bias for each of the local receptive field. This means that all the neurons in the hidden layer detect exactly the same feature, just at different locations in the input image. In addition to the convolutional layers just described, convolutional neural networks also contain pooling layers. Pooling layers are usually used immediately after convolutional layers. What the pooling layers do is to simplify the information in the output from the convolutional layer. Recent convolutional network architectures have 10 to 20 hidden layers and billions of connections between units. Training deep learning networks has taken weeks several years ago, but thanks to progress in GPU and algorithm enhancement, training time has reduced to several hours. Neural networks with time-varying behavior are known as recurrent neural networks or RNNs. A recurrent neural network is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior. Unlike feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs. Early RNN models turned out to be very difficult to train, harder even than deep feedforward networks. The reason is the unstable gradient problem such as vanishing gradient and exploding gradient. The gradient can get smaller and smaller as it is propagated back through layers. This makes learning in early layers extremely slow. The problem actually gets worse in RNNs, since gradients aren't just propagated backward through layers, they're propagated backward through time. If the network runs for a long time, that can make the gradient extremely unstable and hard to learn from. It has been possible to incorporate an idea known as long short-term memory units (LSTMs) into RNNs. LSTMs make it much easier to get good results when training RNNs, and many recent papers make use of LSTMs or related ideas.

• The Journal of the Convergence on Culture Technology
• /
• v.9 no.1
• /
• pp.649-654
• /
• 2023

This paper proposes the design of a neural network structure search model using graph convolutional neural networks. Deep learning has a problem of not being able to verify whether the designed model has a structure with optimized performance due to the nature of learning as a black box. The neural network structure search model is composed of a recurrent neural network that creates a model and a convolutional neural network that is the generated network. Conventional neural network structure search models use recurrent neural networks, but in this paper, we propose GC-NAS, which uses graph convolutional neural networks instead of recurrent neural networks to create convolutional neural network models. The proposed GC-NAS uses the Layer Extraction Block to explore depth, and the Hyper Parameter Prediction Block to explore spatial and temporal information (hyper parameters) based on depth information in parallel. Therefore, since the depth information is reflected, the search area is wider, and the purpose of the search area of the model is clear by conducting a parallel search with depth information, so it is judged to be superior in theoretical structure compared to GC-NAS. GC-NAS is expected to solve the problem of the high-dimensional time axis and the range of spatial search of recurrent neural networks in the existing neural network structure search model through the graph convolutional neural network block and graph generation algorithm. In addition, we hope that the GC-NAS proposed in this paper will serve as an opportunity for active research on the application of graph convolutional neural networks to neural network structure search.

• Proceedings of the IEEK Conference
• /
• 1999.06a
• /
• pp.1041-1044
• /
• 1999

In This paper, we investigated the single step prediction for output responses of chaotic system with multi Input multi output using chaotic neural networks. Since the systems with chaotic characteristics are coupled between internal parameters, the chaotic neural networks is very suitable for output response prediction of chaotic system. To evaluate the performance of the proposed neural network predictor, we adopt for Lorenz attractor with chaotic responses and compare the results with recurrent neural networks. The results demonstrated superior performance on convergence and computation time than the predictor using recurrent neural networks. And we could also see good predictive capability of chaotic neural network predictor.

• Communications for Statistical Applications and Methods
• /
• v.27 no.3
• /
• pp.313-326
• /
• 2020

Deep Learning is the most important key to the development of Artificial Intelligence (AI). There are several distinguishable architectures of neural networks such as MLP, CNN, and RNN. Among them, we try to understand one of the main architectures called Recurrent Neural Network (RNN) that differs from other networks in handling sequential data, including time series and texts. As one of the main tasks recently in Natural Language Processing (NLP), we consider Neural Machine Translation (NMT) using RNNs. We also summarize fundamental structures of the recurrent networks, and some topics of representing natural words to reasonable numeric vectors. We organize topics to understand estimation procedures from representing input source sequences to predict target translated sequences. In addition, we apply multiple translation models with Gated Recurrent Unites (GRUs) in Keras on English-Korean sentences that contain about 26,000 pairwise sequences in total from two different corpora, colloquialism and news. We verified some crucial factors that influence the quality of training. We found that loss decreases with more recurrent dimensions and using bidirectional RNN in the encoder when dealing with short sequences. We also computed BLEU scores which are the main measures of the translation performance, and compared them with the score from Google Translate using the same test sentences. We sum up some difficulties when training a proper translation model as well as dealing with Korean language. The use of Keras in Python for overall tasks from processing raw texts to evaluating the translation model also allows us to include some useful functions and vocabulary libraries as well.

• The Transactions of the Korean Institute of Electrical Engineers A
• /
• v.48 no.10
• /
• pp.1231-1238
• /
• 1999

This paper presents an approach for the optimal heating load Identification using Diagonal Recurrent Neural Networks(DRNN). In this paper, the DRNN captures the dynamic nature of a system and since it is not fully connected, training is much faster than a fully connected recurrent neural network. The architecture of DRNN is a modified model of the fully connected recurrent neural network with one hidden layer. The hidden layer is comprised of self-recurrent neurons, each feeding its output only into itself. In this study, A dynamic backpropagation (DBP) with delta-bar-delta learning method is used to train an optimal heating load identifier. Delta-bar-delta learning method is an empirical method to adapt the learning rate gradually during the training period in order to improve accuracy in a short time. The simulation results based on experimental data show that the proposed model is superior to the other methods in most cases, in regard of not only learning speed but also identification accuracy.

• Proceedings of the Korean Institute of Intelligent Systems Conference
• /
• 1997.10a
• /
• pp.121-124
• /
• 1997

In this paper, applications of self-recurrent neural networks based of adaptive controller to position control of flexible robot link are considered. The self-recurrent neural networks can be used to approximate any continuous function to any desired degree of accuracy and the weights are updated by feedback-error learning algorithm. Therefore, a comparative analysis was mode with linear controller through an simulation. The results are presented to illustrate the advantages and improved performance of the proposed position tracking controller over the conventional linear controller.

• Journal of Communications and Networks
• /
• v.5 no.1
• /
• pp.7-18
• /
• 2003

In this paper, a decision feedback recurrent neural network equalizer and a modified real time recurrent learning algorithm are proposed, and an adaptive adjusting of the learning step is also brought forward. Then, a complex case is considered. A decision feedback complex recurrent neural network equalizer and a modified complex real time recurrent learning algorithm are proposed. Moreover, weights of decision feedback recurrent neural network equalizer under burst-interference conditions are analyzed, and two anti-burst-interference algorithms to prevent equalizer from out of working are presented, which are applied to both real and complex cases. The performance of the recurrent neural network equalizer is analyzed based on numerical results.

• Proceedings of the KIEE Conference
• /
• 2000.07d
• /
• pp.2949-2952
• /
• 2000

A new neural networks and learning algorithm are proposed in order to measure nonlinear heights of complexed road environments in realtime without pre-information. This new neural networks is Error Self Recurrent Neural Networks(ESRN), The structure of it is similar to recurrent neural networks: a delayed output as the input and a delayed error between the output of plant and neural networks as a bias input. In addition, we compute the desired value of hidden layer by an optimal method instead of transfering desired values by back-propagation and each weights are updated by RLS(Recursive Least Square). Consequently. this neural networks are not sensitive to initial weights and a learning rate, and have a faster convergence rate than conventional neural networks. We can estimate nonlinear models in realtime by ESRN and learning algorithm and control nonlinear models. To show the performance of this one. we control 7 degree of freedom full car model with several control method. From this simulation. this estimation and controller were proved to be effective to the measurements of nonlinear road environment systems.

• 제어로봇시스템학회:학술대회논문집
• /
• 1997.10a
• /
• pp.625-628
• /
• 1997

In this paper the new neural network and sliding mode suspension controller is proposed. That neural network is error self-recurrent neural network. For fast on-line learning, this paper use recursive least squares method. A new neural networks converges considerably faster than the backpropagation algorithm and has advantages of being less affected by the poor initial weights and learning rate. The controller for suspension systems is designed according to sliding mode technique based on new proposed neural network.