RECURRENT NEURAL NETWORKS -What Do They Learn and How\ulcorner-

Supervised learnmg 01 recurrent neural networks (RNNs) is discussed. First, we review the present state of art, featuring their major properties in contrast of those of the multilayer neural networks. Then, we concisely describe one of the most practical learning algorithms, i.e. backpropagation through time. Revising the basic formulation of the learning algorithms, we derive a general formula to solve for the exact solution(s) of the whole connection weights w of RNNs. On this basis we introduce a novel interpretation of the supervised learning. Namely, we define a multidimensional Euclidean space, by assigning the cost function E(w) and every component of w to each coordinate axis. Since E=E(w) turns up as a hyper surface in this space, we refer to the surface as learning surface. We see that topological features of the learning surface are valleys and hills. Finally, after explicating that the numerical procedures of learning are equivalent to descending slopes of the learning surface along the steepest gradient, we show that a minimal value of E(w) is the intersection of curved valleys.