Browse > Article
http://dx.doi.org/10.9717/kmms.2020.24.1.013

Proposing Effective Regularization Terms for Improvement of WGAN  

Hahn, Hee Il (Dept. of Information & Communications Eng., College of Engineering, Hankuk University of Foreign Studies)
Publication Information
Journal of Korea Multimedia Society / v.24, no.1, 2021 , pp. 13-20 More about this Journal
Abstract
A Wasserstein GAN(WGAN), optimum in terms of minimizing Wasserstein distance, still suffers from inconsistent convergence or unexpected output due to inherent learning instability. It is widely known some kinds of restriction on the discriminative function should be considered to solve such problems, which implies the importance of Lipschitz continuity. Unfortunately, there are few known methods to satisfactorily maintain the Lipschitz continuity of the discriminative function. In this paper we propose techniques to stably maintain the Lipschitz continuity of the discriminative function by adding effective regularization terms to the objective function, which limit the magnitude of the gradient vectors of the discriminator to one or less. Extensive experiments are conducted to evaluate the performance of the proposed techniques, which shows the single-sided penalty improves convergence compared with the gradient penalty at the early learning process, while the proposed additional penalty increases inception scores by 0.18 after 100,000 number of learning.
Keywords
Deep Learning; Generative Model; Lipschitz Continuity; Training Stability; Wasserstein Distance; Wasserstein GAN; Regularization Terms;
Citations & Related Records
연도 인용수 순위
  • Reference
1 R.L. Dobrushin, "Prescribing a System of Random Variables by Conditional Distributions," Theory of Probability and its Applications, No. 15, pp. 458-486, 1970.   DOI
2 A. Papoulis, Probability, Random Variables, and Stochastic Processes. Third Ed. McGrow-Hill, 1991.
3 C. Villani, Optimal Transport: Old and New, Vol. 338, Springer Science & Business Media, 2008.
4 A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images," Technical Report TR-2009, University of Toronto, 2009.
5 H. Petzka, A. Fischer, and D. Lukovnicov, "On the Regularization of Wasserstein GANs," arXiv:1709.08894v2 [ stat.ML], Sep. 2017.
6 I. Toolstikhin, S. Gelly, O. Bousquet, C. Simon-Gabriel, and B. Scholkopf, "Adagan: Boosting Generative Models," arXiv e-prints, arXiv: 1701.02386, 2017.
7 M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein GAN," arXiv:1701.07875v3 [ stat. ML], Dec. 2017.
8 M. Arjovsky and L. Bottou, "Towards Principled Methods for Training Generative Adversarial Networks," arXiv:1701.04862v1 [ stat, ML], Jan. 2017.
9 A. Radford, "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks," arXiv:1511.06434v2 [ cs.LG], Jan. 2016.
10 T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, "Improved Techniques for Training GANs," In Advances in Neural Information Procesing Systems 29, pp. 2234-2242, 2016.
11 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al., "Generative Adversarial Nets," In Advances in Neural Information Procesing Systems 27, pp. 2672-2680, 2014.
12 D.P. Kingma and M. Welling, "Auto-Encoding Variational Bayes," In Proceedings of the International Conference on Learning Representations(ICLR), 2014.
13 J. Adler and S. Luntz, "Banach Wasserstein GAN," arXiv:1806.06621v2 [ cs.CV], Jan. 2019.
14 H.I. Hahn, "Improving the Performance of WGAN Using Stabilization of Lipschitz Continuity of the Discriminator," Journal of The Institute of Electronics and Information Engineers, Vol. 52, No. 2, Feb. 2019.
15 G. Qi, "Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities," arXiv:1701. 06264v6 [ cs.CV], Mar. 2018.
16 I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, "Improved Training of Wasserstein GANs," arXiv e-prints, arXiv:1704.00028v3 [ cs.LG], Dec. 2017.