Browse > Article
http://dx.doi.org/10.15701/kcgs.2019.25.5.1

Raindrop Removal and Background Information Recovery in Coastal Wave Video Imagery using Generative Adversarial Networks  

Huh, Dong (Kyungpook National University)
Kim, Jaeil (Kyungpook National University)
Kim, Jinah (Korea Institute of Ocean Science and Technology)
Publication Information
Journal of the Korea Computer Graphics Society / v.25, no.5, 2019 , pp. 1-9 More about this Journal
Abstract
In this paper, we propose a video enhancement method using generative adversarial networks to remove raindrops and restore the background information on the removed region in the coastal wave video imagery distorted by raindrops during rainfall. Two experimental models are implemented: Pix2Pix network widely used for image-to-image translation and Attentive GAN, which is currently performing well for raindrop removal on a single images. The models are trained with a public dataset of paired natural images with and without raindrops and the trained models are evaluated their performance of raindrop removal and background information recovery of rainwater distortion of coastal wave video imagery. In order to improve the performance, we have acquired paired video dataset with and without raindrops at the real coast and conducted transfer learning to the pre-trained models with those new dataset. The performance of fine-tuned models is improved by comparing the results from pre-trained models. The performance is evaluated using the peak signal-to-noise ratio and structural similarity index and the fine-tuned Pix2Pix network by transfer learning shows the best performance to reconstruct distorted coastal wave video imagery by raindrops.
Keywords
Coastal wave video imagery; Raindrop removal; Background information recovery; Generative adversarial networks; Video enhancement;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efors, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125-1134.
2 Y. Bcngio, "Deep learning of representations for unsupervised and transfer learning," in Proceedings ICML workshop on unsupervised and transfer learning, 2012, pp. 17-36.
3 J.-H. Kim, J.-Y. Sim, and C.-S. Kim, "Video deraining and desnowing using temporal correlation and low-rank matrix completion," IEEE Transactions on Image Processing, vol, 24, no. 9, pp. 2658-2670, 2015.   DOI
4 S. You, R. T. Tan, R. Kawakami, Y. Mukaigawa, and K. Ikeuchi, "Adherent raindrop modeling, detectionand removal in video," IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 9, pp. 1721-1733, 2015.   DOI
5 X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, "Removing rain from single images via a deep detail network," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3855-3863.
6 L. Shen, Z. Yue, Q. Chen, F. Feng, and J. Ma, "Deep joint rain and haze removal from a single image," in 2018 24th International Conference on Pattern Recognition(ICPR), IEEE, 2018, pp. 2821-2826.
7 L. Wang, B. Yin, A. Guo, H. Ma, and J. Cao, "Skip-connection convolutional neural network for slill image crowd counting," Applied Intelligence, pp. 1-12, 2018.
8 Y. Luo, Y. Xu, and H. Ji, "Removing rain from a single image via discriminative sparse coding," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3397-3405.
9 K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
10 J. Kim, J. Kim, and S. Shin, "Wave celerity estimation using unsupervised image registration from video imagery," Journal of KIISE, vol. 42, no. 10, pp. 1207-1221, 2019.   DOI
11 D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sunkaran, A. S. Ross, N. Milojcvic-Dupont, N. Jaques, A. Waldman-Brown. et al., "Tackling climate change with machine learning." arXiv preprint arXiv:1906.05433, 2019.
12 R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, "Attentive generative adversarial network for raindrop removal from a single image," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. pp. 2482-2491.
13 P. Holman and M. C. Haller, "Remote sensing of the nearshore," Annual Review of Marine Science, vol. 5, pp. 95-113, 2013.   DOI
14 T. Kim, J. Kim, and J. Kim, "Hydrodynamic scene separation from video imagery of ocean wave using autoencoder," Journal Computer Graphics Society, vol. 25, no. 4, pp.9-16, 2019.   DOI
15 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Oruir, A. Cuurville, and Y. Bengio, "Generative adversarial nets," in Advances in neural information processing systems, 2014, pp. 2672-2680.