Browse > Article
http://dx.doi.org/10.3837/tiis.2018.05.019

Multi-focus Image Fusion using Fully Convolutional Two-stream Network for Visual Sensors  

Xu, Kaiping (School of Software, Tsinghua University)
Qin, Zheng (School of Software, Tsinghua University)
Wang, Guolong (School of Software, Tsinghua University)
Zhang, Huidi (School of Software, Tsinghua University)
Huang, Kai (School of Software, Tsinghua University)
Ye, Shuxiong (School of Software, Tsinghua University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.12, no.5, 2018 , pp. 2253-2272 More about this Journal
Abstract
We propose a deep learning method for multi-focus image fusion. Unlike most existing pixel-level fusion methods, either in spatial domain or in transform domain, our method directly learns an end-to-end fully convolutional two-stream network. The framework maps a pair of different focus images to a clean version, with a chain of convolutional layers, fusion layer and deconvolutional layers. Our deep fusion model has advantages of efficiency and robustness, yet demonstrates state-of-art fusion quality. We explore different parameter settings to achieve trade-offs between performance and speed. Moreover, the experiment results on our training dataset show that our network can achieve good performance with subjective visual perception and objective assessment metrics.
Keywords
multi-focus image fusion; fully convolutional neural network; deconvolution decoder; convolution fusion;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Burt, P.J. and E.H. Adelson, "The Laplacian Pyramid as a Compact Image Code," Readings in Computer Vision, vol. 31, no. 4, p. 671-679, 1987.
2 Liu, Z., et al., "Image fusion by using steerable pyramid," Pattern Recognition Letters, vol. 22, no. 9, p. 929-939, 2001.   DOI
3 Toet, A., "Image fusion by a ratio of low-pass pyramid," Pattern Recognition Letters, vol. 9, no. 4, p. 245-253, 1989.   DOI
4 Aslantas V, Toprak A N, "A pixel based multi-focus image fusion method," Optics Communications, vol. 332, no. 4, pp. 350-358, 2014.   DOI
5 Toet, A., J.M. Valeton, and L.J. Van Ruyven, "Merging thermal and visual images by a contrast pyramid," Optical Engineering, vol. 28, no. 7, p. 789-792, 1989.
6 Hossny, M., S. Nahavandi, and D. Creighton, "Comments on 'Information measure for performance of image fusion," Electronics Letters, vol. 44, no. 18, p. 1066-1067, 2008.   DOI
7 Xydeas, C.S. and V. Petrovic, "Objective image fusion performance measure," Military Technical Courier, vol. 36, no. 4, p. 308-309, 2000.
8 Piella, G. and H. Heijmans, "A new quality metric for image fusion," in Proc. of International Conference on Image Processing, Barcelona, Catalonia, Spain, p. III-173-176, 2003.
9 Shannon C E., "A Mathematical Theory of Communication[J]," Bell Labs Technical Journal, vol. 27, no. 4, pp. 379-423, 1948.   DOI
10 J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei, ILSVRC-2012, 2012.
11 Perronnin F, AVA, "A large-scale database for aesthetic visual analysis," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, pp. 2408-2415, 2012.
12 Li H, Chai Y, Li Z., "A new fusion scheme for multifocus images based on focused pixels detection," Machine Vision & Applications, vol. 24, no. 6, 1167-1181, 2013.   DOI
13 Rani, C.M.S., P.S.V.S. Rao, and V. Vijayakumar, "Improved Block Based Feature Level Image Fusion Technique Using Contourlet with Neural Network," International Journal of Soft Computing & Engineering, vol. 3, no. 4, 2012.
14 Li H, Chai Y, Li Z., "Multi-focus image fusion based on nonsubsampled contourlet transform and focused regions detection," Optik - International Journal for Light and Electron Optics, vol. 124, no. 1, 40-51, 2013.   DOI
15 Gao G, Xu L, Feng D.,"Multi-focus image fusion based on non-subsampled shearlet transform," Iet Image Processing, vol. 7, no. 6, pp. 633-639, 2013.   DOI
16 Haghighat M B A, Aghagolzadeh A, Seyedarabi H., "Multi-focus image fusion for visual sensor networks in DCT domain," Computers & Electrical Engineering, vol. 37, no. 5, pp. 789-797, 2011.   DOI
17 Jain, V. and H.S. Seung," Natural Image Denoising with Convolutional Networks," in Proc. of Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada. p. 769-776, 2008.
18 Zhong, J., et al, "Image Fusion and Super-Resolution with Convolutional Neural Network," in Proc. of Chinese Conference on Pattern Recognition, Chengdu, China: Springer Singapore, 2016.
19 Dong, C., et al., "Image Super-Resolution Using Deep Convolutional Networks," Pattern Analysis & Machine Intelligence IEEE Transactions on, vol. 38, no. 2, p. 295-307, 2016.   DOI
20 Long, J., E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. p. 3431-3440, 2015.
21 Liu, Y., et al., "Multi-focus image fusion with a deep convolutional neural network," Information Fusion, vol. 36, p. 191-207, 2017.   DOI
22 Jia, et al., "Caffe: Convolutional Architecture for Fast Feature Embedding," in Proc. of Eprint Arxiv, p. 675-678, 2014.
23 Feichtenhofer, C., A. Pinz, and A. Zisserman, "Convolutional Two-Stream Network Fusion for Video Action Recognition," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. p. 1933-1941, 2016.
24 Nair, V. and G.E. Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines," in Proc. of Icml, p. 807-814, 2015.
25 Noh, H., S. Hong, and B. Han, "Learning Deconvolution Network for Semantic Segmentation," in Proc. of IEEE International Conference on Computer Vision Santiago, Chile, p. 1520-1528, 2015.
26 Kingma, D. and J. Ba, "Adam: A Method for Stochastic Optimization," in Proc. of International Conference for Learning Representations, San Diego, USA, 2015.
27 Lecun, Y., et al., "Gradient-based learning applied to document recognition," in Proc. of Proceedings of the IEEE, vol. 86, no. 11, p. 2278-2324, 1998.   DOI
28 Rumelhart, D.E., G.E. Hinton, and R.J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, no. 6088, p. 533-536, 1986.   DOI
29 Bottou, L., "Large-Scale Machine Learning with Stochastic Gradient Descent," in Proc. of International Conference on Computational Statistics,Paris France, p. 177-186, 2010.
30 Mao, X.J., C. Shen, and Y.B. Yang, "Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections," in Proc. of arXiv preprint arXiv, 1606.08921, 2016.
31 He, K. and J. Sun, Convolutional Neural Networks at Constrained Time Cost, in IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. p. 5353-5360.
32 Forster, B., et al., "Complex wavelets for extended depth-of-field: A new method for the fusion of multichannel microscopy images," Microscopy Research & Technique, vol. 65, no.v1-2, p. 33-42, 2004.   DOI
33 Abdipour, M. and M. Nooshyar, "Multi-focus image fusion using sharpness criteria for visual sensor networks in wavelet domain," Computers & Electrical Engineering, vol. 51, p. 74-88, 2016.   DOI
34 Petrovic, V.S. and C.S. Xydeas, "Gradient-Based Multiresolution Image Fusion," IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, vol. 13, no. 2, p. 228-237, 2004.   DOI
35 Piella, G., "A general framework for multiresolution image fusion: from pixels to regions," Information Fusion, vol. 4, no. 4, p. 259-280, 2003.   DOI
36 Goshtasby, A.A. and S. Nikolov, "Image fusion: Advances in the state of the art," Information Fusion. Vol. 8, no. 2, p. 114-118, 2007.   DOI
37 Zhang, Z. and R.S. Blum, "A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application," Proceedings of the IEEE , vol. 87, no. 8, p. 1315-1326, 1999.   DOI
38 Mount, D.M., N.S. Netanyahu, and J.L. Moigne, "Efficient algorithms for robust feature matching, Pattern Recognition, vol. 32, no. 1, p. 17-38, 1999.   DOI
39 Stockman, G., S. Kopstein, and S. Benett, "Matching Images to Models for Registration and Object Detection via Clustering," IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 4, no. 3, p. 229-41, 1982.
40 Benediktsson, J.A. and I. Kanellopoulos, "Classification of multisource and hyperspectral data based on decision fusion," IEEE Transactions on Geoscience & Remote Sensing, 37, no. 3, p. 1367-1377, 1999.   DOI
41 Mamatha, S.G., S.A. Rahim, and C.P. Raj, "Feature-level multi-focus image fusion using neural network and image enhancement, Global," Global Journal of Computer Science & Technology, vol. 12, no. 10-F, 2012.
42 Wang, Z., et al., "A comparative analysis of image fusion methods," IEEE Transactions on Geoscience & Remote Sensing, vol. 43, no. 6, p. 1391-1402, 2005.   DOI
43 Cvejic, N., et al., "Region-Based Multimodal Image Fusion using ICA Bases," in Proc. of International Conference on Image Processing, Atlanta, Georgia, USA. p. 1801-1804, 2006.
44 Li, S., X. Kang, and J. Hu, "Image fusion with guided filtering," IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, vol. 22, no. 7, p. 2864-2875, 2013.   DOI
45 Li, S., J.T. Kwok, and Y. Wang, "Multifocus image fusion using artificial neural networks," Pattern Recognition Letters, vol. 23, no. 8, p. 985-997, 2002.   DOI
46 Li, S., et al., "Fusing images with different focuses using support vector machines," Neural Networks IEEE Transactions on, vol. 15, no. 6, p. 1555-61, 2004.   DOI
47 Li, H., B.S. Manjunath, and S.K. Mitra, "Multi-sensor image fusion using the wavelet transform," in Proc. of Image Processing, Proceedings. ICIP-94., IEEE International Conference, 1994.
48 Zeiler, M.D. and R. Fergus, "Visualizing and Understanding Convolutional Networks," in Proc. of European Conference on Computer Vision, Zurich, Switzerland, p. 818-833, 2014.
49 Glasner, D., S. Bagon, and M. Irani, "Super-resolution from a single image," in Proc. of International Conference on Computer Vision, Kyoto, Japan, p. 349-356, 2009.