Browse > Article
http://dx.doi.org/10.12989/sss.2022.30.3.287

Investigation of the super-resolution methods for vision based structural measurement  

Wu, Lijun (College of Physics and Information Engineering, Fuzhou University)
Cai, Zhouwei (College of Physics and Information Engineering, Fuzhou University)
Lin, Chenghao (College of Physics and Information Engineering, Fuzhou University)
Chen, Zhicong (College of Physics and Information Engineering, Fuzhou University)
Cheng, Shuying (College of Physics and Information Engineering, Fuzhou University)
Lin, Peijie (College of Physics and Information Engineering, Fuzhou University)
Publication Information
Smart Structures and Systems / v.30, no.3, 2022 , pp. 287-301 More about this Journal
Abstract
The machine-vision based structural displacement measurement methods are widely used due to its flexible deployment and non-contact measurement characteristics. The accuracy of vision measurement is directly related to the image resolution. In the field of computer vision, super-resolution reconstruction is an emerging method to improve image resolution. Particularly, the deep-learning based image super-resolution methods have shown great potential for improving image resolution and thus the machine-vision based measurement. In this article, we firstly review the latest progress of several deep learning based super-resolution models, together with the public benchmark datasets and the performance evaluation index. Secondly, we construct a binocular visual measurement platform to measure the distances of the adjacent corners on a chessboard that is universally used as a target when measuring the structure displacement via machine-vision based approaches. And then, several typical deep learning based super resolution algorithms are employed to improve the visual measurement performance. Experimental results show that super-resolution reconstruction technology can improve the accuracy of distance measurement of adjacent corners. According to the experimental results, one can find that the measurement accuracy improvement of the super resolution algorithms is not consistent with the existing quantitative performance evaluation index. Lastly, the current challenges and future trends of super resolution algorithms for visual measurement applications are pointed out.
Keywords
deep learning; machine vision; structural measurement; super-resolution reconstruction;
Citations & Related Records
Times Cited By KSCI : 4  (Citation Analysis)
연도 인용수 순위
1 Zhang, Z. (2000), "A flexible new technique for camera calibration", IEEE Transact. Pattern Anal. Mach. Intell., 22(11), 1330-1334. https://doi.org/10.1109/34.888718   DOI
2 Zhang, K., Zuo, W. and Zhang, L. (2017), "Learning a single convolutional super-resolution network for multiple degradations", Proceedings of Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 33662-3271.
3 Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B. and Fu, Y. (2018a), "Image super-resolution using very deep residual channel attention networks", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 286-301.
4 Zhang, Y., Liu, S., Dong, C., Zhang, X. and Yuan, Y. (2019), "Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution", IEEE Transact. Image Process., 29, 1101-1112. https://doi.org/10.1109/TIP.2019.2938347   DOI
5 Huang, G., Liu, Z., Van Der Maaten, L. and Weinberger, K.Q. (2017), "Densely connected convolutional networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 2261-2269.
6 Tai, Y., Yang, J. and Liu, X. (2017b), "Image super-resolution via deep recursive residual network", Proceedings of the 30th 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July, pp. 2790-2798.
7 Song, X., Liu, W., Liu, J., Liu, C., Lu, C. and Gao, H. (2019), "Deep CNN jointing low-high level feature for image superresolution", Proceedings of the 10th International Conference on Graphics and Image Processing (ICGIP 2018), Vol. 11069, Chengdu, China, December, pp. 1059-1065. https://doi.org/10.1117/12.2524412   DOI
8 Stark, H. and Oskoui, P. (1989), "High-resolution image recovery from image-plane arrays, using convex projections", J. Optical Soc. Am. A Optics Image Sci., 6(11), 1715-1726.   DOI
9 Tai, Y., Yang, J., Liu, X. and Xu, C. (2017a), "Memnet: A persistent memory network for image restoration", Proceedings of the IEEE International Conference on Computer Vision, Salt Lake City, UT, USA, June, pp. 4539-4547.
10 Tian, J. and Ma, K.K. (2011), "A survey on super-resolution imaging", Signal Image Video Process., 5(3), 329-342. https://doi.org/10.1007/s11760-010-0204-6   DOI
11 Ye, X.W., Dong, C.Z. and Liu, T. (2016b), "Image-based structural dynamic displacement measurement using different multi-object tracking algorithms", Smart Struct. Syst., Int. J., 17(6), 935-956. https://doi.org/10.12989/sss.2016.17.6.935   DOI
12 Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.H. and Liao, Q. (2019a), "Deep learning for single image super-resolution: A brief review", IEEE Transact. Multimedia, 21(12), 3106-3121. https://doi.org/10.1109/TMM.2019.2919431   DOI
13 Yang, W., Wang, W., Zhang, X., Sun, S. and Liao, Q. (2019b), "Lightweight feature fusion network for single image superresolution", IEEE Signal Process. Lett., 26(4), 538-542. https://doi.org/10.1109/LSP.2018.2890770   DOI
14 Ye, X.W., Yi, T.H., Dong, C.Z. and Liu, T. (2016a), "Vision-based structural displacement measurement: System performance evaluation and influence factor analysis", Measurement, 88, 372-384. https://doi.org/10.1016/j.measurement.2016.01.024   DOI
15 Yoon, H., Elanwar, H., Choi, H., Golparvar-Fard, M. and Spencer Jr, B.F. (2016), "Target-free approach for vision-based structural system identification using consumergrade cameras", Struct. Control Health Monitor., 23(12), 1405-1416. https://doi.org/10.1002/stc.1850   DOI
16 Yu, J., Fan, Y., Yang, J., Xu, N., Wang, Z., Wang, X. and Huang, T. (2018), "Wide activation for efficient and accurate image superresolution", arXiv preprint arXiv:1808.08718. https://doi.org/10.48550/arXiv.1808.08718   DOI
17 Zeyde, R., Elad, M. and Protter, M. (2010), "On single image scale-up using sparse-representations", Proceedings of International Conference on Curves and Surfaces, Avignon, France, June, pp. 711-730. https://doi.org/10.1007/978-3-642-27413-8_47   DOI
18 Agustsson, E. and Timofte, R. (2017), "Ntire 2017 challenge on single image super-resolution: Dataset and study", Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, July, pp. 1122-1131.
19 Seif, G. and Androutsos, D (2018), "Edge-based loss function for single image super-resolution", Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, Alberta, Canada, April, pp. 1468-1472. https://doi.org/10.1109/ICASSP.2018.8461664   DOI
20 Liu, K., Ma, Y., Xiong, H., Yan, Z., Zhou, Z., Fang, P. and Liu, C. (2020b), "Medical image super-resolution method based on dense blended attention network", Laser & Optoelectronics Progress, 57(2), 021014. https://doi.org/10.48550/arXiv.1905.05084   DOI
21 Chang, H., Yeung, D.Y. and Xiong, Y. (2004), "Super-resolution through neighbor embedding", Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, Washington, DC, USA, June-July. https://doi.org/10.1109/CVPR.2004.1315043   DOI
22 Ahn, N., Kang, B. and Sohn, K.A. (2018), "Fast, accurate, and lightweight super-resolution with cascading residual network", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 252-268.
23 Blu, T., Thevenaz, P. and Unser, M. (2004), "Linear interpolation revitalized", IEEE Transact. Image Process., 13(5), 710-719. https://doi.org/10.1109/TIP.2004.826093   DOI
24 Caetano, E., Silva, S. and Bateira, J. (2007), "Application of a vision system to the monitoring of cable structures", Proceedings of the 7th International Symposium on Cable Dynamics, Vienna, Austria, December, pp. 225-236.
25 Dong, C., Loy, C.C., He, K. and Tang, X. (2014), "Learning a deep convolutional network for image super-resolution", Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland, September, pp. 184-199.
26 Feng, D. and Feng, M.Q. (2015), "Model updating of railway bridge using in situ dynamic displacement measurement under trainloads", J. Bridge Eng., 20(12), 4015019. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000765   DOI
27 Fujimoto, A., Ogawa, T., Yamamoto, K., Matsui, Y., Yamasaki, T. and Aizawa, K. (2016), "Manga109 dataset and creation of metadata", Proceedings of the 1st International Workshop on Comics Analysis, Processing and Understanding, Cancun, Mexico, December, pp. 1-5. https://doi.org/10.1145/3011549.3011551   DOI
28 Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014), "Generative adversarial nets", (Ghahramani, Z., Welling, M., Cortes, C., et al.), In: Advances in Neural Information Processing Systems, Vol. 27, pp. 2672-2680.
29 He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 770-778.
30 Hansen, R.S., Waldram, D.W., Thai, T.Q. and Berke, R.B. (2021), "Super Resolution Digital Image Correlation (SR-DIC): An Alternative to Image Stitching at High Magnifications", Experim. Mech., 61(9), 1351-1368. https://doi.org/10.1007/s11340-021-00729-2   DOI
31 Hore, A. and Ziou, D. (2010), "Image quality metrics: PSNR vs. SSIM", Proceedings of 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, August, pp. 2366-2369. https://doi.org/10.1109/ICPR.2010.579   DOI
32 Huang, J.B., Singh, A. and Ahuja, N. (2015), "Single image superresolution from transformed self-exemplars", Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June, pp. 5197-5206.
33 Irani, M. and Peleg, S. (1991), "Improving resolution by image registration", CVGIP: Graphical models and image processing, 53(3), 231-239. https://doi.org/10.1016/1049-9652(91)90045-L   DOI
34 Keys, R.G. (2003), "Cubic convolution interpolation for digital image processing", IEEE Transact. Acoust. Speech Signal Process., 29(6), 1153-1160. https://doi.org/10.1109/TASSP.1981.1163711   DOI
35 Bulat, A., Yang, J. and Tzimiropoulos, G. (2018), "To learn image super-resolution, use a GAN to learn how to do image degradation first", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 185-200.
36 Zhang, Y., Tian, Y., Kong, Y., Zhong, B. and Fu, Y. (2018b), "Residual dense network for image super-resolution", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June, pp. 2472-2481.
37 Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. and Shi, W. (2017), "Photo-realistic single image super-resolution using a generative adversarial network", Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July, pp. 105-114.
38 Arbelaez, P., Maire, M., Fowlkes, C. and Malik, J. (2011), "Contour detection and hierarchical image segmentation", IEEE Transact. Pattern Anal. Mach. Intell., 33(5), 898-916. https://doi.org/10.1109/TPAMI.2010.161   DOI
39 Bevilacqua, M., Roumy, A., Guillemot, C. and Alberi-Morel, M.L. (2012), "Low-complexity single-image super-resolution based on nonnegative neighbor embedding", Proceedings of the 23rd British Machine Vision Conference (BMVC), Surrey, UK, September, pp. 1-10. http://dx.doi.org/10.5244/C.26.135   DOI
40 Bouguet, J.Y. (2013), Camera Calibration Toolbox for Matlab, Wwwvisioncaltechedu/bouguetj
41 Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y. and Change Loy, C. (2018), "Esrgan: Enhanced super-resolution generative adversarial networks", Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany, September.
42 Tong, T., Li, G., Liu, X. and Gao, Q. (2017), "Image superresolution using dense skip connections", Proceedings of 2017 IEEE International Conference on Computer Vision, Venice, Italy, October, pp. 4809-4817.
43 Trajkovic, M. and Hedley, M (1998), "Fast corner detection", Image Vision Comput., 16(2), 75-87. https://doi.org/10.1016/S0262-8856(97)00056-5   DOI
44 Wang, Z., Bovik, A.C., Sheikh, H.R. and Simoncelli, E.P. (2004), "Image quality assessment: from error visibility to structural similarity", IEEE Trans. Image Process., 13(4), 600-612. https://doi.org/10.1109/TIP.2003.819861   DOI
45 Mecheri, K., Ziou, D. and Deschenes, F. (2007), "Super-resolution based on interpolation and global sub pixel translation", Proceedings of the 4th Canadian Conference on Computer and Robot Vision (CRV'07), Montreal, QC, Canada, May, pp. 448-458. https://doi.org/10.1109/CRV.2007.62   DOI
46 Cai, J., Zeng, H., Yong, H., Cao, Z. and Zhang, L. (2019), "Toward real-world single image super-resolution: A new benchmark and a new model", Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, October-November, pp. 3086-3095.
47 Cha, Y.J., Chen, J.G. and Buyukozturk, O. (2017), "Output-only computer vision based damage detection using phase-based optical flow and unscented Kalman filters", Eng. Struct., 132, 300-313. https://doi.org/10.1016/j.engstruct.2016.11.038   DOI
48 Martin, D., Fowlkes, C., Tal, D. and Malik, J. (2001), "A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics", Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV 2001), Vol. 2, Vancouver, BC, Canada, July, pp. 416-423. https://doi.org/10.1109/ICCV.2001.937655   DOI
49 Ni, K.S. and Nguyen, T.Q. (2007), "Image superresolution using support vector regression", IEEE Trans. Image Process., 16(6), 1596-1610. https://doi.org/10.1109/TIP.2007.896644   DOI
50 Oh, B.K., Hwang, J.W., Kim, Y., Cho, T. and Park, H.S. (2015), "Vision-based system identification technique for building structures using a motion capture system", J. Sound Vib., 356, 72-85. https://doi.org/10.1016/j.jsv.2015.07.011   DOI
51 Park, S.C., Park, M.K. and Kang, M.G. (2003), "Super-resolution image reconstruction: a technical overview", IEEE Signal Process. Magaz., 20(3), 21-36. https://doi.org/10.1109/MSP.2003.1203207   DOI
52 Wu, L., Su, Y., Chen, Z., Chen, S., Cheng, S. and Lin, P. (2020a), "Six-degree-of-freedom generalized displacements measurement based on binocular vision", Struct. Control Health Monitor., 27(2), e2458. https://doi.org/10.1002/stc.2458   DOI
53 Wang, Z., Chen, J. and Hoi, S.C. (2020a), "Deep learning for image super-resolution: A survey", IEEE Transact. Pattern Anal. Mach. Intell., 43(10), 3365-3387. https://doi.org/10.1109/TPAMI.2020.2982166   DOI
54 Wang, W., Hu, Y., Luo, Y. and Zhang, T. (2020b), "Brief survey of single image super-resolution reconstruction based on deep learning approaches", Sens. Imag., 21(1), 1-20. https://doi.org/10.1007/s11220-020-00285-4   DOI
55 Wu, L.J., Casciati, F. and Casciati, S. (2014), "Dynamic testing of a laboratory model via vision-based sensing", Eng. Struct., 60, 113-125. https://doi.org/10.1016/j.engstruct.2013.12.002   DOI
56 Chen, Y.W. (2011), "Learning-Based Super-Resolution", IEICE Technical Report; 111, 61-66.
57 Park, S.J., Son, H., Cho, S., Hong, K.S. and Lee, S. (2018), "Srfeat: Single image super-resolution with feature discrimination", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 439-455.
58 Schultz, R.R. and Stevenson, R.L. (1996), "Extraction of highresolution frames from video sequences", IEEE Transact. Image Process. A Publicat. IEEE Signal Process. Soc., 5(6), 996-1011. https://doi.org/10.1109/83.503915   DOI
59 Chantas, G.K., Galatsanos, N.P. and Woods, N.A. (2007), "Superresolution based on fast registration and maximum a posteriori reconstruction", IEEE Transact. Image Process., 16(7), 1821-1830. https://doi.org/10.1109/TIP.2007.896664   DOI
60 Dong, C., Loy, C.C. and Tang, X. (2016), "Accelerating the superresolution convolutional neural network", Proceedings of European Conference on Computer Vision, Amsterdam, The Netherlands, October, pp. 391-407. https://doi.org/10.1007/978-3-319-46475-6_25   DOI
61 Elad, M. and Feuer, A. (2002), "Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images", IEEE Transact. Image Process., 6(12), 1646-1658. https://doi.org/10.1109/83.650118   DOI
62 Finkelstein, S.E., Schrump, D.S., Nguyen, D.M., Hewitt, S.M., Kunst, T.F. and Summers, R.M. (2003), "Comparative evaluation of super high-resolution CT scan and virtual bronchoscopy for the detection of tracheobronchial malignancies", Chest, 124(5), 1834-1840. https://doi.org/10.1378/chest.124.5.1834   DOI
63 Freeman, W.T., Jones, T.R. and Pasztor, E.C. (2002), "Examplebased super-resolution", Comput. Graphics Applicat. IEEE, 22(2), 56-65. https://doi.org/10.1109/38.988747   DOI
64 Yang, J., Wright, J., Huang, T.S. and Ma, Y. (2010), "Image superresolution via sparse representation", IEEE Transact. Image Process., 19(11), 2861-2873. https://doi.org/10.1109/TIP.2010.2050625   DOI
65 Wu, Q., Fan, C., Li, Y., Li, Y. and Hu, J. (2020b), "A novel perceptual loss function for single image super-resolution", Multimedia Tools Applicat., 79(29), 21265-21278. https://doi.org/10.1007/s11042-020-08878-7   DOI
66 Xu, Y. and Brownjohn, J.M. (2018), "Review of machine-vision based methodologies for displacement measurement in civil structures", J. Civil Struct. Health Monitor., 8, 91-110. https://doi.org/10.1007/s13349-017-0261-4   DOI
67 Yan, X., Shen, Q. and Liu, X. (2015), "Super-resolution reconstruction for license plate image in video surveillance system", Proceedings of 2015 10th International Conference on Communications and Networking in China (ChinaCom), Shanghai, China, August, pp. 643-647. https://doi.org/10.1109/CHINACOM.2015.7498016   DOI
68 Kim, J., Lee, J.K. and Lee, K.M. (2016a), "Accurate image superresolution using very deep convolutional networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 1646-1654.
69 Kim, J., Lee, J.K. and Lee, K.M. (2016b), "Deeply-recursive convolutional network for image super-resolution", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 1637-1645.
70 Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012), "Imagenet classification with deep convolutional neural networks", Adv. Neural Inform. Process. Syst., 25.
71 Li, J., Fang, F., Mei, K. and Zhang, G. (2018), "Multi-scale residual network for image super-resolution", Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, September, pp. 527-532.
72 Liu, Y., Dong, Z., Lim, K.P. and Ling, N. (2020a), "A densely connected face super-resolution network based on attention mechanism", Proceedings of 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA), Kristiansand, Norway, November, pp. 148-152. https://doi.org/10.1109/ICIEA48937.2020.9248111   DOI
73 Lee, J.H., Jung, C.Y., Choi, E. and Cheung, J.H. (2017), "Visionbased multipoint measurement systems for structural in-plane and out-of-plane movements including twisting rotation", Smart Struct. Syst., Int. J., 20(5), 563-572. https://doi.org/10.12989/sss.2017.20.5.563   DOI
74 Li, B., Wang, B., Liu, J., Qi, Z. and Shi, Y. (2020), "s-LWSR: Super lightweight super-resolution network", IEEE Transact. Image Process., 29, 8368-8380. https://doi.org/10.1109/TIP.2020.3014953   DOI
75 Lim, B., Son, S., Kim, H., Nah, S. and Mu Lee, K. (2017), "Enhanced deep residual networks for single image superresolution", Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July, pp. 136-144.
76 Rasti, P., Uiboupin, T., Escalera, S. and Anbarjafari, G. (2016), "Convolutional neural network super resolution for face recognition in surveillance monitoring", Proceedings of International Conference on Articulated Motion and Deformable Objects, Palma de Mallorca, Spain, July, pp. 175-184. https://doi.org/10.1007/978-3-319-41778-3_18   DOI
77 Lin, Z., He, J., Tang, X. and Tang, C.K. (2008), "Limits of learning-based superresolution algorithms", Int. J. Comput. Vis., 80(3), 406-420. https://doi.org/10.1007/s11263-008-0148-2   DOI
78 Luo, Y., Zhou, L., Wang, S. and Wang, Z. (2017), "Video satellite imagery super resolution via convolutional neural networks", IEEE Geosci. Remote Sens. Lett., 14(12), 2398-2402. https://doi.org/10.1109/LGRS.2017.2766204   DOI
79 Nguyen, M.Q., Atkinson, P.M. and Lewis, H.G. (2005), "Superresolution mapping using a Hopfield neural network with LIDAR data", IEEE Geosci. Remote Sens. Lett., 2(3), 366-370. https://doi.org/10.1109/LGRS.2005.851551   DOI
80 Ojio, T., Carey, C., O'Brien, E.J., Doherty, C. and Taylor, S.E. (2016), "Contactless bridge weigh-in-motion", J. Bridge Eng., 21, 4016032. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000776   DOI
81 Sheikh, H.R., Bovik, A.C. and De Veciana, G. (2006), "An information fidelity criterion for image quality assessment using natural scene statistics", IEEE Transact. Image Process., 14(12), 2117-2128. https://doi.org/10.1109/TIP.2005.859389   DOI
82 Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D. and Wang, Z. (2016), "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June, pp. 1874-1883.
83 Song, X., Dai, Y. and Qin, X. (2016), "Deep depth superresolution: Learning depth super-resolution using deep convolutional neural network", Proceedings of the 13th Asian Conference on Computer Vision, Taipei, Taiwan, November, pp. 360-376. https://doi.org/10.1007/978-3-319-54190-7_22   DOI