Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Real-time geometry identification of moving ships by computer vision techniques in bridge area

  • Li, Shunlong (School of Transportation Science and Engineering, Harbin Institute of Technology) ;
  • Guo, Yapeng (School of Transportation Science and Engineering, Harbin Institute of Technology) ;
  • Xu, Yang (School of Civil Engineering, Harbin Institute of Technology) ;
  • Li, Zhonglong (School of Transportation Science and Engineering, Harbin Institute of Technology)
  • Received : 2018.09.09
  • Accepted : 2019.03.11
  • Published : 2019.04.25
⟨ Previous Next ⟩

Abstract

As part of a structural health monitoring system, the relative geometric relationship between a ship and bridge has been recognized as important for bridge authorities and ship owners to avoid ship-bridge collision. This study proposes a novel computer vision method for the real-time geometric parameter identification of moving ships based on a single shot multibox detector (SSD) by using transfer learning techniques and monocular vision. The identification framework consists of ship detection (coarse scale) and geometric parameter calculation (fine scale) modules. For the ship detection, the SSD, which is a deep learning algorithm, was employed and fine-tuned by ship image samples downloaded from the Internet to obtain the rectangle regions of interest in the coarse scale. Subsequently, for the geometric parameter calculation, an accurate ship contour is created using morphological operations within the saturation channel in hue, saturation, and value color space. Furthermore, a local coordinate system was constructed using projective geometry transformation to calculate the geometric parameters of ships, such as width, length, height, localization, and velocity. The application of the proposed method to in situ video images, obtained from cameras set on the girder of the Wuhan Yangtze River Bridge above the shipping channel, confirmed the efficiency, accuracy, and effectiveness of the proposed method.

Keywords

Acknowledgement

Supported by : National Natural Science Foundation of China (NSFC)

References

  1. Andrew, A.M. (2004), "Multiple view geometry in computer vision", Kybernetes, 30(9-10), 1865-1872.
  2. Bentes, C., Velotto, D. and Tings, B. (2017), "Ship classification in TerraSAR-X images with convolutional neural networks", IEEE J. Oceanic Eng., 43(1), 1-9. https://doi.org/10.1109/JOE.2017.2778479
  3. Celik, O., Dong, C.Z. and Catbas, F.N. (2018), "A computer vision approach for the load time history estimation of lively individuals and crowds", Comput. Struct., 200, 32-52. https://doi.org/10.1016/j.compstruc.2018.02.001
  4. Celik, O., Dong, C.Z. and Catbas, F.N. (2019), Measurement of Human Loads Using Computer Vision, Springer.
  5. Chen, Z., Li, H., Bao, Y., Li, N. and Jin, Y. (2016), "Identification of spatio-temporal distribution of vehicle loads on long-span bridges using computer vision technology", Struct. Control Health Monit., 23(3), 517-534. https://doi.org/10.1002/stc.1780
  6. Chen, Z.Q. and Hutchinson, T.C. (2010), "Image-based framework for concrete surface crack monitoring and quantification", Adv. Civil Eng., 2010.
  7. Chi, S. and Caldas, C.H. (2011), "Automated object identification using optical video cameras on construction sites", Comput.-Aided Civil Infrastruct. Eng., 26(5), 368-380. https://doi.org/10.1111/j.1467-8667.2010.00690.x
  8. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. and Fei-Fei, L. (2009), "Imagenet: A large-scale hierarchical image database", Computer Vision and Pattern Recognition, 2009, Florida, USA, June.
  9. Eum, H., Yoon, C., Lee, H. and Park, M. (2015), "Continuous human action recognition using depth-MHI-HOG and a spotter model", Sensors, 15(3), 5197-5227. https://doi.org/10.3390/s150305197
  10. German, S., Jeon, J.S., Zhu, Z., Bearman, C., Brilakis, I., Desroches, R. and Lowes, L. (2013), "Machine Vision-Enhanced Postearthquake Inspection", J. Comput. Civil Eng., 27(6), 622-634. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000333
  11. Girshick, R. (2015), "Fast R-CNN", Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, USA, June.
  12. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, USA, June.
  13. Goerlandt, F., Montewka, J., Kuzmin, V. and Kujala, P. (2015), "A risk-informed ship collision alert system: Framework and application", Safety Sci., 77(1), 182-204. https://doi.org/10.1016/j.ssci.2015.03.015
  14. Gu, C., Lim, J.J., Arbelaez, P. and Malik, J. (2009), "Recognition using regions", Computer Vision and Pattern Recognition 2009, Florida, USA, June.
  15. He, K., Gkioxari, G., Dollar, P. and Girshick, R. (2017). "Mask rcnn", Computer Vision (ICCV), 2017 IEEE International Conference on Computer Vision, Venice, Italy, October.
  16. Hoskere, V., Narazaki, Y., Hoang, T. and Spencer Jr., B.F. (2018), "Vision-based structural inspection using multiscale deep convolutional neural networks", arXiv preprint arXiv:1805.01055.
  17. Hoskere, V., Narazaki, Y., Hoang, T.A. and Spencer Jr, B.F. (2018), "Towards automated post-earthquake inspections with deep learning-based condition-aware models", arXiv preprint arXiv:1809.09195.
  18. Huang, C.L. and Ma, H.N. (2012), "A moving object detection algorithm for vehicle localization", Proceedings of the 2012 Sixth International Conference on Genetic and Evolutionary Computing.
  19. Hyukmin, E., Jaeyun, B., Changyong, Y. and Euntai, K. (2015), "Ship detection using edge-based segmentation and histogram of oriented gradient with ship size ratio", Int. J. Fuzzy Log. Intell. Syst., 15(4), 251-259. https://doi.org/10.5391/IJFIS.2015.15.4.251
  20. Jahanshahi, M., Kelly, J., Masri, S. and Sukhatme, G. (2009), "A survey and evaluation of promising approaches for automatic image-based defect detection of bridge structures", Struct. Infrastruct. Eng., 5(6), 455-486. https://doi.org/10.1080/15732470801945930
  21. Kong, X. and Li, J. (2018), "Automated fatigue crack identification through motion tracking in a video stream", Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2018.
  22. Kulchandani, J.S. and Dangarwala, K.J. (2015), "Moving object detection: Review of recent research trends", Proceedings of the International Conference on Pervasive Computing, St. Louis, Missouri, USA, March.
  23. Lecun, Y., Bengio, Y. and Hinton, G. (2015), "Deep learning", Nature, 521(7553), 436. https://doi.org/10.1038/nature14539
  24. Lee, B.J., Shin, D.H., Seo, J.W., Jung, J.D. and Lee, J.Y. (2011). "Intelligent bridge inspection using remote controlled robot and image processing technique", Isarc Proceedings, Seoul, Korea, June.
  25. Li, S., Zhu, S., Xu, Y.L., Chen, Z.W. and Li, H. (2012), "Longterm condition assessment of suspenders under traffic loads based on structural monitoring system: Application to the Tsing Ma Bridge", Struct. Control Health Monit., 19(1), 82-101. https://doi.org/10.1002/stc.427
  26. Lin, C.W., Hsu, W.K., Chiou, D.J., Chen, C.W. and Chiang, W.L. (2015), "Smart monitoring system with multi-criteria decision using a feature based computer vision technique", Smart Struct. Syst., 15(6), 1583-1600. https://doi.org/10.12989/sss.2015.15.6.1583
  27. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y. and Berg, A.C. (2016), "SSD: Single shot multibox detector", European Conference on Computer Vision, Amsterdam, The Netherlands, October.
  28. Liu, Y., Cho, S., Spencer, B.F.J. and Fan, J. (2014), "Automated assessment of cracks on concrete surfaces using adaptive digital image processing", Smart Struct. Syst., 14(4), 719-741. https://doi.org/10.12989/sss.2014.14.4.719
  29. Liu, Z., Zhou, F., Bai, X. and Yu, X. (2013), "Automatic detection of ship target and motion direction in visual images", Int. J. Electronics, 100(1), 94-111. https://doi.org/10.1080/00207217.2012.687188
  30. Makantasis, K., Protopapadakis, E., Doulamis, A., Doulamis, N. and Loupos, C. (2015), "Deep convolutional neural networks for efficient vision based tunnel inspection", Proceedings of the IEEE International Conference on Intelligent Computer Communication and Processing, Huston, TX, USA, April.
  31. Narazaki, Y., Hoskere, V., Hoang, T.A. and Spencer Jr., B.F. (2018a), "Automated vision-based bridge component extraction using multiscale convolutional neural networks", arXiv preprint arXiv:1805.06042.
  32. Narazaki, Y., Hoskere, V., Hoang, T.A. and Spencer Jr,. B.F. (2018b), "Automated bridge component recognition using video data", arXiv preprint arXiv:1806.06820.
  33. Oh, J.K., Jang, G., Oh, S., Lee, J.H., Yi, B.J., Moon, Y.S., Lee, J.S. and Choi, Y. (2009), "Bridge inspection robot system with machine vision", Automat Constr., 18(7), 929-941. https://doi.org/10.1016/j.autcon.2009.04.003
  34. Otsu, N. (1979), "A threshold selection method from gray-level histograms", IEEE T Syst. Man Cy, 9(1), 62-66. https://doi.org/10.1109/TSMC.1979.4310076
  35. Ou, J. and Li, H. (2010), "Structural health monitoring in mainland China: Review and future trends", Struct. Health Monit., 9(3), 219-231. https://doi.org/10.1177/1475921710365269
  36. Pan, S.J. and Yang, Q. (2010), "A survey on transfer learning", IEEE T. Knowledge Data Eng., 22(10), 1345-1359. https://doi.org/10.1109/TKDE.2009.191
  37. Redmon, J., Divvala, S., Girshick, R. and Farhadi, A. (2016), "You only look once: Unified, real-time object detection", Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, USA, June.
  38. Ren, S., Girshick, R., Girshick, R. and Sun, J. (2015), "Faster RCNN: Towards Real-Time Object Detection with Region Proposal Networks", IEEE T. Pattern Anal. Machine Intell., 39(6), 1137-1149. https://doi.org/10.1109/TPAMI.2016.2577031
  39. Rowley, H.A., Baluja, S. and Kanade, T. (1998), "Rotation invariant neural network-based face detection", IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Santa Barbara, CA, USA, June.
  40. Sermanet, P., Kavukcuoglu, K., Chintala, S. and Lecun, Y. (2013), "Pedestrian detection with unsupervised multi-stage feature learning", Computer Vision and Pattern Recognition 2013, Portland, Oregon, USA, June.
  41. Simonyan, K. and Zisserman, A. (2014), "Very deep convolutional networks for large-scale image recognition", arXiv preprint arXiv:1409.1556.
  42. Sinha, S.K. and Fieguth, P.W. (2006), "Automated detection of cracks in buried concrete pipe images", Automat Constr., 15(1), 58-72. https://doi.org/10.1016/j.autcon.2005.02.006
  43. Steger, C., Ulrich, M. and Wiedemann, C. (2018), Machine vision algorithms and applications, John Wiley & Sons.
  44. Stockman, G. and Shapiro, L.G. (2001), Computer Vision, Prentice Hall, Upper Saddle River, New Jersey, USA.
  45. Szeliski, R. (2010), Computer vision: algorithms and applications, Springer Science & Business Media, Berlin, Germany.
  46. Vaillant, R., Monrocq, C. and Cun, Y.L. (1994), "Original approach for the localisation of objects in images", Vision, Image and Signal Processing, IEE Proceedings, 141(4), 245-250. https://doi.org/10.1049/ip-vis:19941301
  47. Wang, X. (2011), "Ship target detection and tracking in cluttered infrared imagery", Opt. Eng., 50(5), 057207-057207-057212. https://doi.org/10.1117/1.3578402
  48. Xu, Y., Bao, Y., Chen, J., Zuo, W. and Li, H. (2018), "Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumergrade camera images", Struct. Health Monit., 1475921718764873.
  49. Xu, Y., Li, S., Zhang, D., Jin, Y., Zhang, F., Li, N. and Li, H. (2017), "Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images", Struct. Control Health Monit., 25(2), e2075. https://doi.org/10.1002/stc.2075
  50. Yang, Y., Dorn, C., Mancini, T., Talken, Z., Kenyon, G., Farrar, C. and Mascarenas, D. (2017), "Blind identification of full-field vibration modes from video measurements with phase-based video motion magnification", Mech. Syst. Signal Pr., 85, 567-590. https://doi.org/10.1016/j.ymssp.2016.08.041
  51. Yao, Y., Jiang, Z. and Zhao, D. (2017), "Ship detection in optical remote sensing images based on deep convolutional neural networks", J. Appl. Remote Sens.. 11(4), 1.
  52. Ye, X.W., Dong, C.Z. and Liu, T. (2016), "Image-based structural dynamic displacement measurement using different multi-object tracking algorithms", Smart Struct. Syst., 17(6), 935-956. https://doi.org/10.12989/sss.2016.17.6.935
  53. Ye, X.W., Ni, Y.Q., Wai, T.T., Wong, K.Y., Zhang, X.M. and Xu, F. (2013), "A vision-based system for dynamic displacement measurement of long-span bridges: Algorithm and verification", Smart Struct. Syst., 12(3-4), 363-379. https://doi.org/10.12989/sss.2013.12.3_4.363
  54. Yeum, C.M., Dyke, S.J. and Ramirez, J. (2018), "Visual data classification in post-event building reconnaissance", Eng. Struct., 155, 16-24. https://doi.org/10.1016/j.engstruct.2017.10.057
  55. Zhu, D., Feng, Y., Chen, Q. and Cai, J. (2010), "Image recognition technology in rotating machinery fault diagnosis based on artificial immune", Smart Struct. Syst., 6(4), 389-403. https://doi.org/10.12989/sss.2010.6.4.389