Smart Structures and Systems

Techno-Press (테크노프레스)
권호 표지
DOI QR코드

DOI QR Code

Detection and quantification of bolt loosening using RGB-D camera and Mask R-CNN

  • Chung, Junyeon (Department of Civil and Environmental Engineering, Korea Advanced Institute for Science and Technology) ;
  • Sohn, Hoon (Department of Civil and Environmental Engineering, Korea Advanced Institute for Science and Technology)
  • Received : 2020.08.27
  • Accepted : 2021.01.15
  • Published : 2021.05.25
⟨ Previous Next ⟩

Abstract

Bolt loosening is one of the most common types of damage for bolt-connected plates. Existing vision techniques detect bolt loosening based on the measurement of bolt rotation or the exposure of bolt threads. However, these techniques examine bolt tightness only in a qualitative manner, or require a reference measurement at the initially tightened state of the bolt for quantitative estimation. In this study, the exposed shank length of a bolt is quantitatively measured using an RGB-depth camera and a mask-region-based convolutional neural network but without requiring any measurement from the initial state of the bolt. The performance of the proposed technique is validated by conducting lab-scale experiments, in which the angle and distance of the camera are varied with respect to a target inspection area. The proposed technique successfully detects bolt loosening at exposed shank length over 3 mm with a resolution of 1 mm and 97% accuracy at different camera angles (40°-90°) and distances (up to 65 cm).

Keywords

Acknowledgement

This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. 2019R1A3B3067987).

References

  1. Benkhoui, Y., El Korchi, T. and Reinhold, L. (2019), "UAS-based crack detection using stereo cameras: a comparative study", Proceedings of International Conference on Unmanned Aircraft Systems (ICUAS), Atlanta, GA, USA, June.
  2. Cha, Y.-J., You, K. and Choi, W. (2016), "Vision-based detection of loosened bolts using the Hough transform and support vector machines", Autom. Constr., 71(2), 181-188. https://doi.org/10.1016/j.autcon.2016.06.008
  3. Dubois, E. and Sabri, S. (1984), "Noise reduction in image sequences using motion-compensated temporal filtering", IEEE Trans. Commun., 32(7), 826-831. https://doi.org/10.1109/TCOM.1984.1096143
  4. Fischler, M.A. and Bolles, R.C. (1981), "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography", Commun. ACM, 24(6), 381-395. https://doi.org/10.1145/358669.358692
  5. Grunnet-Jepsen, A., Sweetser, J.N., Winer, P., Takagi, A. and Woodfill, J. (2018), "Projectors for Intel® RealSenseTM Depth Cameras D4xx", Intel.
  6. Hartley, R. and Zisserman, A. (2003), Multiple View Geometry in Computer Vision, Cambridge University Press, Cambridge, United Kingdom.
  7. He, K., Zhang, X., Ren, S. and Sun, J. (2016), "Deep residual learning for image recognition", IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June.
  8. He, K., Gkioxari, G., Dollar, P. and Girshick, R. (2017), "Mask R-CNN", IEEE International Conference on Computer Vision, Venice, Italy, October.
  9. Huynh, T.-C. and Kim, J.-T. (2017), "Quantification of temperature effect on impedance monitoring via PZT interface for prestressed tendon anchorage", Smart Mater. Struct., 26(12), 125004. https://doi.org/10.1088/1361-665X/aa931b
  10. Huynh, T. and Kim, J. (2018), "RBFN-based temperature compensation method for impedance monitoring in prestressed tendon anchorage", Struct. Control Health Monit., 25(6), e2173. https://doi.org/10.1002/stc.2173
  11. Huynh, T.-C., Dang, N.-L. and Kim, J.-T. (2018), "Preload monitoring in bolted connection using piezoelectric-based smart interface", Sensors, 18(9), 2766. https://doi.org/10.3390/s18092766
  12. Huynh, T.-C., Park, J.-H., Jung, H.-J. and Kim, J.-T. (2019), "Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing", Autom. Constr., 105, 102844. https://doi.org/10.1016/j.autcon.2019.102844
  13. Korea Expressway Corporation (2013), "Improvement of bridge inspection system by the damage analysis", Korea Expressway Corporation.
  14. Lin, T.-Y., Maire, M., Belongie, S., Hays, J, Perona, P., Ramanan, D., Dollar, P. and Zitnick C.L. (2014), "Microsoft COCO: common objects in context", Proceedings of European Conference on Computer Vision, Zurich, Switzerland, September.
  15. Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B. and Belongie, S. (2017), "Feature pyramid networks for object detection", Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July.
  16. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S. and Cheng-Yang, F. (2016), "SSD: single shot multibox detector", Proceedings of European Conference on Computer Vision, Amsterdam, The Netherlands, October.
  17. Park, J.-H., Huynh, T.-C., Choi, S.-H. and Kim, J.-T. (2015), "Vision-based technique for bolt-loosening detection in wind turbine tower", Wind Struct., Int. J., 21(6), 709-726. https://doi.org/10.12989/was.2015.21.6.709
  18. Ramana, L., Choi, W. and Cha, Y.-J. (2019), "Fully automated vision-based loosened bolt detection using the Viola-Jones algorithm", Struct. Health Monit., 18(2), 422-434. https://doi.org/10.1177/1475921718757459
  19. Redmon, J. and Farhadi, A. (2018), "YOLOv3: An incremental improvement", arXiv Prepr. arXiv1804.02767.
  20. Ren, S., He, K., Girshick, R. and Sun, J. (2015), "Faster R-CNN: towards real-time object detection with region proposal networks", Neural Inform. Process. Syst., Montreal, Canada, December.
  21. Simonyan, K. and Zisserman, A. (2014), "Very deep convolutional networks for large-scale image recognition", arXiv Prepr. arXiv1409.1556.
  22. Suda, M., Hasuo, Y., Kanaya, A., Ogura, Y. Takishita, T. and Suzuki, Y. (1992), "Development of ultrasonic axial bolting force inspection system for turbine bolts in thermal power plants", JSME Int. J. Ser. 1, Solid Mech. Strength Mater., 35(2), 216-219. https://doi.org/10.1299/jsmea1988.35.2_216
  23. Torrey, L. and Shavlik, J. (2010), "Transfer learning", in Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, IGI Global, PA, USA.
  24. Van Dyk, D.A. and Meng, X.-L. (2001), "The art of data augmentation", J. Comput. Graph. Stat., 10(1), 1-50. https://doi.org/10.1198/10618600152418584
  25. Wang, T., Song, G., Liu, S., Li, Y. and Xiao, H. (2013a), "Review of bolted connection monitoring", Int. J. Distrib. Sens. Netw., 9(12), 871213. https://doi.org/10.1155/2013/871213
  26. Wang, T., Song, G., Wang, Z. and Li, Y. (2013b), "Proof-of-concept study of monitoring bolt connection status using a piezoelectric based active sensing method", Smart Mater. Struct., 22(8), 87001. https://doi.org/10.1088/0964-1726/22/8/087001
  27. Zhang, Y., Sun, X., Loh, K.J., Su, W., Xue, Z. and Zhao, X. (2019), "Autonomous bolt loosening detection using deep learning", Struct. Health Monit., 19(1), 105-122. https://doi.org/10.1177/1475921719837509
  28. Zhao, X., Zhang, Y. and Wang, N. (2019), "Bolt loosening angle detection technology using deep learning", Struct. Control Health Monit., 26(1), e2292. https://doi.org/10.1002/stc.2292