Journal of the Korea institute for structural maintenance and inspection (한국구조물진단유지관리공학회 논문집)

Korea Institute for Structural Maintenance Inspection (한국구조물진단유지관리공학회)
권호 표지
DOI QR코드

DOI QR Code

Development of Fender Segmentation System for Port Structures using Vision Sensor and Deep Learning

비전센서 및 딥러닝을 이용한 항만구조물 방충설비 세분화 시스템 개발

  • 민지영 (한국건설기술연구원 구조연구본부) ;
  • 유병준 ((주)스트라드비전) ;
  • 김종혁 (한밭대학교 건설환경공학과) ;
  • 전해민 (한밭대학교 건설환경공학과)
  • Received : 2021.12.29
  • Accepted : 2022.03.21
  • Published : 2022.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

As port structures are exposed to various extreme external loads such as wind (typhoons), sea waves, or collision with ships; it is important to evaluate the structural safety periodically. To monitor the port structure, especially the rubber fender, a fender segmentation system using a vision sensor and deep learning method has been proposed in this study. For fender segmentation, a new deep learning network that improves the encoder-decoder framework with the receptive field block convolution module inspired by the eccentric function of the human visual system into the DenseNet format has been proposed. In order to train the network, various fender images such as BP, V, cell, cylindrical, and tire-types have been collected, and the images are augmented by applying four augmentation methods such as elastic distortion, horizontal flip, color jitter, and affine transforms. The proposed algorithm has been trained and verified with the collected various types of fender images, and the performance results showed that the system precisely segmented in real time with high IoU rate (84%) and F1 score (90%) in comparison with the conventional segmentation model, VGG16 with U-net. The trained network has been applied to the real images taken at one port in Republic of Korea, and found that the fenders are segmented with high accuracy even with a small dataset.

매립지 위에 건설되는 항만시설물은 바람(태풍), 파랑, 선박과의 충돌 등 극한 외부 하중에 노출되기 때문에 구조물의 안전성 및 사용성을 주기적으로 평가하는 것이 중요하다. 본 논문에서는 항만 계류시설에 설치된 방충설비의 유지관리를 위하여 비전 및 딥러닝 기반의 방충설비 세분화(segmentation) 시스템을 개발하였다. 방충설비 세분화를 위하여 인코더-디코더 형식과 인간 시각체계의 편심 기능에서 영감을 얻은 수용 영역 블록(Receptive field block) 기반의 합성곱 모듈을 DenseNet 형식으로 개선하는 딥러닝 네트워크를 제안하였다. 네트워크 훈련을 위해 BP형, V형, 원통형, 타이어형 등 다양한 형태의 방충설비 영상을 수집하였으며, 탄성 변형, 좌우 반전, 색상 변환 및 기하학적 변환을 통해 영상을 증강시킨 다음 제안한 딥러닝 네트워크를 학습하였다. 기존의 세분화 모델인 VGG16-Unet과 비교하여 제안한 모델의 세분화 성능을 검증하였으며, 그 결과 본 시스템이 IoU 84%, 조화평균 90% 이상으로 정밀하게 실시간으로 세분화할 수 있음을 확인하였다. 제안한 방충설비 세분화 시스템의 현장적용 가능성을 검증하기 위하여 국내 항만 시설물에서 촬영된 영상을 기반으로 학습을 수행하였으며, 그 결과 기존 세분화 모델과 비교하였을 때 우수한 성능을 보이며 정밀하게 방충설비를 감지하는 것을 확인하였다.

Keywords

Acknowledgement

본 논문은 2021년 해양수산부 재원으로 해양수산과학기술진흥원(과제번호20210659)의 지원을 받아 수행되었으며, 이에 감사드립니다. 또한 항 내 시설물 촬영 및 지원에 협조해주신 ◯◯항만공사에 감사드립니다.

References

  1. Ministry of Land, Infrastructure and Transport (2018), Detailed guidelines for safety and maintenance of facilities (performance evaluation), in Korean.
  2. Ministry of Oceans and Fisheries (2018), Harbor and fishing port fender maintenance manual, in Korean.
  3. Ministry of Oceans and Fisheries (2015), Detailed guidelines for safety inspection of port facilities, in Korean.
  4. Ministry of Oceans and Fisheries (2020), Port facility maintenance and mid- to long-term road map establishment project, in Korean
  5. Sakakibara, S. and Kubo M., Ship berthing and mooring monitoring system by pneumatic-type fenders, Ocean engineering, Elsevier, 34(8-9), 2007, 1174-1181. https://doi.org/10.1016/j.oceaneng.2006.07.006
  6. Yamada, S., Sakakibara, S., Miyamoto, M., Nakatani K. (2012), Final acceptance test on advanced ship maneuvering and mooring support system at ship-to-ship transfer operations by fender monitoring system of pneumatic fenders, The twenty-second International offshore and polar engineering conference, Rhodes, 901-908.
  7. Girshick, R., Donahue, J., Darrell, T. and Malik, J. (2014), Rich feature hierarchies for accurate object detection and semantic segmentation, Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, 580-587.
  8. Girshick, R. (2015), Fast r-cnn, Proceedings of the IEEE international conference on computer vision, Santiago, 1440-1448.
  9. Ren, S., He, K., Girshick, R. and Sun, J. (2015), Faster r-cnn: Towards real-time object detection with region proposal networks, Advances in neural information processing systems, Montreal, 91-99.
  10. Redmon, J., Divvala, S., Girshick, R. and Farhadi, A. (2016), You only look once: Unified, real-time object detection, Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, 779-788.
  11. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y. and Berg, A.C. (2016), SSD: Single shot multibox detector, European conference on computer vision, Amsterdam, 21-37.
  12. He, K., Gkioxari, G., Dollar, P. and Girshick, R. (2017), Mask r-cnn, Proceedings of the IEEE international conference on computer vision, Venice, 2961-2969.
  13. Long, J., Shelhamer, E. and Darrell, T. (2015), Fully convolutional networks for semantic segmentation, Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, 3431-3440.
  14. Ronneberger, O., Fischer, P. and Brox, T. (2015), U-Net: Convolutional Networks for Biomedical Image Segmentation, Proceedings of the International conference on medical image computing and computer-assisted intervention, Munich, 234-241.
  15. Dung, C. V. and Anh L. D., Autonomous concrete crack detection using deep fully convolutional neural network, Automation in Construction, 99, 2019, 52-58. https://doi.org/10.1016/j.autcon.2018.11.028
  16. Islam, M.M.M. and Kim, J.-M., Vision-based autonomous crack detection of concrete structures using a fully convolutional encoder-decoder network. Sensors, 2019, 19, 4251. https://doi.org/10.3390/s19194251
  17. Dong, C., Li, L., Yan, J., Zhang, Z., Pan, H., and Catbas, F.N., Pixel-level fatigue crack segmentation in large-scale images of steel structures using an encoder-decoder network. Sensors, 2021, 21, 4135.
  18. Liu, S. and Huang, D. (2018), Receptive field block net for accurate and fast object detection, Proceedings of the European Conference on Computer Vision (ECCV), Munich, 385-400.
  19. Wang, H., Su, D., Liu, C., Jin, L., Sun, X. and Peng, X (2019), Deformable non-local network for video super-resolution, IEEE Access, 7, 177734-177744. https://doi.org/10.1109/access.2019.2958030
  20. Simonyan, K. and Zisserman, A (2015), Very deep convolutional networks for large-scale image recognition, Proceedings of the 3rd International conference on leaning representations, vol. abs/1409.1556v6. [Online].Available: https://arxiv.org/abs/1409.1556