Browse > Article
http://dx.doi.org/10.5345/JKIBC.2021.21.5.397

Extraction of Workers and Heavy Equipment and Muliti-Object Tracking using Surveillance System in Construction Sites  

Cho, Young-Woon (Construction Engineering and Management Institute, Sahmyook University)
Kang, Kyung-Su (Construction Engineering and Management Institute, Sahmyook University)
Son, Bo-Sik (Department of Architectural Engineering, Namseoul University)
Ryu, Han-Guk (Department of Architectural, Sahmyook University)
Publication Information
Journal of the Korea Institute of Building Construction / v.21, no.5, 2021 , pp. 397-408 More about this Journal
Abstract
The construction industry has the highest occupational accidents/injuries and has experienced the most fatalities among entire industries. Korean government installed surveillance camera systems at construction sites to reduce occupational accident rates. Construction safety managers are monitoring potential hazards at the sites through surveillance system; however, the human capability of monitoring surveillance system with their own eyes has critical issues. A long-time monitoring surveillance system causes high physical fatigue and has limitations in grasping all accidents in real-time. Therefore, this study aims to build a deep learning-based safety monitoring system that can obtain information on the recognition, location, identification of workers and heavy equipment in the construction sites by applying multiple object tracking with instance segmentation. To evaluate the system's performance, we utilized the Microsoft common objects in context and the multiple object tracking challenge metrics. These results prove that it is optimal for efficiently automating monitoring surveillance system task at construction sites.
Keywords
construction safety; computer vision; instance segmentation; multiple object tracking; surveillance system;
Citations & Related Records
연도 인용수 순위
  • Reference
1 GitHub: Where the world builds software [Internet]. Deep learning-based Computer Vision Models for PyTorch: GitHub, Inc. 2008 - [cited 2021 Apr 7]. Available from: https://github.com/unerue/boda
2 Xuehui A, Li Z, Zuguang L, Chengzhi W, Pengfei L, Zhiwei L. Dataset and benchmark for detecting moving objects in construction sites. Automation in Construction. 2021 Feb;122:103482. https://doi.org/10.1016/j.autcon.2020.103482   DOI
3 Kuhn HW. The Hungarian method for the assignment problem. Naval research logistics quarterly. 1955 Mar;2(1-2):83-97. https://doi.org/10.1002/nav.3800020109   DOI
4 Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015 Oct 5; Munich, Germany. MN: The Medical Image Computing and Computer Assisted Intervention Society; 2015. p. 234-41. https://doi.org/10.1007/978-3-319-24574-4_28   DOI
5 Angah O, Chen AY. Tracking multiple construction workers through deep learning and the gradient based method with rematching based on multi-object tracking accuracy. Automation in Construction. 2020 Nov;119:103308. https://doi.org/10.1016/j.autcon.2020.103308   DOI
6 He K, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV); 2017 Oct 22-29; Venice, Italy. NJ: Institute of Electrical and Electronics Engineers; 2017. p. 2961-9. https://doi.org/10.1109/ICCV.2017.322   DOI
7 Nath ND, Behzadan AH, Paal SG. Deep learning for site safety: Real-time detection of personal protective equipment. Automation in Construction. 2020 Apr;112:103085. https://doi.org/10.1016/j.autcon.2020.103085   DOI
8 Son H, Choi H, Seong H, Kim C. Detection of construction workers under varying poses and changing background in image sequences via very deep residual networks. Automation in Construction. 2019 Mar;99:27-38. https://doi.org/10.1016/j.autcon.2018.11.033   DOI
9 Kim D. Occupational accident/injury analysis 2009. Ulsan (Korea): Korea Occupational Safety and Health Agency; 2021 Jan;15-22. Grant No.: 118006 Supported by KOSTAT.
10 Heejung. Women who "watch the monitor" [Internet]. Seoul (Korea): Ildaro. 2019 Aug 30 [cited 2021 Apr 7]. Available from: https://ildaro.com/8536
11 Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017 Apr;39(4): 640-51. https://doi.org/10.1109/TPAMI.2016.2572683   DOI
12 Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence. 2017 Jun;39(6):1137-49. https://doi.org/10.1109/TPAMI.2016.2577031   DOI
13 Guo Y, Xu Y, Li S. Dense construction vehicle detection based on orientation-aware feature fusion convolutional neural network. Automation in Construction. 2020 Apr;112:103124. https://doi.org/10.1016/j.autcon.2020.103124   DOI
14 Li Z, Zhou F. FSSD: feature fusion single shot multibox detector. arXiv:1712.00960 [Preprint]. 2017 [cited 2021 Apr 12]. Available from: https://arxiv.org/abs/1712.00960
15 Voigtlaender P, Krause M, Osep A, Luiten J, Sekar BB, Geiger A, Leibe B. Mots: Multi-object tracking and segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). 2019 June 15-20; Long Beach, CA. NJ: Institute of Electrical and Electronics Engineers; 2020. p. 7942-51. https://doi.org/10.1109/CVPR.2019.00813   DOI
16 Truong T, Bhatt A, Queiroz L, Lai K, Yanushkevich S. Instance segmentation of personal protective equipment using a multi-stage transfer learning process. 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2020 Oct 11-14; Toronto, Canada. NJ: Institute of Electrical and Electronics Engineers; 2017. p.1181-6. https://doi.org/10.1109/SMC42975.2020.9283427   DOI
17 Yang Z, Yuan Y, Zhang M, Zhao X, Zhang Y, Tian B. Safety distance identification for crane drivers based on mask R-CNN. Sensors. 2019 Jan;19(12):2789. https://doi.org/10.3390/s19122789   DOI
18 GitHub: Where the world builds software [Internet]. Image Polygonal Annotation with Python: GitHub, Inc. 2008 - [cited 2021 Apr 7]. Available from: https://github.com/wkentaro/labelme
19 Bolya D, Zhou C, Xiao F, Lee YJ. Yolact: Real-time instance segmentation. 2019 IEEE/CVF International Conference on Computer Vision(ICCV). 2019 Oct 27-Nov 2; Seoul, Korea. NJ: Institute of Electrical and Electronics Engineers; 2020. p.9157-66. https://doi.org/10.1109/ICCV.2019.00925   DOI
20 Park Y. Only one person monitors 438 CCTVs [Internet]. Seoul (Korea): Munhwa Ilbo. 2017 Nov 28 [cited 2021 Apr 7]. Available from: http://www.munhwa.com/news/view.html?no=2017112801031627109001
21 Lin TY, Dollar P, Girshick R, He K, Hariharan B, Belongie S. Feature pyramid networks for object detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). 2017 Jul 21-26; Honolulu, HI. NJ: Institute of Electrical and Electronics Engineers; 2017. p. 2117-25. https://doi.org/10.1109/CVPR.2017.106   DOI
22 Luo W, Xing J, Milan A, Zhang X, Liu W, Kim TK. Multiple object tracking: A literature review. Artificial Intelligence. 2021 Apr 293:103448. https://doi.org/10.1016/j.artint.2020.103448   DOI
23 Bewley A, Ge Z, Ott L, Ramos F, Upcroft B. Simple online and realtime tracking. 2016 IEEE international conference on image processing(ICIP). 2016 Sept 25-28; Phoenix, AZ. NJ: Institute of Electrical and Electronics Engineers; 2016. p. 3464-8. https://doi.org/10.1109/ICIP.2016.7533003   DOI
24 Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A. The pascal visual object classes (voc) challenge. International journal of computer vision. 2010 Jun;88(2):303-38. https://doi.org/10.1007/s11263-009-0275-4   DOI
25 Lee YJ, Park MW. 3D tracking of multiple onsite workers based on stereo vision. Automation in Construction. 2019 Feb;98:146-59. https://doi.org/10.1016/j.autcon.2018.11.017   DOI
26 Kim H. Construction safety innovation plan: Reinforcement of management of vulnerable construction, etc [Internet]. Sejong (Korea): Ministry of Land, Infrastructure and Transport. 2020 Apr 24 [cited 2021 Apr 7]. Available from: http://www.molit.go.kr/USR/NEWS/m_71/dtl.jsp?id=95083805
27 Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollar P, Zitnick CL. Microsoft coco: Common objects in context. European conference on computer vision. 2014 Sep;8693:740-55. https://doi.org/10.1007/978-3-319-10602-1_48   DOI
28 LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD. Backpropagation applied to handwritten zip code recognition. Neural computation. 1989 Dec;1(4):541-51. https://doi.org/10.1162/neco.1989.1.4.541   DOI
29 Dalal N, Triggs B. Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition; 2005 Jun 20-25; San Diego, CA. NJ: Institute of Electrical and Electronics Engineers; 2005. p. 886-93. https://doi.org/10.1109/CVPR.2005.177   DOI
30 Park MW, Brilakis I. Continuous localization of construction workers via integration of detection and tracking. Automation in Construction. 2016 Dec;72:129-42. https://doi.org/10.1016/j.autcon.2016.08.039   DOI
31 Ishioka H, Weng X, Man Y, Kitani K. Single camera worker detection, tracking and action recognition in construction site. Proceedings of the 37th International Symposium on Automation and Robotics in Construction (ISARC); 2020 Oct; Kitakyushu, Japan. FL: International Association for Automation and Robotics in Construction (IAARC); 2020. p. 653-60. https://doi.org/10.22260/ISARC2020/0092   DOI
32 Choi M, Choi J. CCTV integrated control center operation status and improvement plan legislative policy report. Seoul, Korea: National Assembly Research Service, NARS; 2019. p. 1-33.
33 Zhang Z. Determining the epipolar geometry and its uncertainty: A review. International journal of computer vision. 1998 Mar;27(2):161-95. https://doi.org/10.1023/A:1007941100561   DOI
34 Zhao Y, Chen Q, Cao W, Yang J, Xiong J, Gui G. Deep learning for risk detection and trajectory tracking at construction sites. IEEE Access; 2019 Mar;7:30905-12. https://doi.org/10.1109/ACCESS.2019.2902658   DOI
35 Redmon J, Farhadi A. Yolov3: An incremental improvement. arXiv:1804:02767 [Preprint]. 2018 [cited 2021 Apr 8]. Available from: https://arxiv.org/abs/1804.02767
36 Kalman RE. A new approach to linear filtering and prediction problems. 1960 Mar;82(1):35-45. https://doi.org/10.1115/1.3662552   DOI
37 Wojke N, Bewley A, Paulus D. Simple online and realtime tracking with a deep association metric. 2017 IEEE international conference on image processing(ICIP). 2017 Sep 17-20; Beijing, China. NJ: Institute of Electrical and Electronics Engineers; 2018. p. 3645-9. https://doi.org/10.1109/ICIP.2017.8296962   DOI
38 Leal-Taixe L, Milan A, Reid I, Roth S, Schindler K. Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv:1504.01942 [Preprint]. 2015 [cited 2021 Apr 8]. Available from: https://arxiv.org/abs/1504.01942
39 Milan A, Leal-Taixe L, Reid I, Roth S, Schindler K. MOT16: A benchmark for multi-object tracking. arXiv:1603.00831 [Preprint]. 2016 [cited 2021 Apr 8]. Available from: https://arxiv.org/abs/1603.00831