Structural Crack Detection Using Deep Learning: An In-depth Review

Abstract

Crack detection in structures plays a vital role in ensuring their safety, durability, and reliability. Traditional crack detection methods sometimes need significant manual inspections, which are laborious, expensive, and prone to error by humans. Deep learning algorithms, which can learn intricate features from large-scale datasets, have emerged as a viable option for automated crack detection recently. This study presents an in-depth review of crack detection methods used till now, like image processing, traditional machine learning, and deep learning methods. Specifically, it will provide a comparative analysis of crack detection methods using deep learning, aiming to provide insights into the advancements, challenges, and future directions in this field. To facilitate comparative analysis, this study surveys publicly available crack detection datasets and benchmarks commonly used in deep learning research. Evaluation metrics employed to check the performance of different models are discussed, with emphasis on accuracy, precision, recall, and F1-score. Moreover, this study provides an in-depth analysis of recent studies and highlights key findings, including state-of-the-art techniques, novel architectures, and innovative approaches to address the shortcomings of the existing methods. Finally, this study provides a summary of the key insights gained from the comparative analysis, highlighting the potential of deep learning in revolutionizing methodologies for crack detection. The findings of this research will serve as a valuable resource for researchers in the field, aiding them in selecting appropriate methods for crack detection and inspiring further advancements in this domain.

1. Introduction

According to the general definition, a crack is a defect that might seriously harm the structure and have adverse effects on them. Cracks are commonly found on the surfaces of various structures, including buildings, highways, bridges, subways, monuments, tunnels, etc. Cracks are one of the early warning signs of deterioration in civil structures. Crack identification is an important task in maintenance and requires specific consideration. To maintain healthy infrastructures and stop future damage, cracks must be promptly found and repaired as their existence reduces the value of the civil infrastructure. The infrastructure must be periodically examined to establish its state, and early crack detection can assist in averting further damage. Moreover, it provides an estimate of the civil structure’s life span and aids in ensuring safety.

In the early days, the assessments of the construction health of structures relied on manual inspections, which are collective efforts amongst many techniques to help identify the cracks in different civil structures. Manual detection, which relies on human resources, is particularly expensive and time-consuming since it demands professional personnel with the proper tools to identify the cracks while ensuring safety constantly. Since no visible record is made during this process, a lot of paperwork is also necessary, and it still needs to be determined how sensitive the crack is. The manual detection method might take a long time and must cover a significant region in huge infrastructures. In addition, it is occasionally required to close the highways, bridges, and tunnels for inspections, and it is challenging to assure inspector safety.

To address this issue, structural health monitoring equipment has been put in place as a first step. One of the initial techniques in detecting cracks from images is traditional image processing used to classify, detect, and segment the crack by taking images. Under the same exposure, cracks are often darker than the nearby pixels in the background. Therefore, the most intuitive method for crack detection is image thresholding. To find the probability of classifying each pixel as either a crack or a non-crack, different intensities are used (Zou et al., 2012). Abbas and Ismael (2021) use statistical techniques to establish a threshold for classifying crack and non-crack pixels based on the intensity of pixels. It is very hard to identify cracks using image processing techniques in most cases. Numerous environmental factors, like multicolor spots, shadows, dust, uneven lighting, numerous backdrop sceneries, and various changes in the dataset, might make it challenging to identify cracks in images. Detecting cracks using image processing algorithms is difficult because they have poor continuity and low contrast with surrounding pixels. Additionally, the hand-crafted features need a lot of processing and are not very robust. Therefore, these features cannot distinguish between the crack and the complicated background, mainly in the case of low-level images.

Almost in every research field, machine learning is a handy and powerful tool. These techniques’ main tasks are feature extraction and identifying the picture regions with cracks. The segmentation method in Staniek (2017)separates the image cracks and background pixels to categorize the image based on the characteristics retrieved from pavements. Segmentation, feature extraction, classification, and parameter quantification comprise the proposed method’s four steps. Other machine learning-based crack identification approaches include road crack detection and characterization based on a simple classifier (Oliveira and Correia, 2013), image binarization (Ahmadi et al., 2018), and Markov-based process (Delagnes and Barba, 1995). These methods worked effectively with images that had clearly visible cracks. Since the technique focuses on feature extraction, extracting crack characteristics from unclear images becomes challenging and useless. Most machine learning procedures demand a high number of structured labeled data. Making different selections at crack image boundaries for complex nonlinear regression might be difficult.

The convolutional neural network (CNN), invented in the 1980s, is the most well-known, advanced, and commonly applied deep learning (DL) technique. Because high-processing machines, significant storage devices, and computationally capable technology were unavailable, the concept did not appeal to researchers. The idea, however, gained popularity as machines’ capacity for computing and information retrieval and storage improved. Then, in Krizhevsky et al. (2012), CNNs were successfully used for classification problems, and these applications performed incredibly well. Fig. 1 illustrates the typical CNN structure.

Fig. 1. A typical CNN structure.

DL can locate cracks using segmentation, classification, or localization. For classification, images are given a crack or non-crack label. In localization, cracks in the image are located, and bounding boxes are created around them so they can be detected. For segmentation, at the pixel level, all the pixels are categorized as either crack or non-crack pixels. The general form of the output of classification, segmentation, and bounding box is shown in Fig. 2.

Fig. 2. Different ways of crack detection using CNN.

The objective of this paper is to review the work done till now on crack detection. This paper will provide a very deep and comprehensive review of the evaluation of the crack detection process, taking motivation from Jan and Seo (2023). It will provide a review of methods used till now for crack detection like image processing, machine learning, and deep learning techniques. Moreover, it will compare the results of the methods, also discuss the limitations of each technique, and provide the future research direction for crack detection in the end.

The rest of the paper is organized as follows: Section 2 will cover crack detection using image processing, while Section 3 will discuss crack detection using traditional machine learning. CNN-based crack detection, which is the main focus of this study, will be presented in Section 4. Furthermore, Section 5 will contain the results and discussion, followed by Section 6 and 7 which will discuss future works and conclusion respectively.

2. Crack Detection Using Image Processing

Image processing techniques were introduced to address the issues associated with manual crack detection. Feature extraction from the image, segmentation, image pre-processing, and crack recognition are the primary components of image processing techniques for crack detection. Techniques for processing images include edge detection, which looks for sudden pixel intensity changes. It is one of the earliest methods for crack identification that uses visual data from acquired images to detect, classify, and segment the crack.

Cracks can be detected from an image using edge information. Edge information is used by Abdel-Qader et al. (2003) in the images to detect cracks in bridges. This study applied four different algorithms for edge detection: fast Fourier transform (FFT), fast Haar transform (FHT), Canny, and Sobel. The final findings demonstrate that FHT was considerably more accurate in detecting cracks than the other three edge-detection methods.

Similarly, Adhikari et al. (2014) proposed an edge detection-based approach for calculating the width of the concrete bridge structure cracks. The change detection methodology based on the Fourier Transform of images has eliminated the traditional need for registering images.

CrackIT (Oliveira and Correia, 2013), a MATLAB-based toolbox for crack detection, is suggested for detecting and characterizing fine cracks with at least a 2 mm width. First, using image processing and pattern recognition algorithms, it preprocesses the images to identify cracks and categorizes them into types. The image is first converted to grayscale, after which Sobel’s filter is used to find the crack in the concrete structure.

Different filters are employed to find cracks by merging the binary crack images. For crack detection, a brand-new method based on the Gabor filter is employed. The actual image is passed through convolution with filters of different orientations (Salman et al., 2013).

Talab et al.(2016) use a threshold value to distinguish between background and foreground pixels. Their approach involves three steps: first, it converts the picture to grayscale, then utilize the image’s edges to identify cracks using Sobel’s approach and develop an image filter. Second, it uses an appropriate threshold to classify all pixels in a binary image into background and foreground categories. Once the region area has been obtained, the area is filtered and changed to the background if the value is less than a certain threshold, and in the end, Sobel’s filtering is utilized to remove any remaining noise, and then the Otsu technique to find significant cracks. Abdel-Qader et al. (2006) proposed the principal component analysis approach, frequently used for dimensionality reduction, to identify cracks in the image sets.

Another image processing technique adopted for crack detection is morphological operations(Yamaguchi et al., 2008). Afterward, Lee et al. (2013) proposed a method to enhance crack detection performance using morphological operations such as background brightness modification, binarization, and shape analysis.

Based on curvature evaluation and mathematical morphology, a three-step method is developed. To define crack-like patterns, segmentation is first done concerning a precise geometric model. Cracks added by morphology are used to improve the image’s quality before the curvature procedure. The resulting image is subjected to a linearized filter to separate the cracks from the corresponding background (Iyer and Sinha, 2005).

Using a grayscale scanning electron microscope, a crack detection approach for 2D images is developed. Five morphological processes and separate filters are used to improve the black-and-white image. Cracks’ width, length, area, aspect ratio, and direction are identified by connected components along with their labels. Filtering uses a user-defined aspect value, and the remaining cracks are statistically analyzed (Arena et al., 2014).

The contrast between adjacent pixels is compared to divide the pixels of crack and non-crack regions of an image and to distinguish them. It divides each image of the pavement into small grid cells of 8 × 8 pixels, and each cell is identified as either a crack or non-crack cell based on the border pixels’ grayscale information (Xu and Huang, 2005).

Another approach in image processing to detect cracks is the statistical method. Statistical properties are retrieved from pavement images by Koutsopoulos and Downey (1993)to segment the crack images. It presents techniques for image enhancement, segmentation, and classification of asphalt pavement distress, emphasizing automated analysis of images of asphalt pavements. At the regional level, the classification system can recognize primitives, at the global level, it also determines which class is present.

The crack detection and classification approach, in the first step from each image which reduces noise and then identifies every crack, is presented by Cubero-Fernandez et al. (2017) for the maintenance and repair of roads. After capturing the images, several procedures like a bilateral filter, morphological filter, logarithmic transformation, and Canny algorithm extract the essential features for highlighting the cracks. A statistical classifier is used, using a decision tree after learning the features to find cracks in road images.

The statistical filtering technique provides the foundation for the two-step crack detection approach. First, it analyzes underground pipe images for crack features, then segments the crack (Sinha and Fieguth, 2006). We can generalize and apply the method of line pixels detection presented by Seo (2020) to crack detection. His work is based on the sum of gradient angle differences (SGAD), where the gradient angle differences are determined by comparing the four pairs of gradients created by eight nearby pixels.

Furthermore, another method to perform line detection appropriately and to find signal-to-noise ratio (SNR) of different methods of line detection analytically is proposed by Seo (2021). Using signal coupling and error propagation, this study quantifies the SNR of three methods of line detection. We can also apply this concept to crack detection to design a suitable crack detection model.

The graph-based approach is employed by Koch et al. (2014) for locating and finding the properties of the crack and then also to detect the crack, although this method necessitates the manual entry ofthe crack’s start and endpoints. This issue was addressed by creating a robot system with machine vision installed on an end linkage of a specifically constructed automobile to investigate fractures beneath bridges(Lee et al., 2010). Then, for crack detection, with a tracing approach, the properties of cracks are extracted. All the image processing methods have been compared in Table 1.

Table 1. Comparison of image processing methods

Sometimes it is very challenging to identify cracks using image processing techniques. Numerous environmental factors, including some noise, different kinds of shadows, dust, some multicolor spots, uneven lighting, numerous backdrop sceneries, and dataset changes, might make it challenging to identify cracks in an image. For creating a CNN model, the background, crack characteristics, camera’s resolution and position, crack angle, length, and width of the acquired image should be considered. A crack might appear as a line with considerable width and edges one or a few pixels wider than that of the background pixels.

Various techniques for crack detection are categorized into two classes with high- and low-level features. Cracks with solid continuity and greater contrast have high-level features and can be quickly and accurately identified. The issue is that approaches relying on low-level characteristics deteriorate when there is background noise. Detecting cracks using image processing algorithms is difficult when they have poor continuity and low contrast with surrounding pixels. That is why we needed another method for crack detection to address these issues. A general flow diagram is presented for crack detection process evolution in Fig. 3.

Fig. 3. Flow diagram of the crack detection process evolution.

3. Crack Detection Using Traditional Machine Learning Methods

In almost every study field, machine learning is an effective tool. Traditional machine learning (supervised learning) was presented in this subject to overcome the problems with crack identification using image processing techniques. Machine learning’s main task is feature extraction, which also involves identifying areas in the image with cracks. The traditional machine learning models learn from the given data, and the field experts choose the features for the algorithms. However, deployment is completely automated after the algorithm has been developed and tested.

A support vector machine (SVM) can compute the geometric values of cracks and categorize cracks with various orientations. Before computing features based on color and texture, background elements are removed using crack segmentation based on machine learning. Crack images can be classified by providing the SVM with these characteristics (Varadharajan et al., 2014).

SVM is used to categorize the crack and non-crack images, and the histogram classification technique is also used by Prasanna et al.(2012).Binary tree and back-propagation-based classification techniques distinguish the crack and non-crack sections of the image by contrasting grayscale values.

Moussa (2011) has produced a reliable automatic pavement evaluation system based on image processing and machine learning. The suggested system could detect cracks, extract their properties, and report each crack’s kind, size, and severity. Feature extraction, classification, segmentation, and parameter quantification are the main steps in his methodology. Finally, SVM is used for categorization.

Deep belief network (DBN) by Wang and Zhang (2017)is another machine learning-based technique for crack detection. This method uses image augmentation, denoising, and other prepossessing techniques to highlight the desired cracks. The system may then determine the area associated with a binary image and choose the connected region based on crack characteristics. Finally, those highlighted cracks may utilize the DBN to automatically detect the crack using the gradient of the image’s pixels.

Oliveira and Correia (2013) presented a crack-detecting method based on classifiers. Crack detection is carried out in the first stage using paradigm learning from the sample. Apart of the available image database is automatically chosen to train an unsupervised system. The algorithm categorizes non-overlapping image blocks as either having crack pixels or not. To describe the discovered cracks that link components, a new classification system is created for the second task, which deals with crack-type characterization. Each crack showing in each image is given the proper label based on the types of cracks listed in the Portuguese distress catalog.

Similarly, using image processing and machine learning methods and based on the image binarization methodology, Ahmadi et al.(2018) proposed a new and integrated model with a heuristic image segmentation strategy for detecting cracks in road images.

Automatic identification of road cracks has proved difficult because of the severe inhomogeneity of the cracks, the complexity of the topology of the crack, the inference of noise, and occurrence of some texture similar to the cracks, and other factors. To solve these problems, Shi et al. (2016) present Crack Forest, a new road crack-detecting system based on random structured forests. It first applies the essential channel characteristics to redefine the tokens that make up a crack and more accurately portray cracks with intensity inhomogeneity. The second step is to include random structured forests to create a high-performing crack detector that can find cracks of any complexity. Finally, it suggests a crack descriptor to adequately describe and separate them from noises.

Moreover, a method is provided to automatically differentiate images of road surfaces with cracks from those without cracks (Cord and Chambon, 2012). Their approach, which is supervised learning-based, is universal and may be used to fix any defect in those images. Typically, they display complete textural information with all the patterns that exhibit small-scale variability and some large-scale regularity. An extensive collection of linear and nonlinear filters characterizes the textural information. AdaBoost-based supervised learning is used to pick the most applicable ones for the application. They used on-road photos captured by a specialized road imaging device and the VisTex image database to evaluate their algorithm on a textural recognition task.

On images with clear and visible cracks, all these machine-learning techniques performed well. Since the technique focuses on feature extraction, extracting crack characteristics from unclear images becomes challenging and ineffective. Deep learning methods have shown promising output compared to the techniques of conventional image processing and traditional machine learning. All machine learning methods have been compared in Table 2.

Table 2. Comparison of machine learning methods

4. Crack Detection Using CNN

CNN is the most actively used, well-established, and advanced deep learning technique. Using CNN, there are three primary methods for crack detection. It can identify cracks using classification, object detection (localization), and segmentation. CNN has made significant progress in pixel segmentation in recent years.

4.1. Crack Detection Using Classification

In classification, images are categorized either as a crack or a non-crack. To do the classification tasks, CNN tends to learn the specific features and characteristics of crack regions from the images. The general architecture used to do classification is divided into two parts. The first part is responsible for layer-by-layer extracting relevant features from given images, which are composed of convolutional layers followed by max-pooling layers in succession. The later part is constituted of fully-connected layers which classify the features extracted by the first part of the model into different classes(Cha et al., 2017).A simple architecture for crack classification is shown in Fig. 4.

Fig. 4. General architecture for crack classification.

To reduce human error and expense, a crack examination in concrete based on a robot approach is proposed by Dorafshan et al. (2018). The work given in this work is based on images that were gathered from the web and are classified as true spall, actual crack, or no crack. The dataset included 954 crack images, 278 spalls images and was separated into three categories: training images, validation images, and test images. The study in question makes use of the VGG-16 CNN architecture. The network has been improved, and while using the SCCS database, the technique classified images with 93.36% and 70% accuracy in the field test.

Another crack detector based on CNN is proposed by Yokoyama and Matsumoto (2017). Two thousand images, divided into five categories, were used to test the suggested model. The architecture contains six layers of convolution, three layers for pooling, a dropout layer, a leaky ReLU activation function, two fully-connected layers, and a classification SoftMax activation function.

Due to surface conditions, humidity, and the lightning effect, crack identification on concrete surfaces is difficult. To address such concerns in an image, a deep learning CNN-based crack detection algorithm is developed (Silva and Lucena, 2018). This experiment makes use of 3,500 images in total. The input image is split into a 256 by 256-pixel patch that contains both cracked and uncracked images. 80% of the image data was for training, leaving 20% for testing. In addition, transfer learning was applied to utilize the characteristics of the VGG-16 architecture, and the suggested model had an accuracy rate of 92.27%.

Finding cracks in complicated pavement conditions is difficult. As a result, a quicker crack detection approach based on R-CNN is suggested by Ibragimov et al. (2022), categorizing various kinds of cracks. The faster R-CNN uses the region proposal network, CNN, and SVM for proposal generation, extracting features, and object detection, respectively. A total of 2,600 images were used for training, and the average accuracy for linear crack was 31.86%, for area crack was 78.88%, and for patching was 87.21%.

Mandal et al. (2018) proposed an image-based road crack detection technique using YOLO v2 (Redmon and Farhadi, 2017). YOLO v2 takes a single look at the image and uses a bounding box with the proper height and width to identify the object. There were 9,053 images total in the dataset, which were split into 7,240 training images and 1,813 testing images. There were eight different sorts of cracks in these images. The learning weights were adjusted from ResNet-101 (He et al., 2016) by transfer. Precision, recall, and F1-score were the three separate matrices used to analyze YOLO v2. Additionally, the technique was compared with a single-shot multibox detector (SSD) (Liu et al., 2016) and a region-based convolutional network (Ren et al., 2017). It shows that YOLO v2 outperformed SSD and area convolutional neural networks by 5% in terms of performance and speed.

Only a patch or an entire image can be classified as a crack or non-crack region by the crack classification algorithms. It employs the sliding window approach, which scans specific pixels repeatedly while just once scanning the image’s boundary pixels. An image will not be correctly processed if a crack appears at the corner or edge of the image. As a result, crack feature extraction for detection using this method is somehow incapable of crack classification with high accuracy.

4.2. Crack Detection through Localization

Object detection is a group of related activities used to locate objects of interest in images. In the case of crack images, it will locate the crack coordinates inside the image and create a bounding box around them. A general architecture for crack localization is presented in Fig. 5.

Fig. 5. A general architecture for crack localization. RPN: region proposal network, FCN: fully-connected convolutional network.

The crack detection was carried out in an object detection context by Cha et al. (2018). This work is an improvement over their earlier studies, which used a sliding window approach to locate cracks in an image classification environment (Cha et al., 2017). There have been two enhancements. The first involves adding a larger data set, while the second involves using a faster R-CNN architecture to address various crack detection challenges in an object detection setting. Their dataset consists of 2,366 images, each with a size of 500 by 375, and has been labeled for five kinds of damages: concrete crack, steel delamination, steel corrosion in medium and high degrees, and bolt corrosion. Then, utilizing this database, the Faster R-CNN architecture is first modified and trained. The results reveal average accuracy (AP) with a mean AP of 87.8%.

An RPN based on multi-scale defect was created by Li et al. (2018) that offer proposed bounding boxes in several layers for improved detection accuracy. Using a database of geo-tagged images and an additional deep architecture, they also carried out crack geo-localization. Notably, the geo-localization module and the crack detection network are both integrated into a single network.

In the crack detection area, CrackDN, a faster R-CNN variant, was proposed by Huyan et al. (2019). This system employs a sensitivity detection network to extract deep features coupled with a pre-trained CNN network, resulting in a quicker training process.

R-CNN (Girshick et al., 2014) architecture is used for detecting cracks firstly by Kim et al. (2018). In this study, potential locations of bridge cracks were initially found using the selective search method. Then, CIFAR-10 and ImageNet datasets were used to train a CNN model for classifying cracks and bounding box formation.

In a large dataset with an object detection setup, Maeda et al. (2018) did crack detection using the single shot multibox detector framework. They used Inception V2 (Ioffe and Szegedy, 2015) and MobileNet (Howard et al., 2017) as the foundational feature extractor modules in the SSD framework. It shows that by using the suggested object detection approach, the kind of damage can be accurately categorized into eight types. According to the findings, they were able to attain recalls and precisions of more than 75%.

Mandal et al. (2018) used the YOLO9000 to detect cracks on a dataset that contained various kinds of cracks. The system was evaluated on 1,813 road images after being trained on 7,240 images obtained from portable cameras. The F1-score was 87.80% for detection without crack class prediction, whereas the F1-score for classification with crack class prediction was 73.94%.

The bounding box approximates the region of interest and hence cannot be utilized to measure the length, density, or crack characteristics. For effective safety analysis, pixel-level classification or segmentation in crack detection is essential.

4.3. Detection of Cracks through Segmentation

Semantic segmentation is a term used in computer vision to describe classification at the pixel level. As an output, labeled masks are generated to segment each crack and non-crack pixel. By generating the segmented crack image, it detects the shape and location of the cracks. CNN has seen considerable progress in crack pixel segmentation in recent years. A simple architecture for crack segmentation is shown in Fig. 6.

Fig. 6. A general architecture for crack segmentation.

To identify cracks automatically at the pixel level, a fully-connected convolutional network (FCN) based on a computer vision approach was developed (Yang et al., 2018). The architecture was composed of a layer of convolution, max pooling layers, ReLU activation, a deconvolutional layer, and the SoftMax function, while cross-entropy is used as a loss function. For the experiment, 800 images were collected and scaled to 224 by 224. The overall results were good but due to information loss during downsampling, the prediction and resolution were poor.

For evaluating density and detecting cracks, an approach based on deep FCN is proposed for the semantic segmentation of concrete crack images(Dung and Anh, 2019). Using the concrete crack dataset, three distinct network architectures, VGGNet (Vo et al., 2018), ResNet (He et al., 2016), and InceptionV3 (Szegedy et al., 2016), are examined for implementation of the suggested technique. The FCN (Long et al., 2015) and VGG-16 (Vo et al., 2018) designs outperformed the others. About 40,000 images were used to train the encoder-decoder FCN architecture, which, for end-to-end classification and segmentation, reached 90% accuracy.

CrackSegNet, an upgraded deep CNN network, is suggested by Ren et al.(2020)to segment cracks pixel-wise in tunnels. The proposed network consists of an encoder, a decoder, skip connections, spatial pyramid max pooling, and dilated convolutions. The encoder path’s backbone network is an improved version of the VGG-16. To overcome the unbalanced class problem, the focus loss function is used. The suggested network was tested using several layers and approaches and obtained 99% accuracy.

Similarly, Liu et al. (2019) propose another crack segmentation approach based on a hierarchical convolutional neural network. The SoftMax layer of the design uses a VGG-16 network to identify the expected values for each pixel. Alayer with side output, deep supervision, and a range of scales is added after each convolutional layer. The outcome is produced by concatenating the characteristics from these side layers, which span several scales and levels. The results show that the proposed model has achieved results that are equivalent to those of state-of-the-art methods.

CrackNet-V, a deep neural network-based crack detection method, is introduced by Fei et al. (2020). The approach identifies and classifies a given pixel as a crack or non-crack in a specified location of a 3D asphalt pavement image. CrackNet-V is built on a ten-layer VGG architecture. It contains a pre-processing layer, eight layers of convolution, a Leaky Rectified Tanh layer, and a final layer with a sigmoid-like function. To minimize loss, a cross-entropy function is used as a loss function that compares the predicted and actual pixels.

CrackNet, a CNN architecture, is presented by Zhang et al. (2017) to automatically identify fractures at the pixel level. CrackNet is applied for pixel-level crack detection on road surfaces. It has a little different architecture than standard CNNs. The number of features has decreased since the pooling layer is excluded. It has two sigmoid layers, two layers of convolution, two fully-connected layers, a ReLU function, and two dropout layers. Using 200 3D test images, the experiment found that the CrackNet can obtain good results, but it requires more processing time.

Moreover, CrackNet-II (Zhang et al., 2018), a modified version of CrackNet (Zhang et al., 2017), is presented. The two main improvements in CrackeNet-II are a denser design with a greater number of hidden layers with fewer parameters and an increased learning capability through the elimination of feature generators. With batch normalization, mini-batch gradient descent, normalized initialization, cross-entropy, dropout, and momentum, CrackNet-II hasten layers of convolution. 3,000 3D images of pavement make up the dataset, which is divided into 2,500 images for training, 300 for validation, and 200 for testing. CrackNet-II performs better than CrackNet and is five times quicker while achieving higher accuracy.

Ni et al. (2019) present a crack delineation network (CDN)for rapid and precise pixel-wise crack segmentation and detection. The framework is made of the GoogLeNet architecture and a CDN. GoogLeNet uses three network components to classify cracks at the pixel level: convolution, inception, and dropout. The CDN’s two network components are consecutive convolutional layers and convolutional feature-map fusion. An stochastic gradient descent optimizer and a cross-entropy are employed to update weights. Net-1 to Net-5 and Net-6 to Net-9 are used in the evaluation. Performance is measured using precision, recall, and the F1-score. Except for Net-6, which employs higher-order feature maps, all networks performed well.

Bang et al. (2019) offer a crack detection approach that employs a CNN network consisting of the encoder and the decoder to classify each pixel as crack or non-crack at the pixel level. The proposed model encodes with ResNet-152 and decodes with deconvolution (SegNet, FCN, and ZFNet). Using the Adam optimizer, the cross-entropy function is used as a loss function. The collection contains 527 black-box images with resolutions of 1,920 by 1,080 (427 for training and 100 for testing).The ResNet-152-based network recognized cracks in images with complicated backgrounds and achieved better results.

Another pixel-level crack-detecting model, the crack segmentation network (CSN), that is based on end-to-end deep learning, is proposed (Lee et al., 2019). CSN was explicitly created for cracks with very complex patterns. CSN tends to scan all the images at once and consists of five convolutional and a recovery block. Convolutional, pooling, ReLU, and batch normalization layers are all present in each convolutional block. The transpose layer in the recovery block does the crack prediction at the pixel level, whereas SoftMax performs the classification. CSN was trained using the MS-COCO dataset, and the results reveal that the suggested technique is very accurate and fast, even when dealing with complicated images.

It is difficult to detect cracks with complicated backgrounds. For this, Kang et al. (2020) propose a crack detection, quantification, and localization technique that combines ResNet-50-based faster R-CNN for detecting cracks, a modified TuFF that employs contrast limited adaptive histogram equalization to segment cracks at the pixel level, and distance transform method to assess the orientation of the crack. The dataset used consisted of 1,200 training and 100 validation images taken from a dataset that was available publicly.

To monitor hydropower stations safely, a method named crack detection technique on dam surface (CDDS) is presented (Feng et al., 2020). The detection technique is done utilizing an unmanned aerial vehicle, which acquired 1,000 images from the surface of a dam, and from that, 504 images were used for further processing. The architecture is made up of an encoder-decoder structure linked by a skip connection. For extracting dense and sparse features, the CDDS uses SegNet architecture. The network utilized 404 to train, 50 to validate, and 50 to test.

Kim and Cho (2020) propose a methodology for detecting multiple damages automatically. The proposed method makes use of the RestNet-101 as a backbone in the Mask R-CNN network to extract critical features and restore spatial information. It uses the feature pyramid network to recognize objects and build a segmentation mask for the cracks. This feature information is used by the region proposal network to identify and locate damage or cracks. The Mask R-CNN employs the log and L1 as the loss functions. The model is first pre-trained on the MS-COCO dataset and then trained again on 765 images of concrete.

Using FCN to detect multiple damages at the pixel level, a new method is proposed (Li et al., 2019). For feature extraction, the suggested FCN model is fine-tuned using DenseNet-121. DenseNet, which is extensively used for images on a large scale, has a dense block that collects all the characteristics that pass to the following layer.In total, 1,375 images with small and thin cracks and holes are captured. To enhance the dataset size, the augmentation approach is utilized, and a total of 2,750 images are split into 80% and 20% for training and testing, respectively. The suggested FCN model achieved a mean accuracy of 98.61% and mean intersection over union (mIoU) of 84.53.

Another network for crack segmentation, termed as CrackSeg (Song et al., 2020), is developed to identify cracks in the road at the pixel level. The dilated convolution and upsampling modules, which are multiscale, are part of the ResNet-based network. To make the dataset, 8,188 images of cracks are collected and divided into 4,738 training, 1,035 validation, and 2,415 testing images. As a result, the network outperformed other networks with 98.79% mean accuracy.

Chen et al. (2020) propose a pavement crack segmentation network (PCSN) based on the SegNet architecture to detect cracks in pavements and bridges. VGG-16 is used in the architecture’s encoder. The proposed PCSN architecture has five convolutional blocks with convolutional and max-pooling layers on the encoder side. The decoder is made up of five blocks, each consisting of convolutional and upsampling layers. For initialization, pre-trained VGG-16 weights are used in the network. As a network optimizer, the suggested network used Adadelta and a cross-entropy as a loss function. From the results, the PCSN attained an average accuracy of 83% at the pixel level.

To identify cracks in bridge concrete structures, a ResNeXt-based framework is presented (Li et al., 2020). The proposed network, which combines the VGG and the Inception network, is an improved version of the initial ResNeXt. The 25,358 crack images used to construct the dataset were taken using a DSLR camera, a flash, a telephoto lens, a distance meter, and multiple sensors. According to their shape and orientation, these gathered crack images were divided into five categories: complicated, wide, oblique, reticulated, and intersecting cracks. The dataset size was increased to 45,358 pictures using the rotation augmentation method as well.

Later, inspired by the results of U-Net(Ronneberger et al., 2015) in biomedical image segmentation, a U-Net-based crack classification technique is suggested (Jenkins et al., 2018). This approach was proposed to tackle the issue of losing fine image details during downsampling. Jan and Seo (2023) use the same method while using residual blocks with an attention model to develop depth maps. The U-Net design comprises a downsampler encoder and an upsampler decoder. To save the fine details, each encoder layer is concatenated to the decoder layer. The encoder part consists of two layers of convolution, two ReLU, and one max-pooling layer, while the decoder is made up of two convolutional, two ReLU, and one up-sampling layer, and one SoftMax layer to perform the classification. In total, there were 118 images in the dataset and split into 80, 20, and 18 for training, validation, and testing, respectively. On two out of three evaluation metrics, the model performed very well and outperformed all the state-of-the-art methods till that time.

Another concrete crack detection technique based on U-Net is used by Liu et al. (2019), which updates weights using the focal loss function using the Adam optimization method. They used 57 images to train the model while 27 images were used to test it. For model evaluation, k-fold cross-validation with k=3 is used.

Another crack segmentation model based on U-Net is presented for pixel-level crack classification (Ji et al., 2018). The U-Net architecture is employed with certain modifications such as input image size, convolutional layers with zero paddings to prevent shrinkage, an Adam optimizer for quicker convergence, and a drop-out layer to minimize overfitting. For classification at the pixel level, the binary cross-entropy loss function is utilized. Two hundred images were used for training the network, and after 37 epochs, obtained a very high accuracy of 99.55%.

To improve crack detection and segmentation accuracy, a pavement crack detection and segmentation approach based on two steps is presented by Liu et al. (2020). The first stage employs a modified YOLOv3 to identify image cracks while the second part uses a modified U-Net for segmenting the discovered cracks. The network is trained using self-taken crack images as well as the Crack Forest dataset. In total, 27,966 images are used, of which 16,780 are used to train the model.

5. Results and Discussion

Crack detection is an important task for maintaining healthy infrastructure. Crack detection model output depends on different factors like the quality and size of the dataset, network architecture, data preprocessing, optimization, training, etc. The training process involves optimizing the model’s parameters using suitable loss functions, optimization algorithms, and regularization techniques. The selection of appropriate hyperparameters, such as batch size, learning rate, and number of epochs, can significantly affect the model’s convergence and generalization ability. Furthermore, different evaluation matrices have been used in previous studies which are not directly comparable with each other. So, in our study, we will compare different factors such as loss function, dataset, mode of training, and implementation tools in addition to the common evaluation matrices.

5.1. Evaluation Matrices

Different matrices have been used in crack detection evaluation. The most common matrices are defined below:

\(\begin{aligned}\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}\end{aligned}\)       (1)

\(\begin{aligned}\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}\end{aligned}\)       (2)

\(\begin{aligned}\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\end{aligned}\)       (3)

\(\begin{aligned}\mathrm{F1-score}=\frac{\mathrm{2TP}}{\mathrm{2TP}+\mathrm{FP}+\mathrm{FN}}\end{aligned}\)      (4)

In Eqs. (1, 2, 3, and 4), TP = true positive, TN = true negative, FP = false positive, and FN = false negative.

In Table 3, a comparison of different key factors used during the training process has been made. These are the factors that heavily affect the training output of a model. We have compared every model architecture, dataset used, loss function, mode of training, and implementation tool.

Table 3. Comparison of the training process of different methods

Our focus was to review different papers, especially in the field of crack segmentation. A comparison of the results of some of the most well-known methods has been shown in Fig. 7. Moreover, in Table 4 the quantitative results and comparison are presented. From the quantitative comparison, it can be seen that method of Ji et al. (2018) achieved higher accuracy, Li et al. (2019) got higher recall, while Song et al. (2020) performed best on precision and F1-score. However, we cannot compare the results with each other directly and decide which model is better because we discussed above these models are trained with different datasets and implementation tools which affect the results. If we want to compare two models’ performance, we must train both with the same dataset and the same GPU setup.

Fig. 7. Results comparison of a few well-known models.

Table 4. Quantitative comparison of different reviewed methods

6. Future Works

The identification of structural cracks still has to be improved even though the existing approaches produce significant outcomes. The following is a description of the research problems that need to be considered for future study.

Most of the CNN architecture till now has a very high number of parameters which requires more training time and a higher GPU. In addition, it is difficult to use such models for real-time crack detection. So, for future research developing a model with fewer parameters and higher accuracy can be considered.

Moreover, we know that a CNN model requires a lot of labeled data to learn from and give accurate results, and data labeling for a new dataset is a very tedious job. So, there is a need to build a new model which can learn from fewer data and achieve high accuracy with better-predicted outputs.

Furthermore, the Segment Anything model (SAM) is an advanced artificial intelligence model capable of segmenting any object in images or videos with high accuracy and efficiency (Kirillov et al., 2023). SAM has demonstrated impressive performance on various image segmentation benchmarks, including COCO. In the future, for crack segmentation, SAM’s capabilities can significantly enhance crack detection and localization in structural health monitoring. Its high-quality and efficient object segmentation capabilities will make it a valuable tool for identifying and localizing cracks in various scenarios.

In addition, till now, using CNN, most of the methods for crack detection are classification, bounding box creation or localization, and crack segmentation. There is very little attention given to finding the crack length, width, and depth, which will help find the severity of the crack and can tell us about the expected life span of a structure. So, it will be the need in the future to develop a fast and accurate CNN model to find the length and depth of the crack.

7. Conclusions

This review has presented an in-depth review of crack detection methods, with a particular emphasis on deep learning-based procedures. The paper discusses the crack detection process starting from image processing techniques, traditional machine learning, and then in the end deep learning-based approaches. The findings highlight the importance of automated crack identification in guaranteeing structural safety and integrity, as well as the shortcomings of existing systems that rely on manual inspection or image processing techniques.

The paper highlighted the advances made in crack identification using deep learning approaches, including CNNs. Deep learning models have outperformed traditional methods at capturing detailed crack patterns, as well as being resistant to changes in crack types, sizes, orientations, and environmental conditions. Their ability to automatically learn hierarchical patterns from large-scale datasets has made them highly effective in correctly and reliably detecting cracks.

Key elements affecting crack detection model performance were discussed, including datasetsize and quality, network architecture selection, preprocessing approaches, model training and optimization, transfer learning methodologies, evaluation measures, and deployment constraints. These variables work together to make deep learning-based crack detection systems successful and practical. Moreover, a few issues have been highlighted and future research directions have been presented to help the researcher in their future research and to improve crack detection techniques further.