Concrete crack segmentation based on convolution–deconvolution feature fusion with holistically nested networks
Shengjun Xu
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorCorresponding Author
Ming Hao
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Correspondence
Ming Hao, School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi 710055, China.
Email: [email protected]
Search for more papers by this authorGuanghui Liu
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorYuebo Meng
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorJiuqiang Han
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
AI & DE Guangdong Province Lab (GZ), No.70 Yuean Road, Guangzhou, Guangdong, 510335 China
Search for more papers by this authorYa Shi
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorShengjun Xu
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorCorresponding Author
Ming Hao
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Correspondence
Ming Hao, School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi 710055, China.
Email: [email protected]
Search for more papers by this authorGuanghui Liu
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorYuebo Meng
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorJiuqiang Han
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
AI & DE Guangdong Province Lab (GZ), No.70 Yuean Road, Guangzhou, Guangdong, 510335 China
Search for more papers by this authorYa Shi
School of Information and Control Engineering, Xi'an University of Architecture and Technology, No. 13 Yanta Road, Xi'an, Shaanxi, 710055 China
Search for more papers by this authorFunding information: Basic Research Foundation of Xi'an University of Architecture and Technology, Grant/Award Numbers: JC1706, JC1703; Special Research Project of Education Department of Shaanxi Province, Grant/Award Number: 18JK0477; Natural Science Foundation of Shaan Xi, Grant/Award Numbers: 2015JM6276, 2017JM6106, 2019JQ-760, 2020JM-473, 2020JM-472; General Projects of Key Research and Development Program of Shaanxi Province, Grant/Award Number: 2021SF-429; National Natural Science Foundation of P. R. China, Grant/Award Numbers: 61803293, 51678470
Summary
Automatic crack detection on concrete surfaces has become increasingly important for the health diagnosis of concrete structures to prevent possible malfunctions or accidents. In this paper, a concrete crack segmentation network based on convolution–deconvolution feature fusion with holistically nested networks is proposed. The proposed network adopts an encoder–decoder structure and uses VGG-16 as the basic feature extraction network. First, considering the problem that the VGG-16 network can extract redundant features in the encoding stage, based on the channel attention mechanism, the channel spatial correlation and global information are used to emphasize crack features to remove redundant features. Second, through the convolution–deconvolution feature fusion module, the deep semantic information of the deconvolution is effectively fused with the shallow features of convolution, which effectively improves the semantic crack feature information extracted at each stage of the VGG-16 network. Finally, based on a multiscale supervised learning mechanism, holistically nested networks are used to fuse the prediction results from different scales, which enhances the network's ability to express linear topological structures and improves the accuracy of crack segmentation. Through a large number of experiments on the Bridge_Crack_Image_Data dataset and CFD dataset, we demonstrate that compared with other deep networks, the proposed network not only achieves better segmentation results for cracks of different widths but is also more robust.
REFERENCES
- 1Gang L, Xueli R, Wenting Q, Biao M, Ying L. Automatic bridge crack identification from concrete surface using ResNeXt with postprocessing. Struct Control Health Monit. 2020; 27:e2620.
- 2Gong Q, Zhu L, Wang Y, Yu Z. Automatic subway tunnel crack detection system based on line scan camera. Struct Control Health Monit. 2021; 28(8):e2776. doi:10.1002/stc.2776.
- 3Byunghyun K, Soojin C. Image-based concrete crack assessment using mask and region-based convolutional neural network. Struct Control Health Monit. 2019; 26:e2381. doi:10.1002/stc.2381.
- 4Lee JS, Hwang SH, Choi IY, Choi Y. Estimation of crack width based on shape-sensitive kernels and semantic segmentation. Struct Control Health Monit. 2019; 27(4):e2504. doi:10.1002/stc.2504.
- 5Stringham GE. Corps of engineers inspection team goes to new heights. Engineer. 2012; 42: 32-33.
- 6Sun Z, Wang C, Sha A. Study of image-based pavement cracking measurement techniques. In Proceedings of the 9th International Conference on Measurement & Instruments (ICEMI09). IEEE, 2009; 140-143. doi:10.1109/ICEMI.2009.5274633.
10.1109/ICEMI.2009.5274633 Google Scholar
- 7Miyamoto A, Konno M, Bruhwiler E. Automatic crack recognition system for concrete structures using image processing approach. Asian J Inf Technol. 2007; 6: 553-561.
- 8Tang J, Gu Y. Automatic crack detection and segmentation using a hybrid algorithm for road distress analysis. IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 2013; 3026-3030. doi:10.1109/SMC.2013.516.
10.1109/SMC.2013.516 Google Scholar
- 9Oliveira H, Correia PL. Automatic road crack segmentation using entropy and image dynamic thresholding. In: 17th European Signal Processing Conference. IEEE, 2009: 622-626.
- 10Lee BY, Kim YY, Yi ST, Kim JK. Automated image processing technique for detecting and analysing concrete surface cracks. Struct Infrastruct Eng. 2013; 9(6): 567-577. doi:10.1080/15732479.2011.593891.
- 11Nguyen CK, Kawamura K, Shiozaki M, Tarighat A. Development of an automatic crack inspection system for concrete tunnel lining based on computer vision technologies. IOP Conference Series Materials Science and Engineering. 2018; 371:012015. doi:10.1088/1757-899X/371/1/012015.
10.1088/1757?899X/371/1/012015 Google Scholar
- 12Bray J, Verma B, Li X, He W. A neural network based technique for automatic classification of road cracks. In The 2006 IEEE International Joint Conference on Neural Network Proceedings. IEEE, 2006; 907-912. doi:10.1109/IJCNN.2006.246782.
10.1109/IJCNN.2006.246782 Google Scholar
- 13Xu Z, Zhao X, Song H, Tao L, Na W. Asphalt pavement crack recognition algorithm based on histogram estimation and shape analysis. Chin J Sci Instrum. 2010; 31: 2260-2266.
- 14Hu Y, Zhao CX. A novel LBP based methods for pavement crack detection. J Pattern Recognit Res. 2010; 5(1): 140-147. doi:10.13176/11.167.
10.13176/11.167 Google Scholar
- 15Chen FC, Jahanshahi MR, Wu RT, Joffe C. A texture based video processing methodology using Bayesian data fusion for autonomous crack detection on metallic surfaces. Comput-Aided Civ Inf Eng. 2017; 32(4): 271-287. doi:10.1111/mice.12256.
- 16Zalama E, Gómez-García-Bermejo J, Medina R, Llamas J. Road crack detection using visual features extracted by Gabor filters. Comput Aided Civ Inf Eng. 2014; 29(5): 342-358. doi:10.1111/mice.12042.
- 17Bu GP, Chanda S, Guan H, Jo J, Blumenstein M, Loo YC. Crack detection using a texture analysis based technique for visual bridge inspection. Electron J Struct Eng. 2015; 14(1): 41-48.
- 18Chun PJ, Izumi S, Yamane T. Automatic detection method of cracks from concrete surface imagery using two-step light gradient boosting machine. Comput Aided Civ Inf. 2020; 36(1): 1-12. doi:10.1111/mice.12564.
- 19Hutchinson TC, Chen ZQ. Improved image analysis for evaluating concrete damage. J Comput Civ Eng. 2006; 20(3): 210-216. doi:10.1061/(ASCE)0887-3801(2006)20:3(210).
- 20Abdel-Qader I, Abudayyeh O, Kelly ME. Analysis of edge-detection techniques for crack identification in bridges. J Comput Civ Eng. 2003; 17(4): 255-263. doi:10.1061/(ASCE)0887-3801(2003)17:4(255).
- 21Albert AP, Nii AO. Evaluating pavement cracks with bidimensional empirical mode decomposition. Eurasip J Adv Sig Pr. 2008; 2008: 1-7.
- 22Yang J, Choi J, Hwang S, An YK, Sohn H. A reference-free micro defect visualization using pulse laser scanning thermography and image processing. Meas Sci Technol. 2016; 27(8):085601. doi:10.1088/0957-0233/27/8/085601.
10.1088/0957?0233/27/8/085601 Google Scholar
- 23Hutchinson TC, Chen ZQ. Improved image analysis for evaluating concrete damage. J Comput Civil Eng. 2006; 20(3): 210-216. doi:10.1061/(ASCE)0887-3801(2006)20:3(210).
- 24Wang G, Tse PW, Yuan M. Automatic internal crack detection from a sequence of infrared images with triple-threshold Canny edge detector. Meas Sci Technol. 2017; 29: 1-15.
- 25Cho HW, Yoon HJ. Comparison of crack extraction performance according to different edge detectors. Appl Mech Mater. 2014; 607: 669-672. doi:10.4028/www.scientific.net/AMM.607.669.
10.4028/www.scientific.net/AMM.607.669 Google Scholar
- 26Wang R, Qi T, Lei B, Li Y, Zhu X. Study on the characteristic extraction of tunnel lining cracks. Chin J Rock Mech Eng. 2015; 34: 1211-1217.
- 27Wang G, Tse PW, Yuan M. Automatic internal crack detection from a sequence of infrared images with triple-threshold Canny edge detector. Meas Sci Technol. 2018; 29(2): 1-15. doi:10.1088/1361-6501/aa9857.
10.1088/1361?6501/aa9857 Google Scholar
- 28Medina R, Llamas J, Gómez-García-Bermejo J, Zalama E, Segarra MJ. Crack detection in concrete tunnels using a Gabor filter invariant to rotation. Sensors. 2017; 17(7):1670. doi:10.3390/s17071670.
- 29Zhang W, Zhang Z, Qi D, Liu Y. Automatic crack detection and classification method for subway tunnel safety monitoring. Sensors. 2014; 14(10): 19307-19328. doi:10.3390/s141019307.
- 30Ziou D, Tabbone S. Edge detection techniques—An overview. International Journal of Pattern Recognition & Image Analysis. 1998; 8(4): 537-559.
- 31Kim H, Ahn E, Cho S, Shin M, Sim SH. Comparative analysis of image binarization methods for crack identification in concrete structures. Cem Concr Res. 2017; 99: 53-61. doi:10.1016/j.cemconres.2017.04.018.
- 32Hu Y, Zhao CX, Wang HN. Automatic pavement crack detection using texture and shape descriptors. IETE Technical Review. 2010; 27(5): 398-405. doi:10.4103/0256-4602.62225.
- 33Huynh P, Ross R, Martchenko A, Devlin J. Dou-edge evaluation algorithm for automatic thin crack detection in pipelines. IEEE International Conference on Signal and Image Processing Applications (ICSIPA). IEEE, 2015: 191-196. doi:10.1109/ICSIPA.2015.7412188.
10.1109/ICSIPA.2015.7412188 Google Scholar
- 34Jiang X, Ma ZJ, Ren WX. Crack detection from the slope of the mode shape using complex continuous wavelet transform. Comput Aided Civ Inf Eng. 2012; 27(3): 187-201. doi:10.1111/j.1467-8667.2011.00734.x.
- 35Li LF, Ma WF, Li L, Lu C. Research on detection algorithm for bridge cracks based on deep learning. Acta Automatica Sinica. 2019; 45(9): 1727-1742.
- 36Cha YJ, Choi W, Büyüköztürk O. Deep Learning-based crack damage detection using convolutional neural networks. Comput -Aided Civ Infrastruct Eng. 2017; 32(5): 361-378. doi:10.1111/mice.12263.
- 37Yang Q, Shi W, Chen J, Lin W. Deep convolution neural network-based transfer learning method for civil infrastructure crack detection. Autom Constr. 2020; 116:103199. doi:10.1016/j.autcon.2020.103199.
- 38Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science. 2006; 313(5786): 504-507. doi:10.1126/science.1127647.
- 39Huang TH, Nie ZY, Wang QG, Li S, Yan LC, Guo DS. Real-time image semantic segmentation based on block adaptive feature fusion. Acta Automatica Sinica. 2021; 47(5): 1137-1148. doi:10.16383/j.aas.c180645.
10.16383/j.aas.c180645 Google Scholar
- 40Yang XC, Li H, Yu YT, Luo XC, Huang T, Yang X. Automatic pixel-level crack detection and measurement using fully convolutional network. Comput Aided Civ Inf Eng. 2018; 33(12): 1090-1109. doi:10.1111/mice.12412.
- 41Wang S, Wu X, Zhang YH, Chen Q. Image crack detection with fully convolutional network based on deep learning. J Comp-Aided Des Comp Graph. 2018; 30(5): 859-867.
- 42Huyan J, Li W, Tighe S, Xu ZC, Zhai JZ. CrackU-net: a novel deep convolutional neural network for pixelwise pavement crack detection. Struct Control Health Monit. 2020; 27(8):e2551. doi:10.1002/stc.2551.
- 43Ren YP, Huang JS, Hong ZY, Lu W, Zou YJ, L J, Shen X H. Image-based concrete crack detection in tunnels using deep fully convolutional networks. Construct Build Mater. 2020; 234:117367. doi:10.1016/j.conbuildmat.2019.117367.
- 44Cao JG, Yang GT, Yang XY. Pavement crack detection with deep learning based on attention mechanism. J Comp-Aided Des Comp Graph. 2020; 32(8): 1324-1333.
- 45Qin Z, Zheng Z, Li QQ, Qi XB, Wang Q, Wang S. DeepCrack: learning hierarchical convolutional features for crack detection. IEEE Trans Image Process. 2019; 28(3): 1498-1512. doi:10.1109/TIP.2018.2878966.
- 46Yang F, Zhang L, Yu S, Prokhorov D, Mei X, Ling H. Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Trans Intell Transp Syst. 2019; 21(4): 1525-1535. doi:10.1109/TITS.2019.2910595.
- 47Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Proc. Int. Conf. Learn. Representations, 2015: 1-14
- 48Du C, Wang YN, Wang CH, Shi CZ, Xiao BH. Selective feature connection mechanism: concatenating multi-layer CNN features with a feature selector. Pattern Recognit Lett. 2020; 129: 108-114. doi:10.1016/j.patrec.2019.11.015.
- 49Ng JY, Yang F, Davis LS. Exploiting local features from deep networks for image retrieval. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2015; 53-61. doi:10.1109/CVPRW.2015.7301272.
10.1109/CVPRW.2015.7301272 Google Scholar
- 50Liu J, Liu BY, Lu HQ. Detection guided deconvolutional network for hierarchical feature learning. Pattern Recognit. 2015; 48(8): 2645-2655. doi:10.1016/j.patcog.2015.02.002.
- 51Xie S, Tu Z. Holistically-nested edge detection. Int J Comput Vis. 2017; 125(5): 3-18. doi:10.1007/s11263-017-1004-z.
10.1007/s11263?017?1004?z Google Scholar
- 52Shi Y, Cui LM, Qi ZQ, Meng F, Chen ZS. Automatic road crack detection using random structured forests. Trans Intell Transp Syst. 2016; 17(12): 3434-3445. doi:10.1109/TITS.2016.2552248.
- 53Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015: 3431-3440, doi:10.1109/CVPR.2015.7298965.
10.1109/CVPR.2015.7298965 Google Scholar
- 54Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for scene segmentation. IEEE Trans Pattern Anal Mach Intell. 2017; 39(12): 2481-2495. doi:10.1109/TPAMI.2016.2644615.
- 55Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015; 9351: 234-241. doi:10.1007/978-3-319-24574-4_28.
10.1007/978?3?319?24574?4_28 Google Scholar
- 56Liu Y, Cheng MM, Hu XW, et al. Richer convolutional features for edge detection. IEEE Trans Pattern Anal Mach Intell. 2019; 41(8): 1939-1946. doi:10.1109/TPAMI.2018.2878849.
