3D multi-resolution attention capsule network for diagnosing multi-pathological types of pulmonary nodules
Zhili Zhang
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorTaohui Xiao
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorYu Fu
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorYuqiang Gao
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorMeirong Ren
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorCorresponding Author
Wentao Cui
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Correspondence
Wentao Cui and Enqing Dong, School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai 264209, China.
Email: [email protected] and [email protected]
Search for more papers by this authorCorresponding Author
Enqing Dong
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Correspondence
Wentao Cui and Enqing Dong, School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai 264209, China.
Email: [email protected] and [email protected]
Search for more papers by this authorZhili Zhang
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorTaohui Xiao
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorYu Fu
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorYuqiang Gao
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorMeirong Ren
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Search for more papers by this authorCorresponding Author
Wentao Cui
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Correspondence
Wentao Cui and Enqing Dong, School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai 264209, China.
Email: [email protected] and [email protected]
Search for more papers by this authorCorresponding Author
Enqing Dong
School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai, China
Correspondence
Wentao Cui and Enqing Dong, School of Mechanical, Electrical & Information Engineering, Shandong University, Weihai 264209, China.
Email: [email protected] and [email protected]
Search for more papers by this authorZhili Zhang, Taohui Xiao, Yu Fu, and Yuqiang Gao are co-first authors.
Funding information: Key Research and Development Project of Shandong Province, Grant/Award Number: 2019GGX101022; National Natural Science Foundation of China, Grant/Award Numbers: 62171261, 81671848, 81371635; Innovation Ability Improvement Project of Science and Technology Small and Medium-sized Enterprises of Shandong Province, Grant/Award Number: 2021TSGC1028
Abstract
To achieve effective diagnosis of multi-pathological types on pulmonary nodules, a three-dimensional (3D) multi-resolution attention (MRA) capsule network (3D MRA-CapsNet-DL) was proposed by using a self-constructed lung CT image dataset. In the construction of 3D MRA-CapsNet-DL, an improved dynamic routing algorithm (DRA) with limiting the update increment (called DL) was first designed to solve the inactivation problem of a large number of vector neurons. Then, MRA was used to build three kinds of multi-path 3D MRA-CapsNet-DL for multi-pathological classification. Besides, we also investigated whether the interpolation perturbation introduced by multi-resolution methods will affect the classification performance. Experiments showed that the proposed 3D MRA-CapsNet-DL can achieve a better classification (79.27% ACC and 0.9056 AUC) effect than classical convolutional neural networks (CNNs), and the interpolation disturbance has favorable and unfavorable effects on the classification performance of multi-path and single-path models, respectively.
CONFLICT OF INTEREST
The authors declare no conflicts of interest.
Open Research
DATA AVAILABILITY STATEMENT
The raw/processed lung CT image dataset generated or analyzed during this study is not publicly available as the Dicom metadata containing information that could compromise patient privacy/consent and the data also forms part of an ongoing study.
REFERENCES
- 1Sung H, Ferlay J, Siegel RL, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021; 71(3): 209-249. doi:10.3322/caac.21660
- 2Rivera MP, Mehta AC, Wahidi MM. Establishing the diagnosis of lung cancer: diagnosis and management of lung cancer: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2013; 143(5): e142S-e165S. doi:10.1378/chest.12-2353
- 3Heerink WJ, de Bock GH, de Jonge GJ, Groen HJ, Vliegenthart R, Oudkerk M. Complication rates of CT-guided transthoracic lung biopsy: meta-analysis. Eur Radiol. 2017; 27(1): 138-148. doi:10.1007/s00330-016-4357-8
- 4Yanagawa M, Niioka H, Kusumoto M, et al. Diagnostic performance for pulmonary adenocarcinoma on CT: comparison of radiologists with and without three-dimensional convolutional neural network. Eur Radiol. 2021; 31(4): 1978-1986. doi:10.1007/s00330-020-07339-x
- 5Xie Y, Zhang J, Xia Y, Fulham M, Zhang Y. Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT. Inform Fusion. 2018; 42: 102-110. doi:10.1016/j.inffus.2017.10.005
- 6Wang Q, Cheng JZ, Liu Z, et al. Multi-order transfer learning for pathologic diagnosis of pulmonary nodule malignancy. 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE; 2018: 2813-2815. doi:10.1109/BIBM.2018.8621407
10.1109/BIBM.2018.8621407 Google Scholar
- 7LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998; 86(11): 2278-2324. doi:10.1109/5.726791
- 8He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; IEEE; 2016: 770-778. doi:10.1109/CVPR.2016.90
10.1109/CVPR.2016.90 Google Scholar
- 9Xie Y, Xia Y, Zhang J, et al. Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT. IEEE Trans Med Imaging. 2018; 38(4): 991-1004. doi:10.1109/TMI.2018.2876510
- 10da Nobrega RVM, Rebouças Filho PP, Rodrigues MB, da Silva SP, Júnior CMD, de Albuquerque VHC. Lung nodule malignancy classification in chest computed tomography images using transfer learning and convolutional neural networks. Neural Comput Appl. 2018; 32(15): 11065–11082. doi:10.1007/s00521-018-3895-1.
- 11Shen W, Zhou M, Yang F, Yang C, Tian J. Multi-scale convolutional neural networks for lung nodule classification. International Conference on Information Processing in Medical Imaging. Springer; 2015: 588-599. doi:10.1007/978-3-319-19992-4_46
10.1007/978?3?319?19992?4_46 Google Scholar
- 12Shen W, Zhou M, Yang F, et al. Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification. Pattern Recognit. 2017; 61: 663-673. doi:10.1016/j.patcog.2016.05.029
- 13Jiang H, Gao F, Xu X, Huang F, Zhu S. Attentive and ensemble 3D dual path networks for pulmonary nodules classification. Neurocomputing. 2020; 398: 422-430. doi:10.1016/j.neucom.2019.03.103
- 14Armato SG III, McLennan G, Bidaut L, et al. The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys. 2011; 38(2): 915-931. doi:10.1118/1.3528204
- 15Ather S, Kadir T, Gleeson F. Artificial intelligence and radiomics in pulmonary nodule management: current status and future applications. Clin Radiol. 2020; 75(1): 13-19. doi:10.1016/j.crad.2019.04.017
- 16Gong J, Liu JY, Jiang YJ, Sun XW, Zheng B, Nie SD. Fusion of quantitative imaging features and serum biomarkers to improve performance of computer-aided diagnosis scheme for lung cancer: a preliminary study. Med Phys. 2018; 45(12): 5472-5481. doi:10.1002/mp.13237
- 17Zhao Z, Chen J, Yin X, Song H, Wang X, Wang J. Establishing assistant diagnosis models of solitary pulmonary nodules based on intelligent algorithms. Cell Physiol Biochem. 2015; 35(6): 2463-2471. doi:10.1159/000374046
- 18Sabour S, Frosst N, Hinton GE. Dynamic routing between capsules. Proceedings of the 31st International Conference on Neural Information Processing Systems; ACM; 2017: 3859-3869. doi:10.5555/3294996.3295142
10.5555/3294996.3295142 Google Scholar
- 19Dempster AP, Laird NM, Rubin DB. Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B Methodol. 1977; 39(1): 1-22. doi:10.1111/j.2517-6161.1977.tb01600.x
10.1111/j.2517-6161.1977.tb01600.x Google Scholar
- 20Do Rosario VM, Borin E, Breternitz M. The multi-lane capsule network. IEEE Signal Process Lett. 2019; 26(7): 1006-1010. doi:10.1109/LSP.2019.2915661
- 21Cheng X, He J, He J, Xu H. Cv-CapsNet: complex-valued capsule network. IEEE Access. 2019; 7: 85492-85499. doi:10.1109/ACCESS.2019.2924548
- 22Rajasegaran J, Jayasundara V, Jayasekara S, Jayasekara H, Seneviratne S, Rodrigo R. Deepcaps: going deeper with capsule networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; IEEE; 2019: 10725-10733. doi:10.1109/CVPR.2019.01098
10.1109/CVPR.2019.01098 Google Scholar
- 23Xiang C, Zhang L, Tang Y, Zou W, Xu C. MS-CapsNet: a novel multi-scale capsule network. IEEE Signal Process Lett. 2018; 25(12): 1850-1854. doi:10.1109/LSP.2018.2873892
- 24Krizhevsky A, Hinton G. Learning Multiple Layers of Features from Tiny Images. Technical report. University of Toronto; 2009.
- 25Dou Q, Chen H, Yu L, Qin J, Heng PA. Multilevel contextual 3-D CNNs for false positive reduction in pulmonary nodule detection. IEEE Trans Biomed Eng. 2016; 64(7): 1558-1567. doi:10.1109/TBME.2016.2613502
- 26Xu X, Wang C, Guo J, et al. MSCS-DeepLN: evaluating lung nodule malignancy using multi-scale cost-sensitive neural networks. Med Image Anal. 2020; 65:101772. doi:10.1016/j.media.2020.101772
- 27Xie Y, Zhang J, Xia Y. Semi-supervised adversarial model for benign–malignant lung nodule classification on chest CT. Med Image Anal. 2019; 57: 237-248. doi:10.1016/j.media.2019.07.004
- 28Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate; 2014. arXiv preprint arXiv:1409.0473.
- 29Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in Neural Information Processing Systems; ACM; 2017: 5998-6008. doi:10.5555/3295222.3295349
10.5555/3295222.3295349 Google Scholar
- 30Cho K, Courville A, Bengio Y. Describing multimedia content using attention-based encoder-decoder networks. IEEE Trans Multimedia. 2015; 17(11): 1875-1886. doi:10.1109/TMM.2015.2477044
- 31Zhang J, Xie Y, Xia Y, Shen C. Attention residual learning for skin lesion classification. IEEE Trans Med Imaging. 2019; 38(9): 2092-2103. doi:10.1109/TMI.2019.2893944
- 32Xu R, Cong Z, Ye X, et al. Pulmonary textures classification via a multi-scale attention network. IEEE J Biomed Health Inform. 2019; 24(7): 2041-2052. doi:10.1109/JBHI.2019.2950006
- 33Veasey BP, Broadhead J, Dahle M, Seow A, Amini AA. Lung nodule malignancy prediction from longitudinal CT scans with Siamese convolutional attention networks. IEEE Open J Eng Med Biol. 2020; 1: 257-264. doi:10.1109/OJEMB.2020.3023614
- 34Bankier AA, MacMahon H, Goo JM, Rubin GD, Schaefer-Prokop CM, Naidich DP. Recommendations for measuring pulmonary nodules at CT: a statement from the Fleischner society. Radiology. 2017; 285(2): 584-600. doi:10.1148/radiol.2017162894
- 35Abadi M, Barham P, Chen J, et al. Tensorflow: a system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16); USENIX Association; 2016: 265-283.
- 36Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size; 2016. arXiv preprint arXiv:1602.07360.
- 37Polat H, Danaei Mehr H. Classification of pulmonary CT images by using hybrid 3D-deep convolutional neural network architecture. Appl Sci. 2019; 9(5):940. doi:10.3390/app9050940
- 38Jin H, Li Z, Tong R, Lin L. A deep 3D residual CNN for false-positive reduction in pulmonary nodule detection. Med Phys. 2018; 45(5): 2097-2107. doi:10.1002/mp.12846
- 39Shaffie A, Soliman A, Fu XA, et al. A novel technology to integrate imaging and clinical markers for non-invasive diagnosis of lung cancer. Sci Rep. 2021; 11(1): 1-10. doi:10.1038/s41598-021-83907-5
- 40Xing W, Sun H, Yan C, et al. A prediction model based on DNA methylation biomarkers and radiological characteristics for identifying malignant from benign pulmonary nodules. BMC Cancer. 2021; 21(1): 1-11. doi:10.1186/s12885-021-08002-4
- 41Mehta K, Jain A, Mangalagiri J, Menon S, Nguyen P, Chapman DR. Lung nodule classification using biomarkers, volumetric radiomics, and 3D CNNs. J Digit Imaging. 2021; 34(3): 647-666. doi:10.1007/s10278-020-00417-y
- 42Gao R, Tang Y, Xu K, et al. Deep multi-path network integrating incomplete biomarker and chest CT data for evaluating lung cancer risk. Medical Imaging 2021: Image Processing. Vol 11596. International Society for Optics and Photonics; 2021: 115961E. doi:10.1117/12.2580730
10.1117/12.2580730 Google Scholar
- 43Lu S, Lu Z, Zhang YD. Pathological brain detection based on AlexNet and transfer learning. J Comput Sci. 2019; 30: 41-47. doi:10.1016/j.jocs.2018.11.008
- 44Lu S, Zhu Z, Gorriz JM, Wang SH, Zhang YD. NAGNN: classification of COVID-19 based on neighboring aware representation from deep graph neural network. Int J Intell Syst. 2022; 37(2): 1572-1598. doi:10.1002/int.22686
