1 INTRODUCTION

The recent pandemic of severe acute respiratory syndrome coronavirus 2 disease (COVID-19) is posing great health concerns globally.^{1, 2} The COVID-19 pandemic has resulted in loss of lives, health, and economic issues.³ Although a large number of trials have been conducted to produce vaccines and/or treat COVID-19, a specific vaccine or therapy is still lacking.^{4, 5} For the diagnosis of COVID-19, reverse transcription-polymerase chain reaction (RT-PCR) is a high sensitive molecular test, but bears inherently a number of limitations.^{6, 7} Furthermore, previous studies have indicated that thoracic computed tomography (CT) is a fast and highly sensitive approach for COVID-19 detection and management.^{8, 9} In this regard, dedicated ultralow-dose CT scanning protocols were recently devised.¹⁰

In connection with the use of CT in COVID-19 management, a wide range of qualitative and quantitative studies have been carried out for diagnostic, prognostic and longitudinal follow-up of patients.^11-13 In these studies, whole lungs or infectious lesions were analyzed and several patterns and features were found to have high diagnostic and prognostic value.^13-18 However, accurate segmentation of lungs and infectious pneumonia lesions remains challenging.¹⁹ Hence, segmentation is the main issue impacting the outcome of both qualitative and quantitative studies.^{12, 19, 20} Although several segmentation approaches including manual delineation, semiautomated²¹ and fully automated²⁰ techniques have been applied to CT images for COVID-19 management, they are still facing serious challenges to produce robust and dependable outcomes.

In medical image segmentation, particularly whole three-dimensional (3D) volumes definition and big data analysis, manual delineation requires experienced trained radiologists, is time consuming, labor-intensive, and suffers from interobserver and intraobserver variability concerns.^{22, 23} Whole lung segmentation is a pivotal step for further analysis, including extraction of the percentage of infection, well aerated portion of the lung, and enabling radiomics and deep learning (DL) analysis of COVID-19 patients.^{14, 17} Conventional algorithms, including rule-based and atlas-based, performed relatively well on normal and mild disease chest CT, but might fail in COVID-19 patients lung segmentation because of different stages of disease with different levels of severity.¹⁹ Furthermore, developing a fully automatic tool for lung and pneumonia COVID-19 lesions is highly desired owing to rapid changes in appearance and manifestation at different stages of the disease.^{13, 19}

Artificial intelligence (AI) algorithms, particularly its two major subcategories, machine learning (ML), and DL, have been widely used for medical image analysis^24-31 and more recently in the segmentation of lung and pneumonia infectious lesions from chest CT images of COVID-19 patients.¹⁵ These studies reported that AI improved the accuracy of lesion detection/segmentation and reduced the bias associated with conventional approaches. In a study by Zheng and coworkers,³² a weakly supervised DL algorithm was applied to chest CT images for automatic COVID-19 detection. Fan et al.³³ presented a COVID-19 lung infection segmentation deep network (Inf-Net) based on semisupervised learning. Furthermore, a number of DL algorithms, namely UNet, UNet++, V-Net, Attention-UNet, Gated-UNet, and Dense-UNet were used for COVID-19 lesion detection and segmentation from chest CT images.^{34, 35}

CT images are commonly acquired on various scanner models using different imaging protocols, and as such, the resulting datasets are heterogeneous, which might lead to inaccuracy in the developed models. Training a robust and generalizable DL model requires a large clean annotated dataset.³⁶ Owing to the relatively recent outbreak of COVID-19 pandemic, producing a large labeled COVID-19 image dataset is impractical. Transfer learning (TL) has received attention to address the lack of large datasets for the implementation of machine/DL-based algorithms.^{37, 38} Various TL-based strategies were used for transferring knowledge from different domains, including natural images to medical images to develop more robust and generalizable models.³⁸

In the present study, we developed a DL-based automated detection and segmentation of lung and COVID-19 pneumonia infectious lesions (COLI-Net) from chest CT images. In this work, large lung and COVID-19 lesions datasets and TL used to train a residual network (ResNet) for lung and pneumonia infectious lesions segmentation.

2 MATERIALS AND METHODS

2.5 Residual neural network

The ResNet proposed by Li et al.^{45, 46} built upon TensorFlow was used for lung and COVID-19 lesions segmentation. The ResNet is composed of 20 convolutional layers where different dilation factors were used for different levels of feature extraction (zero dilatation factor for low-level, two dilatation factors for medium-level, and four dilatation factors for high level). Every two layers were linked together with residual connections (Figure 1). Non-square Dice was used as loss function, and Figure 1 provides descriptive detail of ResNet.

2.7 Evaluation

To evaluate the performance of image segmentation, we calculated Dice similarity coefficient (Equation (1)), Jaccard index (Equation (2)), false negative (Equation (3)), false positive (Equation (4)), mean surface distance (Equation (5)), and mean Hausdorff distance (Equation (6)). In addition, different volume indices were exploited to quantify the portion of infection, including relative volume difference (%), relative volume difference of lesion/lung relative volume (%) (Equation (7)), absolute relative volume difference (%) (Equation (8)), and absolute relative volume difference of lesion/lung relative volume (%). Hounsfield unit (mean) relative difference (%), and Hounsfield unit (mean) absolute relative difference (%) were calculated for lungs and COVID-19 lesions from different segmentations of CT images. In addition, we evaluated the impact of the segmentation on 17 first-order and 10 shape radiomic features in both lungs and COVID-19 lesions. The list of radiomic features is presented in Supplemental Table 1.

(1)

(2)

(3)

(4)

(5)

where is the distance between a point p belonging to the surface of a 3D surface predicted image (P) and its closest distance between the two surfaces P and G (ground truth).

(6)

(7)

(8)

3 RESULTS

Figures 2 and 3 compares visually in 2D and 3D views for different external validation sets of lungs and lesions delineated manually by experienced radiologists and automatically by the DL model. Additional results from the external validation sets are provided in Supplemental Figures 1–13 (2D views) and 14–17 (3D views). Overall, there is good agreement between manual and predicted lung and infectious lesions segmentation in the different datasets. Despite the variability of the subjects among the different centers, COLI-Net performed consistently well in multicentric and multiscanner setting. What stands out from these results is that COLI-Net can detect and segment infectious regions (within lesion segmentation) while excluding arteries and tracheae in lung segmentation.

Table 1 summarizes segmentation quantification metrics for lungs and COVID-19 lesions. It can be seen that the mean Dice coefficients were 0.98 ± 0.011 (95% CI, 0.98–0.99) and 0.91 ± 0.038 (95% CI, 0.90–0.91) for lung and lesions segmentation, respectively. The mean Jaccard index was 0.97 ± 0.022 (95% CI, 0.97–0.97) and 0.83 ± 0.062 (95% CI, 0.82–0.84) for lung and COVID-19 lesions segmentation, respectively. Lung segmentation in Russia datasets exhibited better results compared to the other centers/datasets. This might be attributed to the homogeneity and mild severity of the lesions in this dataset. Supplemental Tables 2–7 summarize lung and lesion segmentation quantification metrics for different external validation sets.

Table 2 summarizes the impact of lung and lesions segmentations on mean Hounsfield unit and volume calculation. Mean relative HU differences (%) of 0.03 ± 0.84 (95% CI, −0.12 to 0.18) and −0.18 ± 3.4 (95% CI, −0.8 to 0.44) were achieved for lungs and lesions, respectively. The relative volume difference for the lung was 0.38 ± 1.2 (95% CI, 0.16–0.59) whereas it was 0.81 ± 6.6 (95% CI, −0.39 to 2) for lesions. The results obtained from the mean Hounsfield unit and volume calculation for lung and infectious lesions for the different external validation sets are presented in Supplemental Tables 8–11.

Figures 4 and 5 depict the Dice similarity index, Jaccard, mean Hounsfield unit, and volume difference box plots for lung and lesions segmentation, respectively. Supplemental Figures 18 and 19 show box plots of Hounsfield unit absolute relative difference (%), absolute relative volume difference (%), false negative, false positive, average Hausdorff distance, and mean surface distance for lung and lesions.

Descriptive statistics of relative volume (lesion/lung) indices are presented in Table 3. A relative error of 0.22 ± 6.3 (95% CI, −0.95 to 1.4) and absolute relative error of 4.7 ± 4.2 (95% CI, 3.9–5.5) were achieved for relative volume (lesion/lung). Supplemental Tables 12 and 13 summarize the results obtained for the relative volume (lesion/lung) index for different external validation sets. Figure 6 depicts boxplot of manual and predicted relative volume lesion/lung differences (%) and absolute/relative error of lesion/lung relative volume errors (%) for different external validation sets.

Figure 7 presents heatmap of the mean relative error of first-order and histogram shape radiomic features in the lung and lesions for different validation sets. Most radiomic features exhibited a mean relative error less than 5% with the highest mean relative error for the lung being −6.95% for range first-order feature and least axis length shape feature (8.68%) in lesions. The heatmap of the mean absolute relative error is depicted in Supplemental Figure 20.

4 DISCUSSION

Chest CT imaging has emerged as a complementary tool for COVID-19 early diagnosis and longitudinal follow-up.⁸ However, a number of challenges still need to be addressed for the accurate diagnosis of COVID-19 and its differentiation from other lung diseases, such as viral and bacterial pneumonia and other respiratory diseases.¹⁷ In this regard, several AI-based solutions exhibiting different levels of accuracy and robustness were proposed and evaluated.^{14, 17}

Another challenging problem that arises in the domain of quantitative analysis of CT images in clinical practice is lung and pneumonia infectious lesions segmentation.¹⁹ At the outset, different complex manifestations (appearance, size, location, boundaries and contrast) of infectious lesions, including consolidation, reticulation, and ground-glass opacity at different stages of the disease (longitudinal changes in the same patients) have been observed.¹³ Furthermore, providing ground truth segmentation for infectious lesion segmentation is challenging owing to interobserver/intraobserver variability, noisy annotations, and the long processing time.¹⁹

Previously developed atlas,⁴⁷ rule,⁴⁸ and hybrid (atlas and rule)⁴⁹ based algorithms for lung segmentation have shown acceptable performance on normal lungs and in the presence of mild pathogens (low density), such as emphysema.⁵⁰ However, they presented limited performance in severe conditions (high density), including pleural effusion, atelectasis, consolidation, fibrosis, and pneumonia.⁵¹ Recent developments in the field of ML have led to a renewed interest in automatic lung segmentation. However, most seminal works in this area used a limited training dataset, predominantly containing normal cases or focusing on one class of pathogeneses, which could impact generalizability for unseen/non-diagnosed test datasets.⁵² In the present study, we applied DL algorithms and TL on CT images obtained from different imaging centers to detect and segment the whole lung and pneumonia infected regions in COVID-19 patients.

A number of previous works attempted to develop automated segmentation algorithms for lung and infectious lesions in COVID-19 CT images. Hofmanninger et al.⁵¹ developed models for lung segmentation and reported a Dice coefficient of 0.98 ± 0.01 for different pathological states (atelectasis, fibrosis, mass, pneumothorax, and trauma). They concluded that diversity in the training dataset is more important than the DL algorithms. Müller et al.⁵³ implemented a 3D U-Net using data augmentation for generating image patches during training for lung and lesion segmentation on 20 annotated CT volumes. They achieved Dice coefficients of 0.950 and 0.761 for lung and lesions, respectively. A modified 3D U-Net (feature variation and progressive atrous spatial pyramid pooling blocks) proposed by Yan et al.³⁴ was developed for lung and infectious lesion segmentation on 861 patients, reporting a Dice similarity index of 0.987 for lung and 0.726 for lesions segmentation. Moreover, comparisons were performed with a dense fully convolutional network (lung: 0.865, lesions: 0.659)⁵⁴; U-Net (lung: 0.987, lesions: 0.688)⁵⁵; V-Net (lung: 0.983, lesions: 0.625)⁵⁶; and U-Net++ (lung: 0.986, lesions: 0.681).⁵⁷ The mean Dice coefficient for lung and lesions segmentation for different external validation sets used in our work were 0.98 ± 0.011 and 0.91 ± 0.038, respectively.

Chen et al.⁵⁸ used the residual attention U-Net for multi-class segmentation of CT images, achieving a Dice coefficient of 0.94 for infectious lesions segmentation. Zhou et al.³⁵ used a modified U-net network through spatial and channel attention mechanisms along with focal Tversky loss in the training process for improving small lesions segmentation. The results were evaluated on 427 slices achieving a Dice coefficient of 0.83. Elharrouss et al.⁵⁹ adopted an encoder-decoder for infectious lesions segmentation using 20 clinical studies from the Italian Society of Medical and Interventional Radiology to report a Dice coefficient of 0.786. They compared the results with U-Net (Dice: 0.439),⁶⁰ Attention-UNet (Dice: 0.583),⁶¹ Gated-UNet (Dice: 0.623),⁶² Dense-UNet (Dice: 0.515),⁶³ U-Net++ (Dice: 0.422),⁵⁷ and Inf-Net (Dice: 0.739).³³ Wang et al.⁶⁴ proposed a robust algorithm for COVID-19 infectious lesions segmentation from CT images (COPLE-Net) designed to learn from noisy labeled data. The algorithm relies on noise-robust Dice loss and mean absolute error loss for generalized Dice loss for robust segmentation of noisy datasets and a modified version of U-Net to better handle infectious lesion segmentation with various manifestations and scales. The best results achieved by COPLE-Net were 0.807 ± 0.099 and 0.160 ± 0.171% as Dice coefficient and relative volume error (RVE [in %]) respectively. Wang et al.⁶⁴ evaluated different DL algorithms, including modified 3D U-Net (3D New-Net U-Net, Dice: 0.704 ± 0.187, RVE: 25.41 ± 24.73%),⁶⁵ modified 2D U-Net (2D New-Net U-Net, Dice: 0.791 ± 0.129, RVE: 18.37 ± 17.43%),⁶⁵ spatial attention gate U-Net (Attention U-Net, Dice: 0.772 ± 0.123, RVE: 19.77 ± 18.41%),⁶¹ spatial and channel “squeeze and excitation” blocks with U-net (ScSE U-Net, Dice: 0.780 ± 0.125, RVE: 18.85 ± 16.69%),⁶⁶ and light-weight power efficient and general purpose CNN (ESPNetv2, Dice: 0.698 ± 0.148, RVE: 23.69 ± 20.26%).⁶⁷ Our proposed COLI-Net approach showed good performance compared to previous studies with a Dice coefficient of 0.91 ± 0.038 (95% CI: 0.90–0.91) and RVE of 0.38 ± 1.2% (95% CI: 0.16–0.59) for pneumonia infectious lesions.

A large labeled dataset is required to build a robust and generalizable model while avoiding overfitting. Previous studies attempted to transfer the knowledge from natural to medical imaging domain, leading to improved accuracy by addressing the issue of limited datasets.^{37, 38} TL was recently applied for the detection and classification of COVID-19 using chest x-ray and CT images.^{68, 69} More recently, Wang et al.⁷⁰ applied four TL methods on COVID-19 CT images for the segmentation of infectious lesions using 3D U-Net. The information was transformed from cancer and pleural effusion data to COVID-19 lesion segmentation. The Dice coefficient increased from 0.673 ± 0.22 to 0.703 ± 0.20 after TL.⁷⁰ They concluded that the transferability of non-COVID-19 data improved the quality of COVID-19 lesion segmentation to build a robust segmentation model. In our study, we exploited TL from a large multicentric lung labeled dataset with various pathologies to overcome the shortcomings of infectious lesion segmentation.

Li et al.⁷¹ used thick-section chest CT images of 531 COVID-19 patients for automatic segmentation of lesions using 2.5D U-net to achieve Dice coefficients of 0.74 ± 0.28 and 0.76 ± 0.29 with respect to manual delineation performed by the two radiologists. The interobserver variability measured by the Dice metric was 0.79 ± 0.25 between two radiologists. They calculated two imaging biomarkers, including the percentage of infection and average infectious HU for severity and progression assessment, resulting in AUC of 0.97. Thick-section CT imaging was recommended for high-pitch scans to decrease the acquisition time and motion artifacts (due to breath holding) and reduce radiation doses to patients.^{10, 72} In our dataset, various slice thicknesses (1–8 mm) have been included to train a robust network against this parameter, which highly impacts image manifestations. The relative error of volume difference for the percentage of infections (lesion/lung) and relative mean HU Diff (%) were 0.22 ± 6.3% (95% CI: −0.95 to 1.4%) and −0.18 ± 3.4% (95% CI: −0.8 to 0.44%), demonstrating the high accuracy of COLI-Net for biomarker generation.

Potential foreseen applications are not limited to the detection and segmentation but could be useful in providing diagnostic and prognostic parameters calculated using lung and infections segmentation to estimate the percentage of infections, and enabling advanced image processing in COVID-19 patients. The existing body of research on pneumonia suggests that the pneumonia severity index (PSI) can potentially be used as a severity marker.⁷³ A recent study classified COVID-19 patients into severe and nonsevere patients based on PSI calculated using CT images.⁷⁴ Different DL algorithms and radiomics analysis approaches using CT images have been examined recently for developing diagnostic (discriminating COVID-19 from bacterial/viral pneumonia) and prognostic (survival, hospital stay, intensive care unit [ICU] admission, risk of outcome) models, which require lung and lesion segmentation.¹⁷ Moreover, calculating the percentage of infection and well-aerated regions in the lung are frequently performed through visual assessment or by simply calculating HU values in the lungs, which is not only time-consuming but also lacks accuracy.

The established model exhibited noticeable performance variation across different COVID-19 patients collected from different countries, centers, with different patient backgrounds, and stages of the disease. Since the quality of CT images depends directly on the scanner model, imaging protocol (tube voltage, tube current, pitch factor, etc.), and reconstruction algorithm, we employed various datasets from different centers to cover a large variability.10, 72 Although the proposed algorithm was evaluated using a multicenter, multiscanner, multinational dataset and patients with a diverse background, stages of the disease, a full-scale adaptation of this model requires further clinical investigation and fine-tuning to the specific image acquisition parameters of a center. This framework provides multiple imaging biomarkers for COVID-19 patients to facilitate the assessment of their clinical relevance in diagnostic (discriminating COVID-19 from bacterial/viral pneumonia) and prognostic (survival, hospital stay, ICU admission, risk of outcome) applications. Further development should involve implementing lung lobes segmentation to calculate all potential imaging biomarkers at the lobes level. In this study, we used only the ResNet architecture for model evaluation. However, further evaluations should be conducted to compare different models, including UNet, VNet, and GAN architectures.

5 CONCLUSION

We set out to develop an automated algorithm capable of segmenting 3D whole lung and infected regions in COVID-19 patients from chest CT images using DL techniques to enable fast, consistent, robust, and human error immune framework for lung and pneumonia lesion detection and delineation. Owing to the complex nature of the problem and high variability in lesion manifestation, TL from whole lungs to pneumonia infection lesions was proposed and implemented to enrich specific COVID-19 pneumonia features identification from clinical studies. Moreover, a multicentric and multiscanner dataset was collected for the development of the DL model to establish an automated and generalizable platform for efficient COVID-19 patients management. The developed AI model was evaluated using a wide range of COVID-19 patients of diverse populations with different stages of the disease from multiple centers around the world to enable big data analysis of COVID-19 for automated progression/regression assessment of pneumonia lesions in follow-up studies, provide diagnostic and prognostic metrics, and enable further advanced image processing.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

AUTHOR CONTRIBUTIONS

Isaac Shiri and Hossein Arabi are the co-first authors of this paper. Mostafa Ghelich Oghli and Habib Zaidi contributed to the study conception and design. Isaac Shiri and Hossein Arabi designed, implemented, and evaluated the image segmentation and ML framework. Yazdan Salimi, Amirhossein Sanaat, Azadeh Akhavanallaf, Ghasem Hajianfar, Dariush Askari, Shakiba Moradi, Zahra Mansouri, Masoumeh Pakbin, Saleh Sandoughdaran, Hamid Abdollahi, Amir Reza Radmard, and Kiara Rezaei-Kalantari collated the datasets. Habib Zaidi contributed to the study conception and design, initial draft of the manuscript and supervision and funding of this work. All authors contributed to the data preparation and revision of the manuscript for important content.