2.1 Overview of proposed CT-image analysis

Grayscale statistics extracted from CT pulmonary-parenchyma images can be exploited to provide valuable insight into the lung's alveolar tissue employed in respiratory processes, and this study focuses on that information. CT-image slices are rapidly analyzed to yield values for a range of grayscale statistical attributes. Interpretations considering combinations of these statistics with graphic representations identify distributions related to the severity of lung impacts related to COVID-19 that, to some extent, overlap with each other. However, DL/ML methods can be applied to combinations of these overlapping, grayscale-attribute distributions to accurately the severity of lung abnormalities of each CT-image slice assessed.

2.2 Ethical approval

The CT analyses of human patients used in this study were performed in accordance with the relevant guidelines and regulations of Namazi Medical Centre and the Medical School of Shiraz University of Medical Sciences (Shiraz, Iran). In particular, the consent for analysis was obtained from each patient and their anonymity was guaranteed. Consequently, the confidentiality of the biographical information pertaining to each patient has been maintained at all times. All CT analytical protocols applied were approved by the Medical School of Shiraz University of Medical Sciences.

2.3 CT-scan slice procurement

The suites of lung CT images were obtained from a representative set of patients. Most of them were shown by laboratory testing to be afflicted with COVID-19. However, images from several patients with negative COVID-19 tests were included in the suite of images compiled. These patients were all under treatment at the Namazi Medical Centre, Shiraz, Iran. The Philips Ingenuity CT-scanning equipment (Philips) located at that medical center provided suites of CT-scan-slice images (0.625 mm thickness) for all patients assessed. Such multislice-CT-scan equipment is now widely used to rapidly deliver high-resolution images.²⁶

CT images considered are all noncontrasted, which are typical of those widely used to assess various lung conditions in addition to those caused by coronaviruses. The CT-scan-slice images utilized are derived from 49 COVID-positive patients displaying a substantial range of visible features known to be commonly associated with COVID-19. Additionally, multiple CT-scan-slice images associated with eight COVID-negative patients are also assessed for comparative purposes. The COVID-negative patients consisted of four women with ages ranging from 18 to 68 years, together with four men with ages varying from 24 to 71 years. These images were compared with those from 23 female COVID-positive patients with ages ranging from 22 to 74 years, together with 26, COVID-positive, male patients with ages ranging from 32 to 86 years. These patients’ images were selected to provide well-balanced distributions of patients’ genders and ages.

2.4 CT-scan-slice image assessment

2.4.5 Workflow of the methodology applied to assess CT images

Figure 1 summarizes the workflow applied in this study to assess and classify thoracic CT images in terms of their COVID-19 impacts.

3.1 Grayscale image statistics

The grayscale statistical-attribute distributions of the CT-extract images evaluated are described in Supporting Information: Table S2. Figure 2 displays and compares box and whisker plots for selected CT-extract-image grayscale attribute distributions relating to the 121 COVID-19-negative individuals and the 392 COVID-19-positive individuals.

Pixel P50, pixel average, and pixel P90 grayscale statistics display some the most obvious distinctions between distributions for COVID-19-negative and COVID-19-positive individuals (Figure 2). It is these more obvious differences, plus the more subtle differences associated with other grayscale attribute statistics that the DL/ML models use in their VS class predictions.

Some of the grayscale statistics relating to the CT-extract images also display strong correlations with their VS classification (Table 2). The Pearson's correlation coefficient (R) values are highly positive for VS versus grayscale, P50, P90, average, variance, and standard deviation. R values for VS versus P10, standard error, and variance/average ratio are moderately positive. On the other hand, the R value for VS versus the percentage of pixels at the average grayscale value is highly negative. The dispersion of the grayscale-attribute distributions, and their correlations with VS, are indicative of systematic variations occurring in these attributes that are spread across the VS classes throughout the CT-extract-image data set.

Spearman's rank correlation coefficient (p) is also calculated for VS versus the grayscale statistical attributes and displayed in Table 2. A feature of the R statistic is that an assumption is made that the two-variable distributions compared both conform to normal distributions, that is, they display parametric behavior. The nonparametric p statistic avoids that assumption as it is calculated based on the ranked position of each data record in the respective distributions instead of their absolute values. For VS versus the grayscale attributes the p and R values are in relatively close agreement (Supporting Information: Figure S3). This implies that the grayscale attribute distributions are relatively symmetrical and not highly skewed.

3.2 CT-extract image grayscale attribute dispersions across VS classes

Figure 3 displays a ternary plot for three grayscale attributes: average; variance; and, pixel% at the average value. The scaling factors applied to the grayscale attributes serve to centralize the spread of data points on this ternary plot. In such plots, the actual values of the three variables are normalized to sum to a constant value (the value of 100 is used for Figure 3).

These normalized values cannot then vary in the diagram independently of each other. Thus, if the normalized values of two of the variables plotted are known, the value of the third can be determined. This configuration creates only two degrees of freedom in the normalized system, allowing the three variables to be displayed in the two-dimensional ternary plots (Figure 3).

The three grayscale attributes (average; variance; pixel% at the average) perform relatively good discrimination between the VS classes (Figure 3). There is an almost continuous progression from VS = 0 (lower-left in ternary plot) to VS = 4 (middle-right in ternary plot). VS Classes 0 and 3 appear in distinct regions on the ternary plot. Nevertheless, overlaps are apparent among VS Classes 1, 2, and 4 and between those classes and VS Classes 0 and 3. Lungs with little or no evidence of abnormal features tend to be associated with relatively low average and variance grayscale values. As abnormal features materialize in the lung-image extracts, grayscale average and variance values progressively increase. At the stage where abnormal features become extensive in the lung-image extracts, the percentage of image pixels at the average grayscale for the image tends to decrease. This tendency forms a distinctive characteristic in the VS Class 4 images.

The images displaying the severest abnormal lung features plot towards the lower-right portion of the ternary plot (Figure 3). Such images show a prevalence of lighter-gray pixels. This causes the grayscale variance value to decline and the grayscale average value to increase. From VS = 0 to VS = 3 the image positions tend to move from south-west to north-east position in Figure 3. Whereas, from VS = 3 to VS = 4 the image positions tend to move in a southerly direction in Figure 3.

Three-variable plots (Figures S4 and S5) reveal other progressive and distinctive trends that exist involving grayscale percentage statistics. The involvement of grayscale P90 in Supporting Information: Figure S4 displays a continuous trend from the base-right portion (VS = 0) to top-left left portion (VS = 3 or 4) with substantial overlap between VS Classes 3 and 4. By involving both grayscale P10 and P90 with grayscale average (Figure S5) the overlap between classes VS = 3 and VS = 4 is somewhat reduced.

The distributions of grayscale statistics from CT-extract images (Supporting Information: Table S2, Figures S4 and S5, and Figure 3) include substantial dispersion and inter-relationships. These can be exploited graphically (and by DL/ML methods) to classify the degree of abnormalities in lungs affected by COVID-19. The three-statistical-attribute combinations displayed in Figures 3, Supporting Information: S4 and S5 discriminate quite well between some of the VS classes. However, overlapping positions, particularly between VS Classes 0 and 1 and VS Classes 3 and 4 make a graphical approach unsuited to definitive VS classification. Involving DL/ML methods to consider multiple grayscale attributes as input variables offers the potential to improve upon the graphical approach.

3.3 Feature selection for DL/ML analysis

Thirteen grayscale-attribute statistics were extracted and computed for each of the (513) CT-extract images (Supporting Information: Table S2). All of these attributes could be employed as independent (input) variables with VS class as the single dependent variable for DL/ML model evaluation. However, as some of these attributes show relatively low correlations with VS class (Table 2 and Supporting Information: Figure S3) trial and error sensitivity analysis was conducted to determine their impact on the DL/ML prediction performance. The following sensitivity tests were conducted:

Test #1: 9 input variables (pixel#, standard error, minimum, variance/average ratio excluded)

Test #2: 10 input variables (pixel#, standard error, variance/average ratio excluded)

Test #3: 11 input variables (pixel# and variance/average ratio excluded)

Test #4: 12 input variables (pixel# excluded)

Test #5: 13 input variables

The results showed that Test #4 with 12-input variables included achieved the highest VS prediction accuracy. This suggests that all 12 of those attributes can provide useful input to the DL/ML models. Those 12 input features were, therefore, selected for detailed DL/ML model analysis.

3.4 K-fold cross-validation analysis

K-fold cross-validation is applied to evaluate, select and justify the percentage splits of the data records between training and testing subsets to be applied in ML/DL analysis. Four specific folds or cases are evaluated for five algorithms. Those cases are: fourfold (75% training: 25% testing), fivefold (80% training: 20% testing), 10-fold (90% training: 10% testing), and 15-fold (93% training: 7% testing). Three runs are executed for each k-fold analysis case, thereby generating meaningful statistics for each case. Twelve distinct splits are evaluated for fourfold case (3 times 4). The number of splits evaluated increases to 45 for the 15-fold analysis (3 times 15). The mean MAE and the standard deviation MAE are recorded for all the splits evaluated for each fold (Table 3).

All four K-folds evaluated generate credible prediction results for the VS and algorithmic scale (AS) classifications (low MAE means and low MAE standard deviations). However, the fivefold (80%:20%) split is selected as this generates the lowest mean MAE and a lower standard deviation than the 10- and 15-fold analysis. Also, the 80%:20% split involves a substantial number (103) of randomly selected images in each testing subset.

3.5 DL/ML model VS class prediction performance

Table 4 shows the VS class prediction accuracy (VS_pred vs. VS_act) for the 11 DL/ML models developed. The results refer to the trained and validated models applied to the entire data set. The number and percentages of misclassified data records is also listed in Table 4. The algorithms are ranked, with the best performing shown as rank 1, and the worst-performing shown as rank 11. The rankings are made based on RMSE and classification error numbers (Table 4). The CNN model is ranked 1 in the VS classification task incurring only 18 misclassifications from the 513 images (96.49% accuracy). Moreover, the CNN model achieved RMSE = 0.19 (with reference to the VS scale of 0–4) and R²= 0.98. The RF model was ranked second in its VS classifications (26 misclassifications; RMSE = 0.26; R²= 0.96; 94.93% accuracy). Also, the ADA, DT, ELM, and KNN models performed well in classifying VS. On the other hand, the MLP, NBC, and QDA did not perform well in classifying VS. The percentage accuracies achieved for this multiclass VS scale prediction analysis compare favorably with the percentage accuracies achieved by recent deep learning studies published for the binary (COVID negative vs. COVID positive) predictions from CT and X-ray scans. For instance, Arora et al.⁶ analyzed 812 CT images and achieved a binary prediction accuracy of 98%; Polsinelli et al.²⁴ analyzed 100 images using CNN and achieved a binary prediction accuracy of 85%; Dansana et al.²³ analyzed 360 CT images using CNN and achieved a binary prediction accuracy of 91%; and Bharati et al.²² analyzed 3411 CT images using DL models and achieved a binary prediction accuracy of 82.42%. On the other hand, Bharati et al.²⁵ analyzed 5606 X-ray images and achieved a binary prediction accuracy of 73%.

Table 4 reveals that certain ML models achieve different rankings based on RMSE compared to rankings based on error numbers. For example, the ELM model has an RMSE-rank of 4 but error rank of 6, whereas ADA has RMSE-rank of 5 but error rank of 4 (Table 4). This is a consequence of the nature of the misclassification errors involved. Misclassifications involving adjacent VS classes have a smaller impact on RMSE than misclassifications that are more than one class away from the correct class.

3.6 Confusion matrix assessment of VS classification

Classification errors made by the DL/ML models may increase, while for the same analysis, RMSE values might decrease (Table 4). Confusion matrices deliver a detailed assessment of multiclass misclassifications on a VS class by class basis. Confusion matrices reveal VS classes predicted with most and least accuracy. The three models considered (CNN; KNN; ADA; Figure S6) perform quite distinctly. CNN (Figure S6A) performs best for VS Class 0 (just 1 misclassification) and, in percentage error terms worst for VS Class 1, although it misclassifies the most data records (6 out of 129) associated with VS Class 3, CNN predicts VS Classes 0, 1, and 2 more convincingly than VS classes 3 and 4. CNN misclassifications only involve erroneous predictions involving adjacent VS classes. Confusion matrices for KNN (Figure S6B) and ADA (Figure S6C) reveal that both models result in 28 misclassifications associated with the 513 CT-extract images evaluated. Nevertheless, the distribution of these misclassifications differs considerably for those two models. KNN has five misclassifications that are beyond the adjacent VS classes, whereas ADA results in eight such extreme misclassifications. Consequently, KNN (RMSE = 0.2887) outperforms ADA (RMSE = 0.3175) overall in its VS class prediction. KNN and ADA struggle most to correctly classify VS Class 2 (11 misclassifications; about 40% of total misclassifications). KNN performs best in classifying VS Classes 0 and 1 (better than CNN). ADA does not deal as well with VS Class 0 as the CNN and KNN models but performs almost as well as the CNN model in its classification of VS Classes 3 and 4.

Analysis of confusion matrices provides a complementary perspective on the RMSE/error count analysis of each model in assessing their overall multiclass classification prediction strengths and weaknesses.

4.1 Formulaic/algorithmic scoring approach to classifying CT-extract images

The relatively good discriminatory performances among the VS classes of the graphical relationships (Figure 3, Supporting Information: Figures S4 and S5), involving just grayscale attributes, makes them exploitable for automated formulaic classification. Such an algorithmic/formulaic scoring system (AS) has the potential to provisionally classify the extent of abnormal lung features in CT-extract images before inspection by a clinician and/or DL/ML analysis. AS scores can complement the VS score assigned by a clinician (i.e., VS based on clinician's expert evaluation; AS based on a defined algorithm). Automating CT-extract-image assessments has the attraction of speed and objectivity. This can provide provisional guidance for a subsequent more detailed expert visual assessment.

An example AS approach involves just five grayscale attributes (P10, average, P90, variance and pixel% at the average value; attributes 10, 2, 12, 5, and 4 in Table 2 and Supporting Information: Figure S3, respectively) corresponding to those used in Figure 3, Supporting Information: Figure S4 and S5 for VS analysis.

Table 5 and Supporting Information: Figure S7 displays the Pearson and Spearman correlation coefficients between the grayscale attribute variables and the AS classes for all CT-extract images. As should be expected, the R and p values are slightly higher than those for correlations between the same variables and the VS classes (Table 2). This is particularly the case for those grayscale variables involved in the formulaic definition of the AS classes.

The 513 CT-extract images assigned to AS classes are shown in Figure 4. An increased degree of segregation of the AS classes is discernible in Figure 4 compared to that achieved for the VS classes in Figure 3. This outcome confirms that the value of AS is a preliminary, automated, objective CT-extract image assessment. The five-attribute AS classification system can be further improved upon by applying DL/ML algorithms to consider all the available attributes.

Table 6 summarizes AS classification prediction accuracy determined by DL/ML models using the same 12 input variables (grayscale attributes) recorded for each of 513 CT-extract images. The results confirm that the DL/ML methods find it easy (fewer misclassifications and much lower RMSE/MAE values) to classify the CT-extract images using the AS classes than the VS classes.

Two of the ML algorithms, RF and DT, make only one classification error (out of 513 CT-extract assignments) in the AS classification exercise. Those two models achieved 99.81% accuracy in classifying AS correctly. That level of classification accuracy is better than that achieved for the VS scale (Section 3.5). It also compares very favorably with the accuracy achieved by published DL studies of CT scans focused on the binary (COVID negative vs. COVID positive) classifications,^{6, 22-25} as already mentioned in Section 3.5.

The CNN (DL model) ranks 6 out of 11 for its AS classification performance. CNN is outperformed by the RF, DT, ADA, GPC, and KNN models for AS classification (Table 6). Nevertheless, the CNN model still achieves and AS classification accuracy of >98%. The confusion matrix analysis (not shown) reveals that DT misclassifies just one CT-extract image of AS = 2 as belonging to Class 1, with RF misclassifying a CT-extract image of AS = 4 as belonging to Class 3.

Alternative combinations of grayscale attributes (distinct from the five attributes used to define AS), and/or alternative classification scales including more or less than four classes, could also be explored. It is possible that such alternative algorithmic systems could also provide useful algorithmic segregation of the CT-extract images in terms of their severity of lung abnormalities. Further research is justified to evaluate such possibilities. Notwithstanding possible alternative algorithmic approaches, the AS system, as defined, is considered to provide a viable algorithmic CT-extract image classification system based on easy-to-measure grayscale attributes.

4.2 Value of DL/ML models in predicting VS and AS classification systems

VS class definitions benefit from the input of a clinician's expertise, potentially taking account of a wider set of information than just image grayscale attributes. However, human judgement, irrespective of the expertise involved, makes VS class assignments by visual inspection somewhat subjective. This means that classifications are likely to vary, in detail, among clinicians. The magnitude of abnormal lung features associated with COVID-19 clearly varies over a broad range, as reflected by the grayscale attribute dispersions recorded (Figures 3, Supporting Information: Figures S4 and S5). Unfortunately, the measured grayscale attributes do not crisply segregate into clusters (e.g., to facilitate simple COVID-19-negative vs. COVID-19-positive classification). This makes the distinction of arbitrarily imposed visual-inspection classes unreliable.

In summary, the potential disadvantages of the VS method are (1) the risk of inconsistency and subjectivity in the VS class assignments of different clinicians; and (2) the slightly overlapping nature of the grayscale attribute clusters relating to the VS groups. Nevertheless, the classification performance of the CNN DL method applied to the prediction of VS classes is extremely good (just 18 misclassifications out of 513 CT-extract images evaluated; 96.49% accuracy). Based on that performance, evaluations with larger datasets may be able to further improve VS classification accuracy. However, it would be difficult to consistently achieve zero VS misclassifications due to the element of subjectivity involved the visual assessment process. By removing that subjectivity and applying a formulaic grayscale assessment, such as the AS system defined and evaluated, more reliable image classification can be reached (Table 6 and Figure 4), achieving up to 99.81% accuracy by DT and RF models. Although the AS method overcomes the disadvantages of the VS method, a remaining disadvantage of the technique is that the extraction of the grayscale image slices from the CT scans currently lacks automation. Future developments planned for the technique are to automate the image slice extraction process.

A comparison of VS Class 0 (COVID-19-negative individuals) and VS Class 1 (COVID-19-positive individuals displaying minimal visual signs in terms of abnormal lung features) classification performance by DL/ML models is worth reflecting upon. CNN and KNN 12-variable models are able to distinguish between these classes with very few errors (Supporting Information: Figure S6). On the other hand, clinicians can only do this, in most cases, with the aid of a rRT-PCR COVID-19 test. This outcome highlights the power of the DL/ML models to use the CT-extract grayscale attributes to achieve valuable distinctions between the most difficult-to-separate VS classes based on visual assessment alone. There is clearly valuable grayscale attribute information available in the CT-scan images. The DL/ML methods offer a rapid and highly effective way of extracting that information in a way that could be used positively to better assess lung damage severity in COVID-19 patients and improve early decision-making on the most suitable care and treatment to administer. Although the use of DL methods for CT-image analysis and pattern recognition is expanding, ML methods should not be discounted as shown by the results of the DT and RF models with the AS data set described.