1 INTRODUCTION

Volatile organic and non-organic compounds (VOC) such as alcohols, aldehydes, and ketones in exhaled breath provide an insight into the metabolic processes taking place in the human body. VOC are emitted by all cells, and differ in their specific composition depending on the type of cell and its current metabolism.¹ Thus, different physiological states can affect VOC composition, reflecting food intake, lifestyle, or exercise.^{2, 3} VOC analysis is a growing field with many possible applications, including diagnostic approaches in medicine.⁴ For example, VOC analysis allowed for differentiation between bacteria species in vitro⁵ and in vivo,⁶ and for detection of influenza virus in swine.⁷ Even different influenza virus subtypes could be discriminated in an infected cell line based on the emitted VOC signature.⁸ In addition, VOC have been discussed to potentially identify patients with metabolic diseases such as diabetes,⁹ with malignant diseases such as neck, bladder, colon, or lung cancer,^9-12 and with lung diseases such as chronic obstructive pulmonary disease or asthma.^{13, 14} With regard to the severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) pandemic, primary studies show that SARS-CoV-2 pneumonia patients could be distinguished from non-infected subjects using VOC-based breath analysis.^15-17

Thus far, mass spectrometry is the method of choice for VOC analysis.¹⁶ However, high acquisition and running costs as well as the need for laboratory support show its limitations. In contrast, metal oxide sensor-based electronic nose (E-Nose) technology provides a cheaper, portable, and easy to use method for breath analysis, especially in point-of-care (POC) environments. Unlike mass spectrometry, E-Nose technology does not identify single compounds, but detects VOC-induced changes of electrical conductivity in a sensor set of metal oxide semiconductors. These changes of conductivity generate a signal pattern for each sample, which is then analyzed by machine learning (ML) algorithms. Various setups of electronic noses with different types of sensor sets have been applied for breath research, for example, sensor arrays of conducting polymers, nanomaterial-based or quartz microbalance-based sensors, colorimetric sensors, and metal oxide sensors. Electrical properties of the metal oxide semi-conductors can be optimized for different VOC detection by adjusting sensitivity through altering sensor film thickness or loading of the metal oxide surface with noble metal.¹⁸ E-Nose technology applied in this study represents a technology transfer from space to health. Previous E-Nose experiments have been conducted to successfully determine bacterial surface contamination on the International Space Station,¹⁹ and the technique has now been adapted and optimized for its use in a medical setting.

With SARS-CoV-2 being the predominant cause for pneumonia during the pandemic, this study aimed to establish breath analysis by E-Nose technology as a diagnostic tool in SARS-CoV-2 infection. The particularity of the present study is that it was conducted as a so-called multi-analyst experiment.^20-23 More specifically, after an optimized pre-analytical sampling method was established, the resulting data was further analysed by three independent teams of data analysts who applied different ML approaches, without consulting each other. This still uncommon multi-analyst approach was inspired by Wagenmakers et al., who state that “one statistical analysis must not rule them all”,. and “a single analysis hides an iceberg of uncertainty” while “multi-team analysis can reveal it”.²³ It is established that the high multiplicity of possible analysis strategies resulting from these uncertainties, if combined with selective reporting, can substantially contribute to the so-called replication crisis in science,²⁴ including in the context of ML. The rapidly growing field of artificial intelligence is unfortunately not immune against irreproducibility issues.²⁵ Multi-analyst approaches are known to strengthen the robustness of results and conclusions obtained from analysis of datasets²⁶ and to show that analytical flexibility can have substantial effects on scientific conclusions.²⁰ Thus, results obtained by the three teams, featuring a number of common aspects but also differences, allow us to formulate more reliable results than a single analysis would do. At the same time, it illustrates the huge uncertainty related to analytical choices and its impact on the results—a still largely unexplored issue in the field of artificial intelligence application in medicine.

2 RESULTS

2.3 Team A

2.3.1 Experiment I: time-independent classifier exploration and variable importance

Experiment I by team A does not explicitly make use of the time structured data, resulting in a typical p > > n (“high—p, low—n”) problem, meaning that the model receives more predictors per subjects than subjects themselves. Figure 1A,B depicts the averaged receiver operating characteristic (ROC) curves for the RF and Glmnet classifier, respectively. Note that the displayed naive confidence intervals might underestimate the true underlying uncertainty due to correlation of the cross-validation iterations, and thus only reflect the order of magnitude of the variability. The RF classifier has an average misclassification error (ME) of 0.13 (AUC: 0.94). The Glmnet classifier performs slightly worse with an average ME of 0.15 (AUC: 0.90). Figure 1C shows the variable importance based on the mean decrease in impurity for the RF classifier. The features with the highest predictive power are mainly features from the upper respiratory tract within the first 10 s of the measurement. This makes sense as the highest variability of the raw measurements can be seen within this period.

Figure 1D depicts the regression coefficients of the Glmnet classifier. The results are not as clear-cut as for classifier RF. Features with the highest impact on the prediction results do not only stem from different sensors but also different time points of measurements. The Glmnet classifier focuses more on later time points of each sensor.

2.3.2 Experiment II: time-dependent feature extraction and variable importance

The second experiment by team A makes use of feature extraction techniques, also termed “dimensionality reduction.” Figure 2A,B depicts the averaged ROC curves for the RF and Glmnet classifier, respectively. The RF classifier has an average misclassification rate of 0.19 (AUC: 0.91). The Glmnet classifier performs better with an average misclassification rate of 0.13 (AUC: 0.92). Figure 2C illustrates the variable importance based on the mean decrease in impurity for the RF classifier. Features with the highest predictive power are features from both upper and lower respiratory tracts. Especially sensor 9 seems to have a high influence on the predictive performance of the model. Regression coefficients of the Glmnet classifier are depicted in Figure 2D, where results again differ from the RF classifier. The Glmnet classifier focuses on both the standard deviations of the upper and lower respiratory tracts for most sensors.

2.4 Team B

2.4.1 Experiments III and IV: feature extraction and hyperparameter tuning on all sensors

These experiments used LightGBM's gradient boosting model and the approach described above. Without hyperparameter optimization, the learner achieved an overall performance of ME = 0.143 (Figure 3A). Combined with Optuna's hyperparamter optimization the learner improved to ME = 0.079 (Figure 3B). F1 results and confusion matrices are provided in Figure S1. Sensor 9 showed the highest importance for classification (Figure 3C,D).

2.4.2 Experiments V and VI: feature extraction and hyperparameter tuning on sensor 9

The importance of sensor 9 led to an additional analysis of this specific sensor alone, where only data generated by sensor 9 was used. LightGBM's gradient boosting model achieved a ME of 0.104 (Figure 4A). Figure 4C depicts time dependent importance of sensor 9 measurements, showing that the first seconds have the highest impact on classification. Thus, subsequent Optuna's hyperparameter optimization was performed on filtered raw data, where only the first 10 s of sensor 9 measurements were used. The train test split was set to 9:1. This resulted in a performance of ME = 0.065, outperforming all previous models (Figure 4B,D). Figure S2 shows F1 results and confusion matrices, respectively.

2.5 Team C

During preliminary testing for approach C, similar results were obtained for three out of four classifiers (ET, DT, RF): for both upper and lower respiratory tract, sensor 9 showed the best classification quality (Figure S3). Variance of classification with regard to hyperparameters was also smallest for this sensor. Classifier RF showed the best results of all classifiers, and sensor 9 measurements from lower respiratory tract samples provided the best results (Figure S3). Thus, it was decided to continue the investigation with classifier RF.

Establishing the ground truth for classifier RF showed not all subjects were identified correctly, although the score of false positives in both cohorts was much lower than during the real leave-one-out test (Table S2 and Figure S3). Sensor 9 again showed superior performance regarding hyperparameter sensitivity (Figure S4). Thus, the following steps were performed with classifier RF and sensor 9 only.

2.5.1 Experiment VII: influence of prevalence and training data on classifier RF and sensor 9

Using a RF classifier with hyperparameter tuning on sensor 9 measurements from lower respiratory tract samples resulted in an overall performance of ME = 0.062. A superior hyperparameter combination could be identified for classifier RF and sensor 9. Table 2 shows the best values for sensitivity and specificity achieved by the classifier.

TABLE 2. Experiment VII: sensitivity P(+|C) and specificity P(‒|nC) for the best hyperparameter combination with classifier random forest (RF) for upper (A) and lower (B) respiratory tract.

Sensitivity upper respiratory tract P(+|C) | (A)	Sensitivity lower respiratory tract P(+|C) | (B)	Specificity upper respiratory tract P(‒|nC) | (A)	Specificity lower respiratory tract P(‒|nC) | (B)
0.94915	0.96610	0.89552	0.91045

• Note: Values for the best hyperparameter combination for classifier RF and sensor 9.

Precision and negative predictive value depend on the probability of encountering SARS-CoV-2 in a test population (prevalence, P(C)). Sensitivity and specificity of the best classifier were determined for values of P(C) = 50%, as there were 63 subjects for each label. P(C) is necessarily variable in the real world. Figure 5A shows how precision and negative predictive value depend on P(C) in the group to be examined.

Table 3 depicts the true/false positive/negative rates as a function of the amount of training data used. At 25%, only 10 out of 63 non-infected subjects were correctly identified as non-infected, all others, including all SARS-CoV-2 subjects, were classified SARS-CoV-2. This poor classification leads to a constant value of “1” for the negative predictive value because all subjects that were identified as uninfected are uninfected, but only 10 out of 63 were correctly classified as uninfected. For larger amounts of training data, precision and negative predictive values increased with increasing amount of training data (Figure 5B,C).

TABLE 3. Experiment VII: true/false positive/negative rates versus amount of training data.

Amount of training data (%)	P(+|C)	P(+|nC)	P(‒|C)	P(‒|nC)
100	0.9047619	0.03174603	0.0952381	0.96825397
75	0.7460317	0.03174603	0.25396825	0.96825397
50	0.5714286	0.03174603	0.42857143	0.96825397
25	1	0.84126984	0	0.15873016

• Note: True positive P(+|C), false positive P(‒|C), true negative P(‒|nC), and false negative P(+|nC) rates are depicted as a function of the amount of training data used. At 25%, only 10 out of 63 non-infected subjects were correctly identified as non-infected, all others, including all severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) subjects, were classified as SARS-CoV-2.

2.5.2 Experimentation with deep learning models

In the multilayer perceptron model, sensors 2 and 9 performed best although the score was worse than with the previous classifiers and the variation of hyperparameters was larger (Figure S5A,B). Therefore, this process was repeated with slightly adapted hyperparameter ranges and 50 variations for sensors 2 and 9 only (Figure S5C,D). In comparison to the more classical ML approach, the multilayer perceptron could not outperform the RF classifier on sensor 9 measurements from lower respiratory tract samples.

3 DISCUSSION

In this study, a rapid, portable, and cheap breath gas analysis protocol was established. Exhaled VOC were measured by an array of ten metal oxide semiconductor sensors and changes in conductivity were recorded.

Raw data were subjected to three independent ML approaches, including RF and gradient boosting tree learners. The multi-analyst experiment shows that E-Nose technology in combination with the presented novel sampling method allows for successful identification between SARS-CoV-2-positive patients and uninfected volunteers with MEs ranging from 0.06 to 0.19. Sensitivity and specificity reached values of about 90% or higher for all three independent teams, underlining the robustness of the data generated.

The teams’ approaches already showed differences in their data pre-processing strategies. While team A analyzed the data both without taking into account time dependencies between features or by taking them into account through a feature extraction procedure, teams B and C did not use feature extraction but different approaches to reduce the number of features manually before training. It is interesting that even if simple variance and min/max measures are extracted from the original time series (team A) or if raw data are reduced drastically to the first 10 s of the measurement and/or only one sensor (team B), the differences in performances are small. Overall, it can be concluded that despite these major differences between the three analytical approaches and their rationales, all results were in a decent range in terms of ME (0.06–0.19). This indicates that this study was successful in demonstrating predictability of the underlying system.

The most substantial difference between the three teams’ results lies in the interpretation of the contributions of the features/sensors to the prediction models’ performance, which is not as clear-cut. All three teams identify sensor 9 and the first seconds of the time series measurement as most informative. However, other model interpretations are conflicting and conclusions cannot be easily drawn without more domain knowledge on the respective sensors. This can be observed in particular if we compare the results of the model-agnostic interpretation methods to the natively interpretable regression coefficients obtained by team A, that show a completely different picture of the importance of the different features.

Highly accurate testing methods for SARS-CoV-2 have been the focus of several studies. Currently available and widely used antigen tests (Ag-RDT) can reach a sensitivity of up to 81.9% and a specificity of up to 98.9%.^{29, 30} More expensive and less widely available, PCR testing has been shown to be superior to antigen testing with a sensitivity and specificity of up to 92.8% and 97.6%, respectively.³⁰ However, accuracy of PCR results vary. Up to 54% of SARS-CoV-2-infected patients may initially obtain a false-negative RT-PCR result,^31-34 and over 60% of errors occur in the preanalytical phase of any diagnostic process.³² These reports indicate that breath gas analysis by E-Nose technology as demonstrated here may be superior to commercially available antigen tests and comparable to PCR testing.

Many studies focused on the identification of specific VOC possibly serving as biomarkers for SARS-CoV-2 infection.^{15, 35} In contrast, the approach chosen here aimed at allocating patterns of VOC-induced changes in conductivity of a sensor array, generating a breath gas fingerprint of each subject. Since several VOC may bind to one or more sensors, and in the absence of a control cohort suffering from infection caused by viruses other than SARS-CoV-2 or by bacteria, it may be argued that the results are limited in their significance, and that biomarker identification would be the preferred option. However, considering the vast differences in the interindividual metabolism which is dependent on many factors such as body weight, nutrition, and other lifestyle factors such as stress, physical activity, or smoking habits, the quest to identify one specific molecule to reliably distinguish between infected and healthy subjects may be achievable only with extremely high subject numbers, if at all.

Instead, the breath gas analysis conducted here successfully and very rapidly provides information on the infectious state of the individual and could be comparable in its diagnostic relevance to other parameters obtained from blood analyses, for example, C-reactive protein elevation, or clinical parameters such as elevated body temperature. Thus, the non-invasive, inexpensive, and rapid result of E-Nose technology-based breath gas analysis may be used to warrant further, more invasive diagnostic measures to determine the exact underlying pathology.

ML technologies have been critically discussed as being instable, highly dependent of the training data and chosen analysis techniques, and potentially subject to cherry-picking and a resulting over-optimism of the reported results.³⁶ In an effort to counteract these mechanisms and to assess the robustness of results, pseudonymized E-Nose data of this study was made available to three independent teams of data analysts as part of a so-called multi-analyst experiment.

Experiments performed by team A indicated that breath analysis by E-Nose technology as demonstrated here, allows for reliable discrimination between patients with SARS-CoV-2 infection and uninfected subjects. MEs were comparable to those of commercially available rapid antigen tests using oral or nasal swabs.^{32, 37} Analyzing feature importance revealed sensor 9 to be the most important for classification. Applying nested cross-validation techniques ensured an unbiased estimation of the classification performance that would be obtained with independent data, while yielding results that are less impressive at first view.

Experiments of team B yielded an overview of the importance of features and time samples based on decision tree models. Again, sensor 9 had the highest impact on classification, and measurements from the first 10 s were the most informative. Experiments performed by team C again showed that models trained on data from sensor 9 were most effective at discriminating SARS-CoV-2 patients from uninfected subjects. Furthermore, exhaled air from the lower respiratory tract yielded better results than air obtained from the upper respiratory tract. A deep learning model could not outperform the more classical ML algorithms.

Adopting this multi-analyst approach offers the advantage of combining a diverse range of skills, perspectives, and data analysis techniques that surpass what any individual analyst or research team could achieve. Moreover, the setting with several independent teams reduces the pressure on each team and thus also reduces the incentives to generate overly optimistic results through so-called “fishing expeditions”—a term that commonly denotes the approach consisting of performing a large number of analyses until one of them yields a nice result, which is then often the result of advantageous random variations. Finally, a multi-analyst experiment allows both to assess the analytical uncertainty associated with data analyses and, in many cases, including the study presented here, to achieve a consensus regarding the main results, which then can be considered robust.

In summary, the three teams used a variety of learners (decision tree, RF, gradient boosting, deep learning, elastic), data preprocessing, and evaluation procedures, illustrating the flexibility of analytical procedures in ML. They obtained different estimates of performance, which however were all in the range [0.06; 0.19]. Additionally, they all identified sensor 9 as the most informative sensor. This could provide a target for developing commercially attractive diagnostic tools. While previous studies show the potential feasibility to apply E-Nose technology as a diagnostic tool,¹⁷ the present study generated an E-Nose protocol for a quick, inexpensive, and statistically reliable detection of SARS-CoV-2 infection, with results similar to PCR tests and superior to POC antigen testing.^{33, 34, 37, 38}

4 METHODS

4.4 Data collection and format

Raw data consisted of one time series for each metal oxide sensor of the E-Nose, that is, 10 sensors resulting in 10 time series for 60 s for each individual. Measurements were available for both upper and lower respiratory tract samples, resulting in a total of 20 time series per individual over 60 s. A binary label indicated if the individual was SARS-CoV-2 negative or positive. Figure 6 shows a graphical depiction of raw E-Nose data obtained from one representative measurement.

4.5 Data analyses

Raw conductivity data were extracted into transfer files, which were then subjected to analysis by three independent parties. In an effort to provide unbiased results, these three teams did not consult each other, thus allowing different impartial choices of analytic approaches. Team A was from the LMU Institute for Medical Information Processing, Biometry and Epidemiology (IBE) with vast experience in medical data processing. Team B was the Artificial Intelligence Company LANZ GmbH, and team C consisted of space physics experts at Airbus Defence and Space GmbH familiar with E-Nose raw data processing (Figure 7). Experiments were numbered consecutively from I to VII. Details for the analytical approaches of the three teams are provided as Supporting Information. Results were reported with varying figures of merit, which were transformed to MEs for better comparability. The lead team refrained from conducting an analysis to ensure non-biased reporting of independent results. All of the results are reported and none were excluded.

AUTHOR CONTRIBUTIONS

Conceptualisation, data curation, investigation, methodology, supervision, validation, visualisation, writing—original draft, and writing—review and editing: T.W. and A.C. Data curation, formal analysis, investigation, methodology, visualisation, writing—original draft, and writing—review and editing: F.P. Formal analysis, investigation, methodology, supervision, validation, visualisation, writing—original draft, and writing—review and editing: M.M.M. Investigation, methodology, supervision, writing—original draft, and writing—review and editing: A.L.B. Formal analysis, investigation, methodology, validation, visualisation, writing—original draft, and writing—review and editing: W.S., J.H., A.K., and D.L. Conceptualisation, methodology, writing—review and editing: M.F., M.D., M.K., and B.A. Investigation, methodology, and writing—review and editing: L.K. and D.M. All authors read and approved the final manuscript.

CONFLICT OF INTEREST STATEMENT

A.K. and J.H. are employees of Airbus Defense & Space, and authors W.S. and D.L. are part of Lanz GmbH, and have no potential relevant financial or non-financial interests to disclose. The other authors have no conflicts of interest to declare.

ETHICS STATEMENT

Ethical approval was obtained from the LMU ethics committee (project no. 19-778), written informed consent was obtained from each subject, and the study was carried out in accordance with the Declaration of Helsinki.