Application of machine learning methods in predicting schizophrenia and bipolar disorders: A systematic review
Abstract
Background and Aim
Schizophrenia and bipolar disorder (BD) are critical and high-risk inherited mental disorders with debilitating symptoms. Worldwide, 3% of the population suffers from these disorders. The mortality rate of these patients is higher compared to other people. Current procedures cannot effectively diagnose these disorders because it takes an average of 10 years from the onset of the first symptoms to the definitive diagnosis of the disease. Machine learning (ML) techniques are used to meet this need. This study aimed to summarize information on the use of ML techniques for predicting schizophrenia and BD to help early and timely diagnosis of the disease.
Methods
A systematic literature search included articles published until January 19, 2020 in 3 databases. Two reviewers independently assessed original papers to determine eligibility for inclusion in this review. PRISMA guidelines were followed to conduct the study, and the Prediction Model Risk of Bias Assessment Tool (PROBAST) to assess included papers.
Results
In this review, 1243 papers were retrieved through database searches, of which 15 papers were included based on full-text assessment. ML techniques were used to predict schizophrenia and BDs. The main algorithms applied were support vector machine (SVM) (10 studies), random forests (RF) (5 studies), and gradient boosting (GB) (3 studies). Input and output characteristics were very diverse and have been kept to enable future research. RFs algorithms demonstrated significantly higher accuracy and sensitivity than SVM and GB. GB demonstrated significantly higher specificity than SVM and RF. We found no significant difference between RF and SVM in terms of specificity.
Conclusion
ML can precisely predict results and assist in making clinical decisions-concerning schizophrenia and BD. RF often performed better than other algorithms in supervised learning tasks. This study identified gaps in the literature and opportunities for future psychological ML research.
1 INTRODUCTION
Schizophrenia and bipolar disorder (BD) are critical and high-risk inherited mental disorders that have debilitating symptoms.1 These disorders are among the severe psychiatric diseases that have many overlaps and similarities with each other and affect the patient's behavior in the family and society. According to the World Health Organization, these disorders are among the top 10 causes of disability worldwide.2 Schizophrenia and BD affect 3% of the world's population.3, 4 Patients with schizophrenia and BD have a higher mortality rate than the general population.5 One of the prominent causes of death in these patients is suicide. In Danish registers, the rate of suicide is reported as 7.8% in men and 4.9% in women with BD.6
There are Five key features that define schizophrenia and BD and other psychotic disorders. These include delusions, hallucinations, disorganized thinking (inferred from speech), grossly disorganized or abnormal motor behavior, and negative symptoms.7 As compared with other disorders, schizophrenia and BD specifically, the presence or absence of specific psychotic symptoms identified as first-rank symptoms (auditory hallucinations; thought withdrawal, insertion, or interruption; thought broadcasting; somatic hallucinations; delusional perception; feelings or actions controlled by external agents) may be particularly helpful for making the diagnosis.8-10
Many patients with schizophrenia and BD experience a long clinical period.7 Symptoms of the disease begin between the ages of 16 and 30. These symptoms fall into three categories: positive (hallucinations, delusions, and mental disorders), negative (lack or absence of facial expressions, feelings of little pleasure, and decreased sense of speech), and cognitive (concentrating and maintaining difficulty).8-10 BD patients experience persistent changes in brain structures, such as enlargement of the third and lateral ventricles of the brain and a decrease in the volume of gray matter in the anterior and middle cerebral cortex, cortical and mesotemporal cortex, and decrease in the posterior abdominal callus volume.11 The economic costs associated with the disease vary from $94 million to $102 billion each year.12 Therefore, this disease imposes a heavy financial burden on the patients, their families, and society.13 Predicting the disease can go a long way in preventing it and controlling its costs. Since it takes an average of 10 years from the onset of the first symptoms to the definitive diagnosis of the disease,14 current approaches cannot effectively diagnose these diseases. Machine learning (ML) techniques are proposed as an effective tool to meet this need.
ML is a domain of artificial intelligence that allows computer algorithms to learn patterns by studying data directly without being explicitly programmed.15 Artificial intelligence using ML is entering the realm of medicine at an increasing pace and has been tested in various clinical applications ranging from diagnosis to outcome prediction.16 The utilization of ML techniques has many advantages, such as recognizing diseases, reducing physician decision-making errors, reducing healthcare costs, and improving the performance of healthcare providers.17
Various models have contributed significantly to the health domain, from rule-based systems to advanced ML models (deep learning). These models have been used in prediction, diagnosis, and treatment in healthcare, such as predicting survival in breast cancer,18 diagnosis and prognosis of COVID-19,19, 20 level of lung cancer,21 etc. ML techniques are also used to diagnose, classify, and predict schizophrenia and BD.22-26 Several ML methods have been used to predict the negative symptoms of schizophrenia based on speech signals22, 27 and to predict the recurrence of schizophrenia.24 Also, many studies have been conducted on the extraction of various features of computed tomography scans and magnetic resonance imaging (MRI) images in the diagnosis and prevention of schizophrenia and BD.28-30
Several algorithms, such as random forests (RF), support vector machine (SVM), and gradient boosting (GB), have been frequently used in this area. The RF method is fast, adaptable, and reliable for mining high-dimensional data. As the name suggests, an ensemble of many decision trees makes up a RF. The RF produces a classification for each tree, and the class voted on the most becomes the prediction.31 SVMs are linear models for classification and regression problems. Several practical problems can be solved through this technique, including linear and nonlinear problems. This algorithm generates a line or a hyperplane to classify the data into classes.32 A GB algorithm trains many models (typically decision trees) in sequential and additive order. The purpose of boosting is to transform weak classifiers into strong classifiers. Each new model in GB is designed to minimize prediction error as much as possible.33
This study aimed to conduct a systematic review of ML algorithms for predicting schizophrenia and BD to help early and timely diagnosis of the diseases to improve patients’ health.
2 MATERIALS AND METHODS
2.1 Information source and search
A systematic search was conducted in PubMed, Web of Science, and Scopus for relevant studies published before January 18, 2020. PRISMA guidelines were followed to conduct this study.34 Two groups of keywords related to: (A) ML and (B) schizophrenia and BD were used to search these databases. The keywords used to identify relevant papers are shown in Appendix 1.
2.2 Inclusion and exclusion criteria
All studies applying ML techniques for predicting schizophrenia and BDs were considered. We included original studies. The search was restricted to English-language publications. Editorials, commentaries, letters, books, presentations, and conference papers were excluded. All types of review studies were also excluded to prevent duplication in data collection.
2.3 Study selection
The selection process was initiated by removing duplicated papers. Then, two authors (MM, MM) independently reviewed the titles and abstracts of all identified studies. The same authors independently reviewed the relevant papers (MM, MM). The disagreements were resolved through discussion and, if required, referred to a third researcher (KB). The reasons for the exclusion of each study were documented during the screening process of the papers. Rayyan QCRI systematic review, a free web and mobile application platform, was used for paper screening.35
We additionally evaluated reference lists of relevant papers for relevant publications.
2.4 Data extraction and synthesis
We developed an Excel data-extraction form to extract specific details of each paper (Appendix 2). Two reviewers (MM, MM) Completed the form. This form consisted of study's location, data utilized, sample size, ML model, accuracy, sensitivity, specificity, area under the receiver operating characteristic curves (AUC), and precision (Table 1). A more detailed table of the PROBAST results is shown in Appendix 3.
Disorder | Data utilized | What is predicted? | Sample size | Machine learning model | Accuracy (%) | Sensitivity (%) | Specificity (%) | AUC (%) | Precision (%) | Risk of bias | First author | Year |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Schizophrenia | EEG | Having Schizophrenia | 30 | SVM | 78.05 | NR | NR | NR | NR | High | Taylor et al.37 | 2017 |
GPC | 80.49 | NR | NR | 87 | NR | |||||||
Schizophrenia | sMRI, fMRI | Cognitive functions | 324 | Meta-analytic cognitive priors | 70.9 (For mental domains) | NR | NR | NR | NR | High | M. Karrer et al.43 | 2019 |
70.8 (For experimental tasks) | ||||||||||||
Schizophrenia | Symptomatic and MRI | Symptom severity | 167 | LR | NR | NR | NR | 81 | NR | Low | Talpalaru et al.44 | 2019 |
SVM | NR | NR | NR | 78 | NR | |||||||
RF | NR | NR | NR | 75 | NR | |||||||
Schizophrenia | MRI | Neurotic, and psychotic symptoms | 34 | SVM | 94 | 76 | 100 | NR | NR | Low | Zarogianni et al.38 | 2016 |
Schizophrenia | Biobank | Having Schizophrenia | 1606: | RBF | NR | NR | NR | NR | NR | High | Bracher-Smith et al.41 | 2019 |
803 SCZ | SVM | NR | NR | NR | NR | NR | ||||||
803 HC | RF | NR | NR | NR | NR | NR | ||||||
GBM | NR | NR | NR | NR | NR | |||||||
NN | NR | NR | NR | NR | NR | |||||||
LR | NR | NR | NR | NR | NR | |||||||
Schizophrenia | Data | Psychotic relapse | 864: | CART | 63.8 | 71.0 | 44.8 | NR | NR | Low | Fond et al.24 | 2019 |
549 baseline | ||||||||||||
315 2 years | ||||||||||||
Schizophrenia | Whole exome sequencing of genotypes and phenotypes | Having Schizophrenia | 5090: | XGBoost | 85.7 | 84.9 | 86.6 | NR | 86.9 | Low | Trakadis et al.42 | 2018 |
2545 SCZ | L1.Logistic | 74.6 | 72.0 | 77.3 | NR | 76.0 | ||||||
2545 HC | SVM | 70.7 | 70.8 | 70.6 | NR | 70.5 | ||||||
RF | 81.7 | 82.0 | 81.3 | NR | 81.1 | |||||||
Schizophrenia | MRI, fMRI | Having Schizophrenia | 211: | Ridge | 87 | NR | NR | NR | NR | Unclear | Salvador et al.29 | 2019 |
96 SCZ | Lasso | 80 | NR | NR | NR | NR | ||||||
115 HC | RF | NR | NR | NR | NR | NR | ||||||
GB | NR | NR | NR | NR | NR | |||||||
Schizophrenia | fMRI | Having Schizophrenia | 86 | SVM | 94.12 | 1 | 89.47 | 94.73 | NR | Low | Nimkar et al.26 | 2018 |
MRI | C5.0 DT | 91.18 | 1 | 84.21 | 91 | NR | ||||||
RF | 91.18 | 1 | 84.21 | 92.1 | NR | |||||||
K-NN | 79.41 | 1 | 63.16 | 81.579 | NR | |||||||
LDA | 79.41 | 80 | 78.95 | 79.47 | NR | |||||||
GP | 92 | 1 | 864 | 92.4 | NR | |||||||
NB | 85.29 | 86.67 | 84.21 | 85.43 | NR | |||||||
Schizophrenia | fMRI | High-symptomatic patients vs. non/low-symptomatic | 174: | EM | 86.9 | 79.8 | 93.1 | NR | 91.9 | Low | Kalmady et al.47 | 2019 |
81 SCZ | ||||||||||||
93 HC | ||||||||||||
Schizophrenia | sMRI | Having Schizophrenia | 606: | SVM | 69 | 69 | 68 | 74 | NR | Low | de Pierrefeu et al.40 | 2018 |
276 SCZ | Enet | 71 | 73 | 68 | 76 | |||||||
330 HC | Enet-TV | 68 | 68 | 68 | 74 | |||||||
Schizophrenia | fMRI | Having Schizophrenia | 144: | NR | NR | NR | NR | NR | NR | High | Silva et al.74 | 2014 |
sMRI | 69 SCZ | |||||||||||
75 HC | ||||||||||||
Bipolar | MRI | Symptom severity | 94: | SVR | NR | NR | NR | NR | NR | High | Sartori et al.30 | 2018 |
35 BD | ||||||||||||
59 HC | ||||||||||||
Schizophrenia or bipolar disorder | MRI | Predict risk of developing Schizophrenia or bipolar disorder in offsprings | 185: | SVM | 77 | NR | NR | NR | NR | High | Hillegers et al.39 | 2018 |
50 SCZ | ||||||||||||
82 BD | ||||||||||||
53 HC | ||||||||||||
Schizophrenia Bipolar | MRI | Having Schizophrenia | 606 | SVM | 69 | 69 | 68 | 74 | NR | Low | Pierrefeu et al.46 | 2018 |
ElasticNet | 71 | 73 | 68 | 76 | NR | |||||||
GraphNet | 70 | 69 | 71 | 75 | NR | |||||||
TV-Enet | 68 | 68 | 68 | 74 | NR |
- Note: “+” indicates high risk of bias/low concern regarding applicability; “−” indicates low risk of bias/high concern regarding applicability; “?” indicates unclear risk of bias/unclear concern regarding applicability; NR = not reported.
- Abbreviations: AUC, area under the receiver operating characteristic curves; BD, bipolar disorder; CART, classification and regression tree; C5.0 DT, C5.0 Decision Tree; EEG, electroencephalography; EM, ensemble model; EN, elastic net; Enet, elasticnet-total variation; Enet-TV, elasticnet-total variation; GB, gradient boosting; GBM, gradient boosting machines; GP, Gaussian process; GPC, Gaussian processes classifiers; HC, healthy controls; K-NN, K-nearest neighbor; LASS, least absolute shrinkage and selection operator; LDA, linear discriminant analysis; LR, multivariate logistic regression; L1.Logistic, Lasso regularized (L1) logistic regression; MRI, magnetic resonance imaging; NB, Naïve Bayes; NN, neural networks; ONR, one nonlinear regression; RBF, linear and radial basis function; RF, random forests; SCZ, schizophrenia; SVM, support vector machine; SVR, support vector regression; Xgboost, extreme gradient boosting.
2.5 Risk of bias (ROB) assessment
To assess the ROB, we used the Prediction Model Risk of Bias Assessment Tool (PROBAST).36 It is a tool for assessing the ROB and the applicability of diagnostic and prognostic prediction model studies. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers to use them to assess the ROB and applicability concerns.
3 RESULTS
We retrieved 1243 papers through database searches. After title and abstract screening 144 papers were identified for full-text assessment. Full-text assessment excluded 129 studies due to various reasons. Fifteen papers met the inclusion criteria (Figure 1). An analysis of the algorithms applied, the inputs they were trained on, the outputs they were trained to predict, and their relative performance statistics are presented. An average number of 185 patients were used in each study (mean = 681.4, SD = 1289.65). The median number of ML algorithms employed in each study was two (mean = 2.67, SD = 1.99). Most of the studies (n = 13) applied ML algorithms only to schizophrenia disorder, and one study applied ML algorithms to both schizophrenia and BDs. All the included studies applied ML algorithms to predict the symptoms of these disorders. Data utilizes in prediction of 10 studies was based on MRI. In eight studies, ML algorithms predicted schizophrenia disorder while in three studies these algorithms predicted symptoms severity.

3.1 Publications and algorithms applied in schizophrenia and BD over time
Over the past decade, many publications applying ML to schizophrenia and BD decision support have increased rapidly. The top two most frequently applied algorithms were SVM and RF. As shown in Table 1, most algorithms have been recently applied to schizophrenia and bipolar.
3.2 Summary of literature included, and algorithms applied
Most studies reported outcomes for more than one ML technique.12, 38, 45 The most frequently developed models used the variants of ML models, including SVM in 10 studies in references,24, 30, 37-42, 44, 46 RF in 5 studies,24, 29, 41, 42, 44 GB in 3 studies29, 41, 42 with the three different types of gradient boosting (GB) machines, Xgboost and GB, logistic regression in 3 studies41, 42, 44 and its extended L1.Logistic, decision tree in 2 studies24, 26 with two types of CART and C5.0 DT, and Gaussian process in 2 studies.26, 37 Several ML techniques each appeared in the one of the studies: Meta-analytic cognitive prior,43 ElasticNet,41 GraphNet,41 TV-Enet,41 linear and radial basis function,42 neural network,42 Ridge,29 Lasso,29 K-NN,26 linear discriminant analysis,26 Naïve Bayes,26 and ensemble model.47
3.3 Predictive model performance evaluation statistics
The performances of the models were evaluated in varied ways. Accuracy and sensitivity (recall)-specificity were the most frequently reported performance statistics (in 10 references24, 26, 29, 37, 39, 40, 42, 43, 46, 47 and 7 of studies,24, 26, 38, 40, 42, 46, 47 respectively). Five of the studies26, 37, 40, 44, 46 reported AUC, while Precision was reported by 2 studies.42, 47
3.4 Comparing algorithm performance
A comparison of the performance of the top three most frequently applied algorithms (SVM, RF, and GB) is shown in Table 2. Insufficient performance data were reported in this corpus to include other algorithms. We compared algorithms based on their accuracy, sensitivity, specificity, AUC, and Precision (Table 2). Multivariate logistic regression, Neural Network, and SVM differed significantly in their sensitivity performance. Based on this table, RF was significantly more accurate than SVM. The accuracy performance of RF and GB was not significantly different. The Specificity of the three algorithms was not significantly different. It showed that RF demonstrated significantly higher sensitivity than SVM.
Performance metrics, mean (SD; n) | |||||
---|---|---|---|---|---|
Accuracy (%) | Sensitivity (%) | Specificity (%) | AUC (%) | Precision (%) | |
Support vector machine (SVM) | 80.48 (11.08, 6) | 78.95 (14.34, 4) | 82.02 (15.34, 4) | 82.24 (11, 3) | 70.5 (–, 1) |
Random forests (RF) | 86.44 (6.7, 2) | 91 (12.73, 2) | 82.75 (2.06, 2) | 83.55 (12.09, 2) | 81.1 (–, 1) |
Gradient boosting (GB) | 85.7 (–, 1) | 84.9 (–, 1) | 86.6 (–, 1) | – | 86.9 (–, 1) |
- Abbreviation: AUC, area under the receiver operating characteristic curves.
3.5 Classification performance
The most commonly reported performance metrics for schizophrenia and BD were sensitivity (recall), specificity, and accuracy.48, 49 Multiple studies also presented positive predictive value (precision)42, 47 and error matrix outcomes as the AUC.26, 37, 40, 44, 46 However, there was no overall consistency as to which specific measures were reported.
3.6 Predictive model validation
In this study, 10 of the 13 studies provided details of a validation process for the applied models.24, 26, 29, 30, 37-43, 46, 47 Different forms of internal cross-validation and holdout datasets were most commonly applied. K-fold cross-validation and leave-one-out cross-validation frameworks were used to split data into training, validation, testing sets, or optimizing model parameters.50
3.7 Risk of bias
We critically reviewed studies for ROB using PROBAST.36 Our analysis revealed that except for six, all studies had some bias due to a low number of participants, lack of external validation, and failure to meet the study's goal (Table 1).
4 DISCUSSION
In this study, the applications of ML to support clinical decision-making in schizophrenia and BD were reviewed. Results suggested that the use of ML in schizophrenia and BD is rapidly growing.51, 52 There is substantial room for further applications of ML technologies to schizophrenia and BD data. Many ML applications and modeling methods have been used based on the findings. A wide variety of successfully predicted outcomes could facilitate decision-making. The number of ML publications on schizophrenia and BD has rapidly increased over the past few years.53-55
According to this study, most schizophrenia and BD ML publications were in the domain of MRI and fMRI data. Few ML studies have focused on EEG data. The applied modeling approaches were substantially heterogeneous. Variation in the studies was related to algorithms applied, input and output variables used, and methods for assessing predictive model performance.
In this paper, SVM and RF were the most commonly applied algorithms, compared to other studies, applying SVMS56-59 and RF.60, 61 Consistent with a previous study,62 RF frequently outperformed most other algorithms on supervised learning tasks. Since both SVM and RF are discriminative, they can handle large amounts of data, and capture nonlinear relationships across input features.44 They were selected to predict disease outcomes, often demographics, clinical history, and investigation-related features were used.49 AUC, sensitivity, accuracy, specificity, and precision performance metrics were the most commonly reported.49 The accuracy of RF was significantly higher than that of SVM, and compared to SVM, the precision of GB was significantly higher. To facilitate future modeling, the input and output variables were synthesized. Gaps in knowledge and opportunities concerning the development of additional clinical decision support systems to improve the care of schizophrenia and BD patients were highlighted to clinicians and data scientists. To our knowledge, this systematic review is the first attempt to characterize the deployment of ML methods in schizophrenia and BD. We identified two different topic clusters, and keywords for each of the schizophrenia and BD subdomains by applying ML technologies. The use of these topics and keywords in the systematic review helped to deeply understand the past foci of the relevant literature and determine the current state of the knowledge. This understanding helps to identify gaps to be filled out by future researches. ML provides a series of technologies, from linear regression to RF, that can effectively predict outcomes for improving clinical decision-making in schizophrenia and BD. Based on the results, wherever large, labeled datasets were available, RF has been the top-performing algorithm in the ML methods due to its higher accuracy than multivariate logistic regression.62
RF is faster to train and can readily handle larger numbers of predictors. It employs fewer parameters71 and does not need cross-validation (it generates an internal unbiased estimate of the generalization error (test error) as the forest building progresses).
5 LIMITATIONS AND FUTURE RESEARCH
It is more likely to publish positive and significant findings.72, 73 Many of the included studies evaluated the performance of multiple algorithms and reported the inferior performance of some. However, since we could not find studies reporting failed ML modeling activities, publication bias and selective outcome reporting may have influenced our results. Few studies have comprehensively reported predictive model performance statistics. Hence, we had relatively small sample sizes for our statistical analysis of algorithm performance. Larger sample sizes could have demonstrated additional significant performance differences between algorithms. More research is suggested to examine AUC, accuracy, sensitivity, and specificity to facilitate robust model performance evaluation and compare studies. Future studies can contribute to the convergence of ML with schizophrenia and BDs. Future studies can use similar predictors and outcomes on bigger databases containing the information of patients with schizophrenia and BDs. This may help predict the same or additional outcomes that can be used to deploy systems based on these models. The number of software to predict and support schizophrenia and BDs decision-making is low. Further models have required these two disorders. Despite the richness of electronic health record datasets, they are not utilized sufficiently and are suitable for conducting ML studies. SVMs and transfer learning can utilize video data during real-time patient examination, diagnosis and prognosis, or intraoperative anatomical identification. SVMs can predict the postoperative status of patients on an hourly basis to provide safer proactive management of patient status.
6 CONCLUSION
ML effectively supports making clinical decisions before, during, and after the onset of schizophrenia and BDs. Due to the evident heterogeneity of apply ML in psychology and the popularity of distinctive ML studies, there is a substantial potential for future similar studies. Because of the accurate prediction of various operative outcomes, RF seems more effective than other algorithms in schizophrenia and BDs. Deploying ML by designing sound clinical decision support systems can lower complications and increase the quality and safety of health services provided to schizophrenia and BD patients.
AUTHOR CONTRIBUTIONS
Mahdieh Montazeri: project administration. Mitra Montazeri: formal analysis; writing—review & editing. Kambiz Bahaadinbeigy: conceptualization. Mohadeseh Montazeri: data curation; writing—original draft. Ali Afraz: conceptualization; writing—original draft.
CONFLICTS OF INTEREST
The authors declare no conflicts of interest.
TRANSPARENCY STATEMENT
The lead author Mitra Montazeri affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
APPENDIX 1: KEYWORDS USED TO IDENTIFY RELEVANT PAPERS
PubMed
Web of Science
Scopus
We included original studies. The search was restricted to English-language publications. Editorials, commentaries, letters, books, presentations, and conference papers were excluded. All types of review studies were also excluded to prevent duplication in data collection. We had no filter on the date.
APPENDIX 2: DATA EXTRACTION FORM
A | B | C | D | E | F | G | H | I | J | K | L | M | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | First author | Year | Data utilized | What is predicted? | Disorder | Sample size | Machine learning model | Accuracy (%) | Sensitivity (%) | Specificity (%) | AUC (%) | Precision (%) | Risk of bias |
2 | Taylor et al.34 | 2017 | EEG | having Schizophrenia | Schizophrenia | 30 | SVM | 78.05 | NR | NR | NR | NR | High |
APPENDIX 3: MORE DETAILED TABLE OF THE PROBAST RESULTS
Study | Risk of bias | ||||
---|---|---|---|---|---|
Participants | Predictors | Outcome | Analysis | Overall | |
Taylor et al.37 | Unclear | Low | Low | High | High |
M. Karrer et al.43 | Low | Low | Low | High | High |
Talpalaru et al.44 | Low | Low | Low | Low | Low |
Zarogianni et al.38 | Low | Low | Low | Low | Low |
Hillegers et al.39 | High | Unclear | Unclear | High | High |
Pierrefeu et al.46 | Low | Low | Low | Low | Low |
Bracher-Smith et al.41 | High | Unclear | Unclear | High | High |
Fond et al.24 | Low | Low | Low | Low | Low |
Trakadis et al.42 | Low | Low | Low | Low | Low |
Salvador et al.29 | Unclear | Low | Low | Unclear | Unclear |
Nimkar et al.26 | Low | Low | Low | Low | Low |
Kalmady et al.47 | Low | Low | Low | Low | Low |
Sartori et al.30 | Unclear | Low | Low | High | High |
de Pierrefeu et al.40 | Low | Low | Low | Low | Low |
Silva et al.74 | Low | Low | Low | High | High |
Open Research
DATA AVAILABILITY STATEMENT
The authors confirm that the data supporting the findings of this study are available within the article or its supplementary materials.