1 INTRODUCTION

Proper diagnosis of epilepsy and prediction of its course are essential for effective management and patient counseling. Diagnostic and prognostic studies, therefore, aim to identify markers that can support these processes.¹ Estimates of these markers are conventionally reported at the group level and used to infer population-wide conclusions. The emerging field of personalized medicine, however, targets the search for individualized markers—so-called biomarkers²—rather than group-averaged differences.

Biomarkers may have different applications but are generally defined as “characteristics that are measured as indicators of physiological processes, pathogenic processes, or biological responses to an exposure or intervention, including therapeutic interventions”.³ In the epilepsy field, quite some progress has been made in the discovery of both diagnostic and prognostic biomarkers.⁴ Besides epileptiform EEG activity as an important, but not entirely accurate, biomarker used in the diagnosis of epilepsy,^{5, 6} various microRNAs,^7-9 proteins,^{10, 11} metabolites,¹² and immune system components^{13, 14} have been proposed as biomarkers for the diagnosis of seizures and epilepsy subtypes. MRI markers, mainly network connectivity measures^{15, 16} and specific EEG (network) features^17-20 have diagnostic value as well and seem to be able to predict epilepsy severity and refractoriness. Genetic markers, including polygenic risk scores (PRS),²¹ are also receiving growing interest as biomarkers of epileptogenesis and epilepsy risk, diagnosis, and prognosis (for overviews, see: Refs 22, 23).

Despite the large expansion in publications reporting potential biomarkers in the epilepsy field—and beyond—very few biomarkers have yet found their way to clinical practice. Specific disease-related challenges, such as seizures near the time of biomarker sampling and the use of anti-seizure medications (ASMs), might play a role in this, as they can have a direct effect on biomarker levels (e.g., miRNA)²⁴ and thus distort measurements. Another important contributing factor is that statistical significance is still often incorrectly interpreted as the power to identify personal traits, resulting in low discriminative power.²⁵ Group-level significance is based on average rather than individual differences, whereas the performance of a biomarker is defined by both the difference and the variation between individuals. Therefore, new markers should be evaluated on their discriminative performance rather than on statistically significant group differences before labelling them as biomarkers.²⁶

This study aimed to reinforce the idea that caution is needed in identifying and reporting differences in markers at the group level as biomarkers that can discriminate between subjects at the individual level. We do not claim that this idea is new,²⁵ but merely wish to illustrate the importance of this concept for the interpretation and potential usage of epilepsy biomarkers. Therefore, we first assessed the relationship between significant group-level differences and a marker's power to discriminate between individuals in a simulated setting. Secondly, we provide a real data example evaluating a potential biomarker—EEG beta power oscillations—for diagnosing generalized epilepsy in children previously, as published in Epilepsia.²⁷ Thirdly, we evaluated the recent scientific literature on new potential biomarkers in epilepsy.

2 MATERIALS AND METHODS

2.1 Data simulation

Biomarkers may quantify different types of outcomes: binary, continuous, or correlational (Figure 1). The logarithmic (log) transformed odds ratio from binary data, as well as Pearson's correlation coefficient (r) from correlational data, is convertible into the same standardized mean difference (SMD) obtained from continuous data. The SMD, also named Cohen's d, is one of the most commonly used effect size measures, indicating how many standard deviations (SDs) two group means differ.²⁸ Hence, we restricted our simulations to Cohen's d outcome.

We performed two different sets of simulations. The first set was used to evaluate the impact of both sample size and effect size on discriminative power. We generated multiple normally distributed datasets sampled from hypothetical populations, reflecting outcomes in a patient and control group (control mean: 100; standard deviation: 15), with varying sample sizes (25–800) and effect sizes (.25–2.50). Equally sized datasets with similar parameters were generated and used as independent validation data to evaluate the discriminative power. The second set of simulations, used to evaluate the relationship between variability and discriminative power, was based on similarly generated datasets but with a fixed mean of the control and patient group (100 and 115, respectively), a fixed sample size (400 subjects per group), and varying SDs (10–35). The simulation code is available via the Zenodo platform (DOI: 10.5281/zenodo.7095386).²⁹

We evaluated the group-level differences in outcome between the patient and control groups using the Z value. The discriminative power of the simulated biomarkers was evaluated using receiver operating characteristic (ROC) curve analysis³⁰ (Box 1, Figure 2).

BOX 1. Z value and ROC analysis

Evaluation of statistical significance

The Z value can be used to characterize the difference between two groups. With a significance level of .05 and the corresponding 95% confidence interval (CI), Z values smaller than −1.96 and greater than1.96 will yield significant results.³¹

Evaluation of predictive power

The receiver operating characteristic (ROC) curve analysis can be used to evaluate discriminative power.³⁰ An ROC curve is a graphical representation of the true-positive rate (TPR; i.e., sensitivity) against the false-positive rate (FPR; i.e., 1-specificity). The TPR represents the proportion of study subjects correctly classified as patients (TP) out of the total number of patients (TP + FN). Similarly, the FPR is the proportion of subjects incorrectly classified as patients (FP) out of all control subjects (TN + FP). The TPR and FPR can be calculated for every possible threshold value of a biomarker or test. An ROC curve is generated by plotting the TPR and FPR across varying thresholds (Figure 2).³² The area under the curve (AUC) summarizes the area underneath the entire ROC curve across all possible thresholds and thus provides an overall and combined measure of sensitivity and specificity. The AUC ranges between 0 and 1, where .5 indicates that the model does not perform better than chance. The closer the AUC comes to 1.0, the better a biomarker can discriminate between patients and controls.^{30, 32}

3 RESULTS

3.1 Data simulation

3.1.1 Sample size

Sample size has a strong relationship with group-level significance, as an increase in sample size directly increases the Z value. Therefore, large sample sizes give significant differences at the group level, even with a small effect size of Cohen's d .25 (Figure 3A). By contrast, sample size hardly has any effect on discriminative power, expressed as the AUC. In the case of an effect size of Cohen's d .25, the AUC was .55 (95% CI: .51–.58) with 25 subjects per group and increased to .57 (95% CI: .53–.61) with 200 subjects per group (Figure 3B, Table S1).

3.1.2 Effect size

In contrast to the sample size, the effect size strongly affects the AUC. The greater the effect size of a biomarker, the better its discriminative power (Figure 4). A single, normally distributed biomarker requires a Cohen's d of at least 1.25 to reach an AUC of .80 (Figure 4C), considered as the lower limit of good discrimination.^{32, 40} Biomarkers with both a sensitivity and specificity of .8 or 80%—instead of combined in the AUC—require an even greater effect size, namely a Cohen's d of 1.66 (Figure S1). Effect sizes of 1.25 and 1.66 correspond to odds ratios (OR) of 9.65 (Table S2) and 20.31, respectively. The combined impact of sample size and effect size on the AUC is shown in Figure S2.

3.1.3 Data variability

The AUC is also highly dependent on data variability, expressed as SD. Since Cohen's d is calculated by subtracting the means of the control group and patient group, divided by the pooled SD, an increase of the SD directly leads to a decrease in Cohen's d. This translates, as shown in Figure 4, into a decreased AUC (Figure 5).

3.2 Example: EEG beta band power in generalized epilepsy

A total of 587 children (54.9% boys), with a mean age of 8.8 ± 4.2 years at the first EEG recording, were evaluated at the FSC. Previous seizures (other than in the context of epilepsy) occurred in 97 children (16.5%), and 110 children (18.7%) had other neurological problems in their medical history. Of the 587 children, 136 (23.2%) were diagnosed with focal epilepsy and 66 (11.2%) with generalized epilepsy. The remaining 385 (65.6%) children did not have epilepsy and served as controls. For analysis, we only included the controls and children with generalized epilepsy.

The beta band Cz power was significantly higher in children with generalized epilepsy than in children without epilepsy, with a median difference of −.075 (CI: −.14 to −.0064; p = .034) (Figure 6A). However, univariable model predictions, with the Cz power as the sole predictor, for the presence of generalized epilepsy had poor discriminative power, with an AUC of .58 (CI: .51–.65) (Figure 6B,C). Adding the variables, sex, seizure history, age at EEG recording, and neurological history to the model, improved the overall performance to an AUC of .67 (CI: .60–.74) (Figure 6C,D).

4 DISCUSSION

This study demonstrates the discrepancy between group-level significance and the individual-level discriminative power of potential biomarkers. In contrast to what is often implied when significant group differences are put into perspective, the statistical significance of mean group differences is a very poor indicator of the utility of a variable as an individual biomarker. Rather than significance, the effect size directly impacts discriminative power and is thus more suitable for evaluating a variable's biomarker potential.

Are group-level differences not important at all? The importance of one (group-level) or the other (individual-level) outcome depends on the research question and the stage of the research. In fact, most biomarker research starts with the search for group-level differences, as this is the basis for estimating the effect size. Therefore, also sample size matters. Although sample size does, as shown, not affect discriminative power, effect size estimates can only be precise and accurate with an appropriate sample size.⁴¹ Both underestimates and overestimates of the effect size can result in a distorted view of a variable's potential to be a biomarker.

Based on the required effect size, many individual variables will probably not qualify as good biomarkers. Moderate to even large effect sizes, traditionally defined as Cohen's d's of .5–1.0,⁴² do not translate into sufficient discriminative power if tested in isolation. Our results show that an effect size of 1.25, corresponding to an OR of almost 10, is needed for a variable to reach an AUC value of at least .8. Biomarkers with both a sensitivity and specificity of .8 require an even larger Cohen's d of 1.66, which corresponds to an OR greater than 20. We found a relatively large number of biomarkers with a large effect size in our concise literature search. This number might be inflated as we only reviewed publications explicitly reporting a “biomarker” in the title or abstract. Nonetheless, 37.5% of the studies reported a biomarker with an effect size smaller than a Cohen's d of 1.25. The minimal required effect size for individual biomarkers promotes the use of multivariable approaches. As illustrated in our data example, combining multiple (clinical) variables with smaller effect sizes could increase the overall effect size and discriminative power. Multivariable approaches also better suit the complexity of the pathophysiology of epilepsy.

Our study has limitations. We only simulated normally distributed data, whereas typical epilepsy outcomes are often distributed according to more complex patterns. Moreover, our simulated training (i.e., the original) and validation datasets had the same distribution parameters, while in a real-case scenario, those datasets are likely to differ—at least to some extent—in distribution as they are collected from independent populations. Hence, our simulation results might be too optimistic. Secondly, we collected our example data retrospectively, which is generally regarded as a suboptimal approach for evaluating a diagnostic biomarker.³⁰ Nevertheless, we best approximated a prospective study by sampling patients and controls from a suspected, not yet diagnosed population. Additionally, to keep our example as clear as possible, we quantified the discriminative ability of the EEG data models using the same data from which the models were developed.⁴³ Model validation on independent data most likely would have yielded even worse discriminative power, strengthening our general message to be careful with p values and focus on sensitivity, specificity, and the AUC. Lastly, besides traditional ROC analysis, other biomarker evaluation methods exist,^{44, 45} particularly for assessing a biomarker's added value, clinical utility, or healthcare impact. Although we did not cover these methods here, we are aware that discovering biomarkers with sufficient discriminative power is the first rather than the last step of the biomarker evaluation process, as discriminative power does not necessarily translate into added value in clinical practice or improved outcomes for the patient or healthcare system.⁴⁶

We do not intend to present a pessimistic view of biomarker discovery efforts, as we believe biomarkers will have an invaluable role in personalized medicine in epilepsy.⁴⁷ Instead, we aim to promote the use of appropriate biomarker selection methods and increase methodological knowledge. This study might aid with translating published effect sizes into hypothetical AUC values. In line with this, we recommend researchers to always report effect sizes instead of only p values.⁴⁸ Moreover, we call for the objective reporting of study results without unsupported or unjustified claims on biomarker potential. Growing knowledge and awareness of the methodology of biomarker research on both the authors' and interpreters' side will help move the field in the right direction and contribute to the further evolution of precision medicine.

CONFLICT OF INTEREST STATEMENT

None of the authors have any conflicts of interest to disclose. We confirm that we have read the journal's position on issues involved in ethical publication and that this report is consistent with those guidelines.