2.1 Subject information

Approval for this retrospective study was obtained from the Institutional Review Boards at the Children's Hospital of Orange County (CHOC) and the University of California Los Angeles (UCLA), with the requirement for informed consent waived. Twenty subjects diagnosed with LGS (7 females, median age 7.4 years, age range 1.0–18.8 years) were retrospectively identified using ICD 9 and 10 diagnostic codes from the clinical record at CHOC, with visits and EEG studies performed between January 2012 and June 2020. The electroclinical diagnosis of LGS was confirmed by a board-certified pediatric epileptologist. The median time from seizure onset to EEG collection in the cohort was 4.2 years and the median time from LGS diagnosis to EEG collection was 0.1 years (Supplementary Table S1). Twenty healthy control subjects (8 females, median age 8.2 years, age range 1.0–17.7 years) were selected from a cohort of fifty subjects collected in a prior study. The twenty healthy subjects were selected such that they were approximately age-matched to the LGS group. The cohort of fifty control subjects was retrospectively identified from the clinical record at UCLA with visits between February 2014 and July 2018.¹² Controls were included if they had (1) no known neurological disorders, (2) a normal overnight video-EEG, (3) EEG events that were not seizures and were deemed neurologically normal by the attending neurologist, and (4) no use of anti-seizure medications. There were no significant differences in age between control and LGS subjects (P = 0.672, Student's t-test); the breakdown by sex was approximately the same between both groups.

2.2 EEG acquisition and preprocessing

All EEG data were recorded using the Nihon Kohden EEG acquisition system, with nineteen scalp electrodes placed according to the international 10–20 system (Fp1, Fp2, F3, F4, C3, C4, P3, P4, O1, O2, F7, F8, T3, T4, T5, T6, Fz, Cz, Pz). The data were recorded at a sampling rate of 200 Hz or downsampled to 200 Hz. For each LGS subject, an EEG clip of NREM sleep lasting 10 minutes containing no arousals or artifacts was selected. First, periods of NREM sleep were identified between the times of 12:00–4:00 AM by a pediatric epileptologist (MR). Three subjects were not asleep during this time window; one subject had NREM sleep from 4:00–6:00 AM and two subjects had NREM sleep from 6:00 PM to 12:00 AM the night before. Then, using an automated artifact detection algorithm, the first clean 10-minute clip of NREM sleep was selected for analysis. For each control EEG recording, a pediatric epileptologist clipped 20–30 minutes of NREM EEG using a randomization method to select the time point, as described in Smith et al. (2021).^{12, 13} For each subject, one 10-minute clip of clean, continuous, NREM sleep EEG with no automated or clinically detected artifacts was selected for analysis. All EEG data were re-referenced to the common average and filtered with a zero-phase shift digital filter from 0.5–55 Hz. All electronic data were deidentified and analyzed offline using custom MATLAB (Mathworks) scripts.

2.3 Identifying EEG events of interest

The EEG data were first prewhitened in the time domain using first-order backward differencing to counteract the expected 1/f decrease in power.^14-16 The time-varying power spectrum of the EEG for frequencies from 1 to 50 Hz was subsequently obtained using the Stockwell transform,¹⁷ in increments of 1 Hz.

We then identified regions of high power in the time-varying power spectrum that exceeded a threshold of 250, which corresponded to a z-score of 1.69 for the EEG power and resulted in selection of 4.6% of all time-frequency values. This relatively low value for the threshold was chosen to ensure that all relevant waveforms, particularly epileptiform activity, would be captured. This fixed threshold was used across all frequency bands and subjects, as the prewhitening step served to normalize the power across frequency bands and between LGS and control subjects.

For each EEG, we defined events of interest (EOIs) to be regions of high power in the time-varying power spectrum that were continuous in time (Figure 1). Specifically, a single EOI consisted of consecutive time points in which the power for at least one frequency exceeded the threshold, with a minimum duration of 100 milliseconds. This minimum length was chosen to fully capture the duration of an epileptic spike, the shortest abnormal waveform we expected to see. Note that epileptic spikes and other sharp waveforms will appear to have a longer duration in the time-varying power spectrum, such that even a 20 millisecond spike can have a time-frequency duration exceeding 100 milliseconds (Supplementary Figure S1). EOIs were identified in the EEG data from the Fz electrode, as the frontocentral location should be minimally impacted by eye movements and muscle artifact and maximally sensitive to events such as sleep spindles and GPFAs.¹⁰

2.4 Time-frequency image features for each EOI

2.5 Clustering analysis

Prior to clustering, each feature was normalized by converting the values to a z-score based on the mean and standard deviation of each feature across all control and LGS EOIs. Control and LGS EOIs were collectively partitioned into twelve different clusters using K-means.¹⁸ Each EOI was treated as a single observation; each observation consisted of six variables, which were the z-scores of the features: height, duration, spread, density, MP, and FPP. A large number of clusters was used to ensure that all combinations of TFI features were represented, thus resulting in a broad distribution of EOIs for visual analysis. For organizational purposes, the clusters were named in descending order based on the sum of the six TFI feature z-scores, such that cluster one had the highest sum of the six TFI feature z-scores, and cluster twelve had the lowest sum.

2.6 Visual analysis of EOIs

To ensure the selection of a representative distribution of EOIs for visual classification, 120 EOIs were randomly selected from each cluster (60 from LGS patients and 60 from control subjects). If a cluster contained fewer than sixty control or LGS EOIs, the remaining EOIs were selected from the other cohort. Cluster one consisted of only 30 EOIs, and all EOIs were included in this cluster. In total, 1350 out of 11 708 EOIs were selected for visual analysis. Three board-certified pediatric epileptologists (DA, DS, SH) from two different institutions (CHOC and UCLA) each classified 900 of the 1350 EOIs, so each EOI was independently classified by two different raters. The order of EOIs was randomized. For each EOI, raters were given a fifteen-second segment of EEG, starting 10 seconds prior to the EOI and ending 13 seconds after the EOI began, with a red line indicating when the EOI started and ended (Figure 3). Raters were blinded to the subject type, subject number, and the time at which the EEG segment was recorded. For each EOI, raters were asked to determine (1) the subject type, which could be either a control or LGS subject and (2) the waveform type, which could be spike and slow wave (SSW), GPFA, seizure, sleep spindle, vertex sharp, muscle, artifact, other event, or nothing. EOIs were viewed and classified using a custom GUI that was designed using MATLAB in consultation with the clinical team.

2.7 Statistical analysis

For the subject type, we also determined the accuracy of the visual classification relative to the ground truth of whether the subject had been diagnosed with LGS or not. The accuracy of the subject type labels was calculated as a percentage based on the subject type from which each EEG EOI originated.

3.1 Automated EOI detection identified a broad range of waveform types from all subjects

A total of 11 708 EOIs were detected using the process described in Section 2.3, with 6744 EOIs coming from the twenty control subjects (n_controls = 365.0 [259.0–443.0] EOIs per subject; reported as the median [Q1-Q3] for all results) and 4964 EOIs coming from the twenty LGS subjects (n_LGS = 232.5 [156.0–324.5]). The minimum number of EOIs for a single subject was 49, and the maximum number of EOIs was 560. There were no clusters composed of EOIs from a single subject; on average, each subject contributed EOIs to 8.5 different clusters (Supplementary Figure S2). Each cluster was characterized by a different combination of the six features (Supplementary Figure S3). For example, the EOIs in cluster one had a long duration and high spread, while EOIs in cluster two had high density and high mean power. Some clusters had low values of all features; this was expected, as EOIs were identified using a relatively low power threshold to maximize the sensitivity of initial detection. The broad distribution of EOIs across subjects and clusters and the unique characterization of each cluster based on the six features suggest that our algorithm was successful in identifying a broad range of waveforms that were well-represented in both LGS and control subjects. Clustering EOIs using a power threshold of 200 (7.7% of the data) and 300 (3.0% of the data) produced qualitatively similar clusters (Supplementary Figure S4). For the visual analysis, 60% of the EOIs (n = 813 out of 1350) were from LGs subjects, and 40% (n = 537) were from control subjects.

3.2 Classification of subject type was accurate and consistent between raters

The IRR for the determination of subject type (ie, LGS vs. control) associated with each EOI was favorable, with κ = 0.790 and ICC = 0.790 (95% confidence interval [CI] 0.769–0.809) across the 1350 EOIs selected for visual analysis (Table 1). Both control and LGS subject labels had similar agreement between raters, with an EAR of 0.806 and 0.813, respectively (Table 2). The mean accuracies for subject classification were 84.4%, 84.7%, and 86.0% for the three reviewers, with significantly lower accuracies for EOIs from LGS subjects compared to EOIs from control subjects for all three raters (Supplementary Table S2; P < 0.05, Mann–Whitney U-test). There were no significant differences in accuracy between the three raters across all LGS and control subjects (P > 0.05, Mann–Whitney U-test; 3 of 3 comparisons).

3.3 Interrater reliability for waveform type was inadequate, with some very low agreement rates

The IRR for waveform type had κ values ranging from 0.538 to 0.572, with a mean value of 0.558 (Table 1). The ICC between raters was also below adequate, with an ICC of 0.669 (CI 0.638–0.698). The highest number of consistent ratings between reviewers were for EOIs labeled as SSWs (n = 338, EAR = 0.710), nothing (n = 391, EAR = 0.592), and sleep spindles (n = 118, EAR = 0.490), although the EAR values for nothing and sleep spindles did not indicate high levels of agreement. The high number of EOIs labeled “nothing” further confirms the highly sensitive nature of the automated algorithm for identification of waveforms. The EAR values for other waveforms, such as GPFAs (n = 35 with matching labels, EAR = 0.285) and vertex waves (n = 11, EAR = 0.122), were low, indicating significant disagreement in identifying LGS waveforms and sleep architecture, respectively (Table 2).

Mismatches in waveform labeling mostly occurred when one rater classified the EOI as nothing (Figure 4). Of the 439 EOIs with a disagreement in waveform labels between the two reviewers, 270 EOIs (61.5%) were labeled as nothing by one reviewer. This happened most frequently for nonepileptiform waveforms; for example, mismatched labels of spindle/nothing occurred 83 times, and labels of vertex/nothing occurred 54 times, accounting for a total of 31.2% of all EOIs with mismatched labels. Of the waveform types typically associated with epilepsy, SSWs had the highest EAR, but 37.0% of all SSWs with mismatched labels occurred when one rater marked nothing. For GPFAs, the mismatches occurred when the second rater chose SSW (38.6%), muscle artifact (21.6%), nothing (18.2%), and sleep spindles (12.5%).

4.1 Automated waveform detection in subjects

In all subjects, the automated process for EEG analysis identified transient waveforms with high power. Of the 794 EOIs with rater agreement, epileptiform waveforms such as SSWs and GPFAs came from fifteen LGS (two control) and eight LGS (zero control) subjects, respectively. In contrast, normal sleep EEG waveforms such as spindles and vertex waves were only seen in three LGS (eighteen controls) and zero LGS (nine controls), respectively. This is consistent with a prior LGS sleep study, which reported an absence of sleep spindles in most LGS subjects.²⁴ Sleep spindles and vertex waves may also be difficult to recognize in LGS subjects due to the presence of frequent epileptiform discharges during NREM sleep.^24-26

4.2 Rater accuracy for subject identification

The accuracy for classifying subject type was generally good across all three raters, with an average accuracy of 85% across all EOIs. The rater accuracy was similar between controls and LGS subjects in clusters one through six, where raters had a mean classification accuracy of 96.5% for control EOIs and an accuracy of 91.3% for LGS EOIs. However, the rater accuracy in clusters seven through twelve was 95.3% in controls, compared to 64.3% in LGS subjects. This difference was expected, as control subject EEG should only contain normal physiological waveforms, which are unlikely to be mistaken for epileptiform activity. In contrast, LGS subjects have both epileptiform waveforms and normal physiological waveforms, of which the latter are likely to result in mismarking of an EOI as coming from a control subject. Because LGS EOIs more frequently occurred in clusters one through six, rater accuracy for EOIs from LGS subjects decreased as the cluster number increased, with cluster twelve having the lowest accuracy by far (Supplementary Table S3).

4.3 Interrater reliability in waveform classification

The combined, blinded rating of both LGS and control subject EOIs in the same study provides an important benchmark for IRR of EEG waveforms. Using an approximately equal mix of physiological and pathological EOIs, let us accurately measure how frequently an EOI might be interpreted as benign or even “nothing” and prevented bias toward choosing labels associated with pathological waveforms. However, this broad mix of EOIs coupled with the use of nine different labels for visual analysis could have contributed to the low values of IRR and EAR for waveform classification. To verify that this was not the case, we recalculated the metrics using three broader categories consisting of: (1) pathological (SSW, GPFA, seizure), (2) physiological (spindle, vertex), and (3) others (muscle, artifact, none, other). The use of fewer labels did not substantially improve IRR (κ =0.558 to κ =0.628) or the EAR (pathological = 0.724, physiological = 0.447, others = 0.623) (Supplementary Figure S5). The similarities in rater agreement when using broad categories suggest that the disagreement between clinicians is not due to subtle differences between waveform types.

4.4 Limitations

There are several limitations to our study related to the EOI detection and subject data. First, the EOI detection was accomplished using the Fz channel, which captures the frontocentral activity typical of spindles, GPFA, and generalized activity, but is insensitive to localized transients occurring elsewhere on the head. Second, the EEG clips were not selected from a specific stage of sleep. Short clips of EEG may not be representative of a subject's brain activity over longer periods of time, as paroxysmal events such as GPFAs and SSWs may appear infrequently or not at all within a 10-minute time window. However, both limitations would only affect the number of different types of waveforms in the data set and not the raters' ability to classify them.

The LGS subjects used in this study had a wide range of ages and were diverse in terms of seizure and LGS duration and severity of EEG findings. This could have affected our results in a few ways. This could be seen as a strength of the study, as the rater accuracy for labeling EEG segments as LGS or control was quite high, despite the diversity of the LGS cohort. However, it is possible that the accuracy could have been higher in a more homogeneous cohort. With respect to estimation of IRR, drawing individual EEG EOIs from this broad cohort ensured that raters labeled the full spectrum of possible EEG waveforms associated with LGS. Because of this, there were likely “nonstandard” examples of different waveform types, which could have reduced the IRR. However, this represents a realistic scenario, where each EEG requires robust and accurate visual interpretation, regardless of whether its features are typical.

Rater accuracies may have also been affected by the difficulty of classifying individual waveforms compared to conventional EEG visual analysis. The GUI was designed to mimic standard clinical viewing software, but EOI labeling was done using fifteen seconds of isolated EEG, without having the full EEG study for context. We compensated for this by allowing raters to use the entire segment of EEG as context, but this may have influenced some labels by introducing other salient EEG features within the segment. For example, an EOI with short duration (~100 ms) may not initially look significant, but if an epileptiform discharge occurs later in the 15-second segment, the rater may be more likely to choose a label associated with a pathological waveform. While the use of a deidentified short EEG segment instead of the entire study can reduce bias, future studies may want to evaluate clinically marked waveforms in tandem with automated EOI detections to compare the differences between these two methods. Agreement may be higher in paradigms in which raters assign subject type based on review of full-length clinical EEG, rather than 15-second samples. Conversely, in paradigms with diverse patient composition (eg, a cohort including normal controls, patients with LGS, and patients with other forms of epilepsy), the identification of patients with LGS may be more challenging and IRR is likely to be lower.

4.5 Future work

Given the stated limitations of this work, there are three questions that could be addressed by subsequent studies, using the same computational analysis and visual marking framework. (1) Is disagreement between raters more likely to occur when there is high uncertainty about the selected label? This could be answered by having raters indicate their level of certainty for each waveform label. If label mismatches occur when raters have high certainty, this could indicate a fundamental disagreement about how standard waveforms should be defined. (2) If raters are provided with greater context for each EEG EOI, does this improve the interrater reliability? For example, raters could assess multiple EOIs from a given subject, with the ability to view all EOIs prior to labeling them. This could be done using all EOIs from a single subject, or it could be subdivided by clusters (eg, label cluster 1 EOIs from subject 1, then cluster 1 EOIs from subject 2, etc.). In theory, this should improve the IRR, particularly for established biomarkers such as epileptiform discharges and the slow spike and wave pattern; results for newer biomarkers such as GPFA may be more varied. (3) What is the reliability of visual labeling in a more realistic scenario in which the patient may have a type of epilepsy other than LGS? Labeling an EOI as “GPFA” is likely to be considerably easier when the rater knows that the subject either has LGS or is a healthy control. Therefore, this study could be repeated using a broader cohort including the various types of epilepsy that are most common in this age group, as well as healthy controls.

Improvements could be made to the computational algorithm used for EEG EOI selection, as well. Future implementations could utilize multiple channels in the time-frequency analysis, rather than focusing on only channel Fz. Moreover, the use of EEG clips longer than 10 minutes would increase the likelihood of capturing waveforms such as GPFA and SSW in each subject.