2.1 Study setting and data collection

The two participating hospitals (the Royal Adelaide Hospital and Flinders Medical Centre) are tertiary care centers with multiple specialized and general neurology outpatient departments, including epilepsy clinics. These outpatient departments are typically staffed by consultants and neurology advanced trainees. In Australia, advanced trainees are usually postgraduate year 5 or higher specialty trainees. These participating hospitals do not currently have the capacity to perform stereoelectroencephalography. When patients undergo evaluation for epilepsy surgery candidacy, such as with stereoelectroencephalography, they are typically referred for evaluation at other centers. Data were collected from the inception of the implementation of the institutional electronic medical record at each participating center to July 2022. Therefore, the inclusion period for each center was from March 2020 and July 2021, respectively.

Notes entered in outpatient neurology visits in which the word “epilepsy” was used were collected. For each patient, the first recorded note from the time of inception of the electronic medical record was included. The reason for this selection criteria was that, for existing patients, in the first visit after transfer to the electronic medical record, previous documentation was frequently transcribed from paper notes into the electronic medical record to provide a site of reference moving forwards. Additionally, for patients who were seen for the first time after the introduction of the electronic medical record, a first visit typically includes a full history with associated comprehensive documentation. When the first note was not a clinic note (a medical note written by a doctor following in-person or phone review of a patient—as opposed to administrative and other notes), these notes were included in the study since the algorithms ability to correctly identify clinic notes is also a significant factor that will influence subsequent applicability.

Manual review of the included notes was conducted by investigators to extract the following variables: Whether an included note was in fact a clinic review note; whether a patient had a diagnosis of epilepsy (this was either based on a mentioned diagnosis by the treating doctor or description of patient having 2 or more separate occasions of unprovoked seizures); age; gender; the presence of focal onset seizures; the presence of generalized onset seizures; whether a lesion was described as being present on magnetic resonance imaging (MRI); whether a focal abnormality was described as being present on positron emission tomography; the number of antiseizure medications previously trialed; whether the patient was seizure-free at the time of review (either by explicit use of this term or having been described not to have had seizures since a preceding appointment or intervention); whether they had previous epilepsy surgery; whether they were referred for epilepsy surgery as a result of the clinic encounter; whether they had any absolute or relative contraindications to epilepsy surgery; and whether they met prespecified criteria for referral for epilepsy surgery workup. The presence of absolute and relative contraindications was considered to include expressed opposition to surgical management, progressive incurable imminently life-limiting illness, and significant cognitive impairment.¹¹ The criteria for referral for epilepsy surgery workup were focal epilepsy, failure of at least two described appropriate antiseizure medication trials, age ≤70, and the absence of described contraindications. For patients who were considered epilepsy surgery workup candidates based on the prespecified criteria, but who were not referred in the included clinic note, subsequent case note review was undertaken in September 2023 to enable the determination of outcome for these individuals with at least 12-month follow-up.

It is acknowledged that certain criteria described here may be debated, particularly regarding aspects of relative and absolute contraindications.¹² The described, relatively narrow criteria were selected to provide a number of variables for the machine learning analysis on which its performance could be evaluated. Similar methods could be employed for alternative criteria.

In the collection of these variables, only the included clinic note was used for reference. In other words, when there was insufficient information in a given note to ascertain a piece of data (such as the number of antiseizure medications previously trialed), other components of the electronic medical record were not reviewed to determine this information. The reason for this approach was that it was acknowledged that a certain proportion of patients would have insufficient information available in a clinic note to make an evaluation as to whether they may be a candidate for epilepsy surgery. Since the machine learning analysis was intended to be performed on a note-wise basis, the gold standard that was required for comparison against the machine learning was the best conclusion that was able to be drawn from the individual note in question (see Section 4).

2.2 Machine learning analysis

The free-text document for each patient was the input for the machine learning analyses. Data preprocessing involved negation detection, punctuation removal, stop-word removal, and Porter stemming. Gendered pronouns were removed during the preprocessing stage. The Natural Language Toolkit (NLTK) library was used to perform negation detection, which involves appending a suffix of (“_NEG”) to any terms that follow a negating term (such as “not”) in a sentence.¹³ Text was then transformed into a term frequency inverse document frequency (TF-IDF) array using n-grams of one to three word stems in length. Data were then split into training and testing datasets with a random 80%: 20% split.

XGBoost, logistic regression, and random forest models were then developed on the training data set using fivefold cross-validation. Hyperparameter tuning and architecture experimentation were conducted at this stage. Cutoff scores were derived with Youden's index.¹⁴ After each model had been optimized on the training data set, performance analysis was evaluated on the holdout test data. Area under the receiver operator curve (AUROC) values were generated with 95% confidence intervals (95% CI) calculated via bootstrapping. The primary outcome of the study was the AUROC for the best performing model in the classification of individuals meeting criteria for consideration of epilepsy surgery referral. Calibration graphs were produced with four bins. The application of cutoff scores enabled the calculation of metrics including sensitivity, specificity, and F1 scores, which were calculated as standard. Misclassifications of the best performing model, and word stems with the highest/lowest regression coefficients, were collected for epilepsy surgery workup suitability classifications. Analyses were undertaken using open-source Python and R libraries, including SciKit-Learn and XGBoost.^{15, 16}

2.3 Ethical approval

This study was conducted with institutional review board approval from the Central Adelaide Local Health Network and Southern Adelaide Local Health Network Research Ethics Committees (reference numbers 14750 and 35.22).

3.1 Patient characteristics

There were 457 notes included in the study. Of these notes, 307 were clinic notes, and 250 of 307 notes were for patients with epilepsy. Of the patients with epilepsy, the mean age was 45.3 (SD 17.8) and there were 135 female (54%). There were 117 patients with focal seizures (46.8%) and 49 (19.6%) with generalized seizures. Two of these patients were described to have both focal and generalized seizures. There were 86 (34.4%) individuals in which the type of seizure was not specified in the included note. The number of patients with a described MRI was 72 (28.8%) and the number with a PET was 5 (2%). The number of patients who had a focal MRI lesion described as being present was 40 (16%). There were no patients that were described to have a focal PET abnormality. Accordingly, 32 and 5 patients had a non-lesional MRI and PET, respectively. Patient characteristics are displayed in Table 1.

3.2 Epilepsy treatment

The median number of ASM that had been described to be trialed previously was 2 (IQR 1–3, minimum 0, maximum 6). There were 129 (51.6%) patients who had trialed ≥2 ASM. There were 130 (52%) patients who were seizure-free at the time of review. There were 31 individuals with either a relative or absolute contraindication to epilepsy surgery. The number of patients referred for epilepsy surgery as a result of the clinic visit was 5 (2%), and 4 (1.6%) had previously undergone epilepsy surgery. There were 37 (14.8%) individuals who fulfilled the prespecified criteria for epilepsy surgery referral without described contraindications, 32 (12.8%) of whom were not referred for epilepsy surgery in the given clinic visit. There was no described reason for non-referral in all of these cases.

3.3 Follow-up outcomes

In the 32 identified clinic notes in which there was no referral but individuals fulfilled the prespecified criteria for referral, all patients had at least 12-month follow-up period prior to outcome determination. In this group, there were four (12.5%) of patients who were subsequently referred for epilepsy surgery evaluation. There were 11 (34.4%) of patients who were described to have been content with their seizure frequency and not pursued epilepsy surgery evaluation as a result. There were seven (21.9%) of patients who achieved seizure freedom with medications. There was one (3.1%) patient in whom lack of medical engagement and medication non-adherence was considered to preclude epilepsy surgery evaluation. There was one (3.1%) patient for whom epilepsy surgery was considered too high risk. There were four (12.5%) cases in which there was no described reason for not referring for epilepsy surgery evaluation. There was one (3.1%) patient who moved interstate. There were three (9.4%) patients who died.

3.4 Machine learning analysis

The machine learning natural language processing performance characteristics are summarized in Table 2. With respect to the primary outcome of the study, the tested models performed similarly and were largely equivalent. Given the prespecified analysis plan to examine misclassifications of the best performing model, the random forest model was selected as it provided the numerically highest AUROC. This model returned an AUROC of 0.97 (95% CI 0.93–1.0) for this task (see Figure 1). When results were dichotomized, this model provided a sensitivity of 1.0 and specificity to 0.93 for the classification of those meeting the criteria for epilepsy surgery workup referral. However, the examination of the calibration graph revealed that the model was not well calibrated (see Figure 2), which is consistent with the predominance of false positives (and absence of false negatives) returned by the model.

With respect to misclassifications, the random forest model in the identification of individuals suitable for epilepsy surgery workup referral (random forest) had no false-negative results on the holdout test dataset. In other words, it can be seen that the random forest model correctly identified all patients who were indeed referred for epilepsy surgery as a result of their clinic visit that were in the test dataset. Therefore, the only examples of misclassifications for this model were false positives. One example of a false positive was a telehealth consultation regarding a patient with epilepsy on oxcarbazepine with adequately controlled seizures but significant drowsiness on the medication, who was planned for oxcarbazepine levels, a sleep deprived electroencephalogram and a brain magnetic resonance imaging scan with an “epilepsy” sequence. This patient was not classified as fulfilling the criteria for referral for multiple reasons, including lack of information regarding their seizure onset (it was unclear from the note whether they had focal onset or generalized onset seizures) and insufficient evidence of ≥2 appropriate antiseizure medication trials. However, it could be suggested that it is most likely that the patient has focal seizures (given the choice of oxcarbazepine) and that they most likely have previously failed other antiseizure medications (since to access oxcarbazepine on the Australian Pharmaceutical Benefit Scheme, one must previously have had seizures that were unable to be controlled by other agents).

When examining the word stems with the most extreme regression coefficients in the prediction of epilepsy surgery referral candidature, patterns relating to medical treatment and note types were observed. In particular, word stems with strong positive regression coefficients typically related to focal epilepsy, and to second- or third-line antiseizure medications (e.g., “lacosamid,” “seizur,” “tempor,” “focal,” “perampanel,” and “oxcarbazepin”). There were also several words that alone may indicate a degree of overfitting, although in context may relate to lack of seizure freedom, namely “month” and “time.” There were also word stems that related to note types that were not clinic notes (e.g., inclusion of fax numbers in administrative notes).