Reliability and predictive validity of the Standardized Infant NeuroDevelopmental Assessment neurological scale

SINDA's neurological scale

SINDA's neurological scale has been designed as a screening tool that: (1) is applicable in the first year of life after the neonatal period, that is in the age range of 6 weeks to 12 months corrected age; (2) covers all infant neurological domains; (3) is standardized, that is it has an identical set of items and criteria in the age range at issue; (4) results in a score that is largely independent of the infant's age; (5) is easy for general paediatricians to use and takes about 10 minutes to perform (including recording of the scores); (6) includes a substantial proportion of items evaluating the quality of spontaneous movements; and (7) assists the prediction of developmental outcome.

The neurological scale has five domains assessing spontaneous movements (eight items), cranial nerve function (seven items), motor reactions (five items), muscle tone (four items), and reflexes (four items; see Appendix S1, online supporting information). The assessment procedures, definition of the items, and criteria for typical and atypical performance have been described in the manual (hitherto unpublished).

Each item is scored as pass or fail according to simple and well-defined criteria. For many items, consistent asymmetry results in the assignment of ‘fail’. Seven of the eight items on spontaneous motility evaluate movement quality in terms of variation versus stereotypy, the eighth item addressing the quantity of motility. The classification of movement variation versus stereotypy is based on clinical observation, not on video assessment as in the assessment of general movements.1 This implies that only striking stereotypies, including consistent asymmetries, are recorded as atypical, in line with clinical practices.7, 10-12 The inclusion of seven items on movement quality also means that asymmetries, such as a head-turn preference with an accompanying asymmetry in arm movements, and eventually in hand movements which occurs relatively frequently in 2-month or 3-month-old infants, only results in a reduction of 2 or 3 points. The latter may be interpreted as a minor dysfunction. This contrasts with strikingly stereotyped movements in all parts of the body that will result in a 7-point reduction of the score. The cranial nerve function domain includes items assessing facial and oral motor behaviour, eye movements, and reactions to light and sound. The motor reactions domain consists of items evaluating the infant's reaction to postural stimulation; examples are the pull-to-sit manoeuvre and vertical suspension. Muscle tone is evaluated separately in neck and trunk, arms, legs, and feet. The reflexes domain does not only contain tendon reflexes, but also the footsole response and footsole sensibility (i.e. the response of the infant's foot to gentle tickling). We decided to exclude responses such as the Moro response, the palmar and plantar grasp responses, and the parachute reaction from the examination as they are clearly age-dependent. The latter interfered with our aim to develop an assessment that was largely independent of the infant's age in the first postterm year.

The aim of the present study was to assess the following properties of SINDA's neurological scale in a sample of infants at risk of motor and mental developmental disorders: (1) intrarater and interrater reliability; (2) item difficulty and item discrimination; (3) dependency on infant age; and (4) validity to predict adverse developmental outcome at 24 months or older corrected age. This means that the present study is a first phase in the validation of SINDA's neurological scale. One of our next steps is to test the scale's predictive properties in the general population.

Participants

The study is a centre-based longitudinal case series, consisting of 181 infants (92 males, 89 females) who had been admitted to the Centre for Child Neurology in Frankfurt, Germany (SPZ Frankfurt-Mitte). Social paediatric centres in Germany are specialized outpatient clinics for infants at risk of or with a neurodevelopmental disorder. From May 2012, SINDA's neurological scale was incorporated in SPZ Frankfurt-Mitte's clinical routine. In the present study infants were consecutively included when they had their first visit when aged between 6 weeks and 12 months between May 2012 and November 2014, and had detailed outcome data reported in the medical records at 24 months or older corrected age. The latter included a neurological examination and a standardized neurodevelopmental assessment in 177 (98%) infants. Infants were excluded if they had (1) a progressive neurological disorder (n=4); (2) a behavioural state incompatible with SINDA (n=1); or (3) a phenotypical expression of a genetically determined developmental disorder as this may cause assessor bias (n=13, all trisomy 21). Also excluded were the infants who had a SINDA, but no follow-up assessment at 24 months or older. The latter was mostly because of their clinical status requiring no or less specialized follow-up. Table 1 summarizes the background characteristics of the study group. The study was approved by the ethical committee of the Medical Faculty of Heidelberg University, Germany (S-021/2017).

SINDA

SINDAs were performed by the seven general paediatricians (of whom three were in training for paediatric neurology) of SPZ Frankfurt-Mitte. These paediatricians had received the SINDA manual (unpublished material) and been trained in using SINDA through video sessions and attending life assessments performed by one of SINDA's developers (HP). SINDA's neurological scale has been described in the introduction (see also Appendix S1). Each of the 28 items is scored as pass (1) or fail (0). The number of passed items is added to form SINDA's neurological score, with a maximum of 28 points and the various domain scores.

Neurodevelopmental assessment at 24 months or older

At a median age of 29 months corrected age (range 24–57mo), the children had a follow-up assessment by the clinical team of SPZ Frankfurt-Mitte (consisting of seven paediatricians and two psychologists). The paediatrician in charge of the follow-up assessment knew the medical history of the child and the child's SINDA scores. However, the paediatrician was not aware of the significance of the SINDA scores, as that was undetermined at that time. The follow-up assessment consisted of a neurological and physical examination by one of the paediatricians, and a standardized developmental assessment by one of the psychologists. The neurological examination was the standardized assessment described by Michaelis and Berger.11, 12 The diagnosis of CP was based on this assessment, according to the criteria of the Surveillance of Cerebral Palsy in Europe.13 In children aged less than 43 months, the developmental assessment consisted of the Bayley Scales of Infant Development, Second Edition measure;14 in three older children other standardized tests were used for mental development (Snijders-Oomen Non-Verbal Intelligence Test revised version,15 Wechsler Preschool and Primary Scale of Intelligence,16 and the ET 6–6 developmental assessment),17 and in two children for motor development (Movement Assessment Battery for Children, Second Edition18 and ET 6–6). In another four children, whose neurological exam showed typical function, developmental outcome was based on developmental screening by the paediatrician. This screening showed average or above average performance. At the time of our study, the German norms of the Bayley Scales of Infant Development, Third Edition were not available and application of the US-norms had some problems; therefore the Bayley Scales of Infant Development, Second Edition was used.19 The Bayley Scales of Infant Development, Second Edition results in two outcome scores, the Psychomotor Development Index and the Mental Development Index. Outcome was classified as typical or atypical, with atypical outcome implying the presence of a clear neurological syndrome such as CP or the presence of a Mental Development Index and/or Psychomotor Development Index lower than 70 or its equivalent.

Evaluation of psychometric properties and statistical analyses

Interrater and intrarater reliability was calculated on the basis of 24 videotaped SINDA neurological examinations; this number is in line with similar studies in the field.5 The videos were selected by the clinical staff, who were not involved in the reliability study. Care was taken to create a sample that included variation in age and neurological dysfunction (for details see Table SI, online supporting information). Three assessors (MH-A, JP, and HP), masked to the infant's age and clinical history, independently assessed the videos twice with an interval of 14 to 18 months. The intrarater agreement was based on the two assessments of the three examiners. To determine the interrater agreement, the second assessment of the three examiners was used. The video-assessments did not allow for the evaluation of the item ‘pupillary reaction’. Of the remaining 648 items (27 items for 24 infants) another 21 single item-assessments (3.2%) had to be excluded because of impaired visibility on the video. Thus a final set of 627 videotaped items of 24 neurological assessments was used for intrarater and interrater reliability.

Reliability was calculated using two approaches.20 First, Cohen's kappa coefficient ĸ, a robust measure for rater agreement, was used to determine intrarater and interrater reliability for the categorical single items (only possible for two raters).21 According to Fleiss,22 kappa values of 0.40 to 0.75 are rated as fair to good. Second, intraclass correlation coefficients (ICC 2,k) were calculated to evaluate intrarater and interrater reliability of domain scores and the total score. ICC calculation is a proper inferential statistical method for quantitative measurements; ICCs can be derived for multiple raters.23

Special attention was paid to item analysis, an important issue in the development of tests. It refers to statistical methods used to select valuable items for inclusion in the test under construction. We calculated two parameters: item difficulty and item discrimination. Evaluation of these parameters was based on all 181 SINDA assessments available. To calculate item difficulty for dichotomous items – as in SINDA – the number of positive items (in our case atypical performance) is divided by the number of examinees; this results in a proportion, on the scale of 0 to 1. Item discrimination was based on a point-biserial correlation of the single items (0=atypical, 1=typical) with the total score. The item discrimination provides an estimate of the degree to which a particular item is measuring the same aspects as the test as a whole. Its value can range between 0 and 1, with 1 indicating a perfect discrimination.

To assess whether SINDA's neurological scale was largely independent of age, the association between the infant's corrected age at assessment and the neurological (total) score was evaluated with the Spearman rank correlation coefficient (rho [ρ]).

Predictive validity of the SINDA assessment was calculated for two different outcomes (atypical outcome and CP). We determined a score below the 25th centile of the study group as ‘at risk’, a cut-off also meeting face validity (Fig. 1). Next, sensitivity and specificity of the at risk score for atypical neurodevelopmental outcome or CP at follow-up was determined, including their 95% confidence intervals (CI).

Most statistical analyses were performed on a personal computer system using SAS Version 9.4 (SAS Institute, Cary, NC, USA). ICC calculations were performed using the statistical package R version 3.4.2 (R Foundation for Statistical Computing, Vienna, Austria). We used parametric and non-parametric statistics when applicable. For instance, age at examination and neurological total score did not meet the criteria of normal distribution, we therefore used median and centile values as descriptors, and we applied Spearman rank correlation coefficients and χ² calculations as tests.

Reliability

The duration of the video recordings of the neurological assessments varied between 4 minutes 45 seconds and 10 minutes 52 seconds (median: 8min 20s). For the single items, the Cohen kappa values of intrarater agreement were 0.601 (HP), 0.644 (MH-A), and 0.718 (JP), denoting substantial agreement. The ICCs of the intrarater agreement of the total scores varied between 0.923 and 0.947, indicating excellent agreement; the ICCs of the domain scores reflected good to excellent agreement (Table 2).

For the single items, the Cohen kappa values of interrater agreement were 0.572, 0.672, and 0.667, again implying substantial agreement. For the domain scores, all but one of the ICCs of the interrater agreement were higher than 0.750, the single value of the reflexes domain (0.573) being the exception to the rule (Table 2). The ICCs of the interrater agreement of the total scores indicated excellent agreement, with an overall ICC (2,k) of 0.965.

Item difficulty, item discrimination, and age-dependency

The easiest item was ‘pupillary reaction’ (item 14, cranial nerve function) which was scored atypical by one infant only (0.6%); the most difficult item was ‘pull-to-sit’ (item 16, motor reactions) which was scored atypical in 71 infants (39%). Mean item difficulty was 20% (Figure S1, online supporting information).

The point-biserial correlation analysis indicated that the mean discrimination index was 0.477, indicating adequate discrimination. The correlation coefficients ranged from 0.107 (‘pupillary reaction’, item 14, cranial nerve function) to 0.702 (‘spontaneous movements of the hands’, item 3, spontaneous movements; ‘resistance against passive movements of the arms/arm traction’, item 22, muscle tone); see Figure S1, online supporting information.

Item difficulty and item discrimination revealed, for instance, that the item ‘pull-to-sit’ (item 16, motor reactions) is an item on which infants frequently ‘failed’ (atypical in 39%). Its item discrimination of 0.366 indicated that the contribution of this item to the test result as whole is only moderate. The item difficulty and item discrimination also indicated that the item ‘pupillary reaction’ is an item on which infants infrequently scored atypical, and that its contribution to the overall risk score was limited (item difficulty 0.107). Nevertheless, we considered the item neurologically too important to be left out of SINDA's neurological scale.

The infant's corrected age at assessment was not associated with the neurological score (ρ=–0.019; p=0.803).

Predictive validity

The SINDA neurological (total) scores ranged from 7 to the maximum of 28. Preliminary analysis indicated that the lowest 25th centile meant a score of 21 points or lower. Therefore, we calculated the sensitivity and specificity of a SINDA neurological score of 21 points or lower. This cut-off for at risk was first used to calculate sensitivity and specificity for atypical neurodevelopmental outcome at 24 months or older, which was diagnosed in 56 children: sensitivity was 0.893 (95% CI 0.781–0.960) and specificity was 0.936 (95% CI 0.878–0.979; Table 3, Fig. 1).

Twenty-nine children were diagnosed with CP (Table 1). The sensitivity to predict CP was 1.000 (95% CI 0.881–1.000) and its specificity was 0.809 (95% CI 0.738–0.868; Table 3). Twenty-seven children had an atypical outcome but not CP, most often consisting of a low Mental Development Index. The sensitivity to predict atypical outcome in the group of children without CP was 0.778 (95%CI 0.577–0.914) and specificity was 0.936 (95% CI 0.878–0.972; Table 3).