Holland Sleep Disorders Questionnaire: a new sleep disorders questionnaire based on the International Classification of Sleep Disorders-2
Summary
The primary objectives of this study were to construct a self-assessment questionnaire for sleep disorders based on the International Classification of Sleep Disorders-2, and to evaluate the questionnaire’s psychometric properties with respect to its total score and the individual scores for each of the six sleep disorders. In total, 1269 patients, clinically diagnosed with a sleep disorder, and 412 subjects without sleep complaints were enrolled into this study. Principal components analysis confirmed that the Holland Sleep Disorders Questionnaire differentially represented the six symptom clusters associated with International Classification of Sleep Disorders-2 classifications. The Holland Sleep Disorders Questionnaire’s total score distinguished patients with a clinically diagnosed sleep disorder from individuals without sleep complaints, with area under the receiver operating curve P(A) of 0.95. The internal reliability coefficient alpha was 0.90 and, applying the Youden criterion as cutoff score, the overall accuracy was 88% (κ: 0.75). Subsequently, the six diagnostic groups of sleep-disordered patients could be differentiated reliably, with P(A) values ranging between 0.69 and 0.95, alpha coefficients ranging between 0.73 and 0.81 and an overall percentage of 85% correctly classified patients (κ = 0.80), indicating a substantial to excellent agreement between the primary diagnoses and the Holland Sleep Disorders Questionnaire classifications.
Introduction
Although many general sleep assessment questionnaires have become available (see e.g. Spoormaker et al., 2005), a questionnaire based on the International Classification of Sleep Disorders (ICSD-2, 2nd edn; American Academy of Sleep Medicine, 2005) is lacking. As this classification system is required for centre accreditation by the European Sleep Research Society and the American Academy of Sleep Medicine, it is used by the majority of sleep disorders centres for making clinical diagnoses as well as medical record-keeping and maintenance of a patient database (Buysse et al., 2003). The scores on a previously completed ICSD-based questionnaire may help clinicians to increase the efficiency of their diagnostic intake by allowing them to focus quickly on the main sleep disorder category (or categories, in cases of comorbidity) and/or possibly refer to another specialist.
The primary objectives of this study were: to construct a self-assessment questionnaire for sleep disorders based on the ICSD-2, named the Holland Sleep Disorders Questionnaire (HSDQ); and to evaluate HSDQ’s psychometric properties with respect to its total score and the individual scores for each of the six sleep disorders, as defined in the ICSD-2.
Methods
Subjects
Clinical interviewing and history-taking were conducted by experienced neurologists and supplemented by the use of sleep diaries, actigraphy and ambulatory polysomnography.
The final version of the questionnaire was completed by 891 consecutive patients [380 females; mean ± standard deviation (SD) age: 45.3 ± 13.1 years; 511 males, mean ± SD age: 47.5 ± 14.2 years] referred to the Medical Center Haaglanden Sleep Center in The Hague from February 2008 to June 2009, and by 201 control subjects (102 females; mean ± SD age: 43.3 ± 11.5 years; 99 males, mean ± SD age: 50.5 ± 12.8 years). Control subjects were recruited among the patient’s partners, recognizing that these partners were well informed about sleep disorders. Only those control subjects were included who rated themselves as ‘very good’, ‘good’ or ‘neither good nor poor’ sleepers. Identical inclusion criteria applied for an additional group of 211 control subjects recruited among psychology students (156 females; mean ± SD age: 21.0 ± 6.0 years; 55 males, mean ± SD age: 21.1 ± 7.1 years) who completed the HSDQ on two occasions, 1 month apart. For the calculation of the predictive power of the HSDQ, another group of 378 patients (186 females; mean ± SD age: 46.8 ± 14.6 years; 192 males, mean ± SD age: 46.7 ± 13.3 years) was included. They received the following primary diagnoses: insomnia (191), sleep-related breathing disorder (SBD; 73), hypersomnia (25), circadian rhythm sleep disorder (CRSD; 24), parasomnia (23) and sleep-related movement disorders (RLS/PLMD; 42). Control subjects were taken from the same group as used in the questionnaire analyses. In total, 1269 patients and 412 control subjects were enrolled into this study. Written consent was obtained from all participants. The study was approved by the Ethical Committee of the Department of Psychology, University of Amsterdam and conformed to the Declaration of Helsinki.
Questionnaire
In total, 40 items were derived from the descriptions of the main features, predominant symptoms and diagnostic criteria formulated in the ICSD-2 for the six categories of sleep disorders. Each item was cast in the format of a self-description (i.e. ‘I have difficulty falling asleep’) and individuals were instructed to select one of five responses: ‘this applies to me for the past 3 months’: ‘not at all’ (score 1), ‘usually not’ (score 2), ‘sometimes’ (score 3), ‘usually’ (score 4) or ‘completely’ (score 5). Following item, factor and reliability analyses of 300 (first version) and 337 (adapted second version) questionnaires, a stable solution was reached with a total of 32 items.
Analyses
Statistical procedures were performed using the Statistical Package for Social Sciences (spss version 17.0; SPSS Inc., Chicago, IL, USA). To test construct validity, a principal components analysis (PCA) was carried out on the data of 891 patients and 201 control subjects. PCA was followed by a varimax rotation as well as a direct oblimin rotation of the six most important factors. The internal consistency of the HSDQ was checked by calculating Cronbach’s alpha for the entire questionnaire and for its six factors separately. Test–retest reliability was assessed by calculating Pearson’s correlation coefficients for the two data sets of a repeatedly tested, separate group of 211 control subjects. An inter-rater reliability analysis using the kappa statistic was performed to determine consistency between the primary diagnoses and the HSDQ classifications for the 378 patients and 201 control subjects.
Receiver operating characteristic (ROC) curves (Bewick et al., 2004) were used to assess the diagnostic accuracy of the HSDQ, as indexed by P(A), the area under the curve. ROC curves were obtained for the total score (i.e. the mean of all 33 relevant item scores) and for the six individual factor scores, respectively. Following the suggestion by Fresco et al. (2003), we calculated three cutoff scores: optimal sensitivity (the score that optimized sensitivity without reducing specificity to less than chance); optimal specificity (the score that optimized specificity without reducing sensitivity to less than chance); and optimally balanced sensitivity and specificity, also referred to as Youden’s criterion. If the cost of the two types of error (i.e. false positive and false negative classifications) is taken to be equal, then Youden’s criterion is the optimal cutoff.
Results
Construct validity
PCA revealed six factors, represented by 32 items and accounting for 55.7% of the variance. The results of a varimax (orthogonal) rotation of the six factors closely fitted the original intention for the construction of the HSDQ, i.e. to represent the six sleep disorder categories as described in the ICSD-2. Because the six sleep disorder categories can be expected to correlate in varying degrees (e.g. insomnia and CRSD), the six factors were also subjected to a direct oblimin (oblique) rotation, allowing for correlated factors. The results of the two rotations showed almost identical clusters of items for the six factors, which attests to the stability of the factor solution found. If a particular item received more than one factor loading with a value >0.40, that item was assigned to the factor for which it had the largest value. Two exceptions were made, however. The items ‘I lie awake at night for a long time’ and ‘I have difficulty falling asleep in the evening’ were assigned to both the insomnia and the CRSD factors because difficulties initiating and maintaining sleep are included in the diagnostic criteria of both sleep disorders.
Reliability
For HSDQ’s total score a Cronbach’s alpha of 0.90 was obtained, while the coefficients for the six categories of sleep disorders ranged from 0.73 (for SBD) to 0.81 (for CRSD). Pearson’s correlation coefficients for the two test sessions of 211 control subjects ranged from 0.65 (for SBD) to 0.78 (for CRSD) (all P < 0.001), while the overall correlation was 0.79 (P < 0.001).
Diagnostic accuracy
HSDQ’s discriminative ability was assessed in two steps. The first step was to test its ability to discriminate between individuals with sleep disorder(s) and those without any sleep disorder. As specified in Table 1, ROC analysis of HSDQ’s total scores for 378 patients versus 201 control subjects revealed a significant discriminative ability [P(A) = 0.946, P < 0.001; 95% confidence interval: 0.925–0.966]. When Youden’s index was applied (cutoff score = 2.02), 90.5% of the patients and 86.0% of the control subjects were classified correctly, with an overall percentage of 88.0% and a kappa value of 0.75 (P < 0.001).
| Comparisons | ROC: area under the curve | Cutoff values | |||||
|---|---|---|---|---|---|---|---|
| P(A) | SE | 95% Confidence interval | Optimized sensitivity (sens.) | Optimized specificity (spec.) | Youden’s criterion (sens., spec.) | ||
| Lower bound | Upper bound | ||||||
| All patients (n = 378*) versus controls (n = 201) | 0.946** | 0.010 | 0.925 | 0.966 | 1.50 (0.99) | 2.44 (0.98) | 2.02 (0.90, 0.86) |
| Insomnia (n = 173) versus other patients (n = 169) | 0.690** | 0.029 | 0.634 | 0.746 | 3.70 (0.82) | 4.20 (0.69) | 3.68 (0.82, 0.51) |
| Parasomnia (n = 19) versus other patients (n = 323) | 0.946** | 0.022 | 0.902 | 0.990 | 1.47 (0.99) | 3.33 (0.98) | 2.42 (0.90, 0.90) |
| CRSD (n = 21) versus other patients (N = 321) | 0.849** | 0.040 | 0.771 | 0.926 | 2.75 (0.91) | 3.92 (0.90) | 3.41 (0.81, 0.75) |
| Hypersomnia (n = 25) versus other patients (n = 317) | 0.910** | 0.027 | 0.857 | 0.962 | 1.90 (0.96) | 3.85 (0.95) | 2.90 (0.88, 0.82) |
| RLS/PLMD (n = 39) versus other patients (n = 303) | 0.887** | 0.028 | 0.832 | 0.941 | 1.90 (0.97) | 3.60 (0.95) | 2.70 (0.82, 0.77) |
| SBD (n = 65) versus other patients (n = 277) | 0.891** | 0.019 | 0.854 | 0.928 | 2.00 (0.97) | 3.75 (0.93) | 2.87 (0.86, 0.81) |
- CRSD, circadian rhythm sleep disorder; HSDQ, Holland Sleep Disorders Questionnaire; SBD, sleep-related breathing disorder; SE, standard error.
- Cut-off values were calculated for optimal sensitivity (sensitivity value between brackets), optimal specificity (specificity value between brackets) and optimally balanced sensitivity and specificity (Youden’s criterion; sensitivity and specificity values between brackets).
- *The total number of patients exceeds the sum of the six diagnostic groups that were involved in the receiver operating characteristic (ROC) analyses, because the latter do not include the patients (36 in total) who did not meet the Youden criterion for HSDQ’s total score.
- **Different from P(A) = 0.50 (P < 0.001).
In the second step, the data of the patients identified in the first step were used to analyse the ability of the HSDQ to differentiate between the six main categories of sleep disorders. This was tested by six ROC analyses, each applied to the relevant factor scores for the patients with a particular primary sleep disorder versus the corresponding factor scores for the pooled patients diagnosed with other primary sleep disorders. As shown in Table 1, five P(A) values ranged between 0.849 and 0.946, indicating very good discriminating performance. For insomnia, however, a moderate P(A) value of 0.690 (P < 0.001) was obtained, with its 95% confidence interval not overlapping those of the other curves. Applying the appropriate Youden’s criterion to each diagnostic group, the percentages of correctly classified patients ranged between 81.5 and 98%, insomniacs representing the lowest range limit. Overall (including the control subjects), 84.5% of the participants were classified correctly, with a kappa value of 0.80 (P < 0.001) indicating a substantial to excellent agreement between the primary diagnoses and the HSDQ classifications.
Discussion
This report presents a reliable and accurate diagnostic questionnaire, based on the predominant symptoms and diagnostic criteria formulated in the ICSD-2.
As demonstrated by ROC analyses for the six distinct sets of factor scores, the discriminating performance of the insomnia subscale obtained the lowest score (0.69) for the P(A) measure of overall accuracy, whereas for the parasomnia subscale the highest score of 0.95 was observed. Very similar results have been obtained previously, as Roth et al. (2002) reported P(A) values of 0.72 and 0.95, respectively, for the primary insomnia and parasomnia subscales of their Global Sleep Assessment Questionnaire. This relative outlier position of insomnia is not unexpected, however, as its diagnostic criteria overlap to some extent with the criteria for some other sleep disorders, most notably those for CRSD. In addition, insomnia has a high comorbidity with other sleep disorders, as borne out by our finding that large percentages of patients from the other diagnostic groups (53% of the patients with parasomnia, 57% of those with CRSD, 60% of those with hypersomnia, 54% of those with RLS/PLMD and 37% of those with SBD) not only exceeded the HSDQ’s cutoff value for their respective diagnostic category, but also that for insomnia. Apparently, also within the domain of sleep disorders, insomnia can be considered as a transdiagnostic process, i.e. a disorder that is common not only across a range of psychiatric disorders (Borsboom et al., 2011; Harvey, 2008) but also across several, if not all, sleep disorders. In many cases of co-occurrence of disorders it is difficult or impossible to establish the ‘primary’ disorder, by definition making a unique classification impossible. Thus, the moderate discriminating performance of the insomnia subscale does not reflect a weakness of the HSDQ but, instead, emphasizes the common observation that symptoms of insomnia considerably overlap those of other sleep disorders.
The HSDQ is intended for use in a sleep disorders centre to indicate sleep domains which merit further investigation. This questionnaire can also be very useful in case of other disorders [e.g. attention deficit hyperactivity disorder (ADHD), autism, depression, anxiety disorder] known for their frequent association with sleep disorders. In those cases early identification of these comorbidities is of great interest, in particular as it has a bearing on the choice of treatment (e.g. Ohayon, 1997). In addition, the HSDQ suggests itself as a good candidate for use as a comprehensive and practical screening list for sleep disorders in primary care or for screening the general population for common sleep disorders. As only a limited number of screening questionnaires are available, focusing almost exclusively on sleep apnoea (e.g. Baumel et al., 1997), the HSDQ would be an asset to the existing repertoire of screening tools.
As the structure of the upcoming DSM-V will show a closer resemblance to the more detailed ICSD-2, the DSM-V can be expected to provide more guidance to the clinician in formulating the treatment plan in the case of a comorbid sleep disorder. Simultaneously, the revision of the ICSD-2 is under way. As the diagnostic criteria may change substantially, some items of the HSDQ may need to be reformulated, and consequently the questionnaire itself may need revalidation. In that case, the current and the next version of the HSDQ may provide the opportunity to compare their factor structures and to evaluate the impact of the revision.
Conflict of Interest
None of the authors have any conflicts of interest to declare.
