Breaking the Silence: Exploring the Potential Effects of Explicit Instructions on Incorporating Income and Leisure in TTO Exercises
ABSTRACT
Objectives: An underexplored question in the debate on incorporating productivity costs as costs or effects in a cost-effectiveness (CE) analysis is whether people include effects of ill health on income in health state valuations (HSV). The same holds for the actual inclusion in HSV of the effects of ill health on leisure. This study aims to test whether respondents to HSV using time trade-off (TTO) questions include income and leisure effects without instructions. Moreover, it tests the consequences of explicit instructions to include or exclude income effects.
Methods: Three questionnaires were administered among the general public. Respondents were asked to value three distinct EuroQol descriptive system health states using TTO. In version 1, respondents were asked afterwards whether they included income effects. In versions 2 and 3, respondents were instructed upfront to include or exclude income effects. They were furthermore asked whether they included leisure effects.
Results: A total of 222 respondents completed the questionnaire. In version 1, 64% of the respondents spontaneously included income effects, not resulting in differences in mean valuations. In versions 2 and 3, 88% included leisure time, resulting in a significantly lower TTO value in one health state. With explicit instructions, respondents instructed to include income gave lower HSV for the worst health state presented.
Conclusions: Respondents in our sample did not consistently include income effects and leisure effects. Including income effects only had (some) effect on TTO valuations after an explicit instruction. If these results are confirmed in future research, this implies that income effects may be best captured on the cost-side of the CE ratio. Spontaneous inclusion or exclusion of leisure time appeared to be more influential.
Introduction
Productivity costs remain a controversial cost category in the context of health economic evaluations. Not only is their inclusion debated, mostly in relation to the perspective chosen in the evaluation [1], also the correct way to estimate productivity costs is a much debated topic [2–5]. One of the ongoing debates is whether productivity costs are best measured at the cost or the effect side of the cost-effectiveness (CE) ratio in a cost-utility analysis [5–11]. This debate was especially triggered by the publication of the US guidelines on CE in 1996 [10], which specified that productivity costs should not be valued in monetary terms (and captured at the cost side of the CE ratio) as is normally the case. Rather, they were to be included at the effect side of a CE ratio. To ensure this, respondents in health state valuations (HSV) should consider the possible income effects of ill health, which, according to the US guidelines, would be the case as long as respondents are not explicitly instructed to exclude these effects. Although explicit instructions to include income effects could be considered superior, silence on this topic would be sufficient to ensure the incorporation of income effects by respondents in HSV [10]. This recommendation received quite some attention and criticism [5,7,8,11,12]. One of the main objections to measuring the effects of ill health on income through quality-adjusted life-year (QALY) elicitation is that it will not lead to accurate measurement of productivity costs [7,8,11,12]. Moreover, despite the suggestion of the US Panel to instruct respondents to include income effects, all but one of the most popular multi-attribute utility instruments (EuroQol-5D, Health Utilities Index, Quality of Well-being Scale, Rosser and 15D) remain silent on the matter. Only the Health Utilities Index is explicit in this respect and excludes rather than includes these effects [13]. This, by the way, is consistent with the original instructions for use of the time trade-off (TTO) [14]. The silence of the other instruments has been explained as a deliberate attempt to avoid alluding to income rather than an implicit way to include these effects [7]. But still, the current silence in most HSV exercises does not ensure the exclusion or the inclusion of income effects by respondents. This means that to some extent, income effects might be included in HSV, which would result in double counting when productivity costs are also captured at the cost side of the CE ratio. On the other hand, unless all respondents would include correctly estimated income effects in HSV, the approach advocated in the US guidelines will not result in an accurate estimation of productivity costs—even apart from other objections [9].
Unsurprisingly, therefore, it has been suggested that research should be undertaken to find out whether respondents consider income effects of reduced health in HSV when these are silent in this respect, whether this affects their subsequent valuations, and what effect explicit instructions regarding inclusion or exclusion would have on HSV [13]. To date, however, few studies have been conducted in this area. The results seem to indicate that a majority of the respondents do not include income effects in HSV [15–18]. Moreover, the effect of incorporating income effects in HSV appears to be extremely limited [17]. Still, the available evidence is limited, often used a visual analog scale (VAS) as valuation instrument and focused on spontaneous inclusion rather than on (the effect of) instruction. Still, explicit instructions may be considered the appropriate way to prevent either an underestimation or double counting of productivity costs [6,9,11,12]. Given the theoretical debate, two types of explicit instruction appear most relevant. First, one could instruct respondents to incorporate all possible effects of ill health on income in their valuations (in line with the recommendations of the US Panel). Second, one could oppositely instruct respondents to exclude these effects and measure all productivity costs in a monetary value (using the human capital or friction cost method). To our knowledge, the effects of such instructions have been tested only once [17]. Different explicit instructions on including or excluding the effects of ill health on income, or giving no instructions at all, did not lead to significant differences in HSV on a VAS. The latter may have been a limitation in establishing effects of income considerations in HSV because scale compatibility (i.e., the tendency of respondents to give more weight to the subject of focus in the response scale, potentially underweighting other aspects) may result in relative insensitivity of the VAS instrument regarding income effects [17]: the respondents may have been inclined to focus mainly on health itself instead of focusing on other aspects related to ill health.
Although there is debate in this area as well [19], the TTO method currently seems to be the preferred valuation technique for health states rather than standard gamble techniques, VAS, or person trade-offs, and indeed, much used tariffs have been based on TTO elicitations [20,21]. It seems especially relevant therefore to investigate the extent to which respondents consider income changes in TTO exercises and whether the TTO method is equally insensitive as the VAS seems to be to differences in explicit instructions on including or excluding income effects. This article reports on a study investigating this.
Although the main focus is on productivity costs and despite the fact that there is little debate on whether leisure time should be incorporated in the QALY measure [6,9,10,12], it needs noting that the amount of empirical research on whether respondents in fact include leisure time in HSV is equally limited [15,17,18]. If respondents do not consistently include these effects on leisure, even though we expect them to, it is also important to see what effect this has on subsequent HSV. We explicitly highlight the effects of including the negative consequences of ill health for leisure time in TTO valuations in this article.
The structure of this article is as follows. First, we highlight the current knowledge in this area. Then, we present the design of the new study. Next, we highlight the results and subsequently discuss them.
Previous Studies
As mentioned, few empirical studies have been performed to assess respondents' income (and leisure) considerations in HSV. To our knowledge, four studies have been conducted to find out whether respondents include the effects of ill health on income in HSV; the studies of Meltzer et al. [16], Sendi and Brouwer [15], Brouwer et al. [18], and the study of Krol et al. [17]. The first three studies focused on whether respondents include income effects and the latter additionally focused on the effect of instructions on including and excluding these effects. In the studies of Sendi and Brouwer [15], Brouwer et al. [18], and Krol et al. [17], respondents were furthermore asked whether they had included the effects of leisure time in their HSV.
Meltzer et al. [16] tested how giving information about the financial consequences of ill health affects respondents' considerations and valuations using the TTO method. Respondents were asked whether they had thought of financial consequences of illness while answering the TTO questions. Even when respondents were informed that no form of disability payment existed, less than 25% of the respondents included the financial consequences of illness in their valuations. Without any information on income losses, less than 15% included income effects. Respondents who indicated to have (spontaneously) included income, without previous information on income loss, had lower TTO scores than respondents who did not consider income changes.
Sendi and Brouwer [15] performed a small sample test among 20 health professionals to find out whether the effects of ill health on income and leisure time would be included in HSV when no explicit instructions are provided. Respondents were asked to value a disease-specific health state on a VAS. Forty percent of the respondents included the effects of ill health on income, whereas 60% did not. The respondents who included income effects had a significantly lower mean VAS score than the respondents who did not include these effects. In the experiment of Sendi and Brouwer [15], respondents were also asked whether they had considered effects of ill health on leisure time. It turned out that 75% of the respondents indicated to have considered these effects. Although the valuations of respondents who had not included leisure time were higher than the valuations of those who did, the difference was not significant.
Brouwer et al. [18] asked 75 respondents of the general public to value three health states on a VAS. The respondents were asked afterwards whether they had considered income effects and leisure time effects of ill health. Respectively, 31% included income effects and 61% had included leisure time effects. Although income consideration did not result in significant changes in HSV, the incorporation of leisure proved to be influential in the valuation, but only for the most severe health state.
Krol et al. [17] asked 185 respondents of the general public, divided into three groups, to value three health states on a VAS. The first group was asked afterwards whether they had considered income effects in their valuations. Thirty-six percent spontaneously included income. The second group was instructed to consider income effects in their valuations and the third group was instructed to assume that income would not change due to ill health. There were no differences in the valuations of the three groups of respondents. The second and the third group of respondents were furthermore asked whether they had included the effects of ill health on leisure time. Eighty-four percent included these effects. There were no significant differences between the valuations of respondents who included or excluded effects on leisure time.
The results of the above studies seem to indicate that inclusion of income in HSV is inconsistent without explicit instructions. Nevertheless, the results of the studies of Brouwer et al. [18] and Krol et al. [17] suggest that inclusion of income effects does not change HSV when using a VAS as elicitation method. This may reflect that the VAS is rather insensitive in capturing these effects of ill health on income. The results regarding the effects of including leisure time on HSV are inconclusive.
This study aims to further investigate respondents' considerations on income and leisure in HSV using TTO, and additionally focuses on the effects of explicit instructions regarding the inclusion of income.
A New Study
To increase the comparability with previous studies, we drew our study sample in a similar way as was done in the research of Krol et al. [17] and (with exception of the QALY-elicitation method) we used similar questionnaires.
To test the inclusion of income and leisure time as well as the influence of instruction, three different questionnaires were constructed. The study sample was drawn from the general public in various public places (mainly individuals traveling by public transportation). Respondents were approached and asked whether they were willing to participate in this study. If they agreed to participate, the respondents in our convenience sample were randomly handed one of the three different versions of the questionnaires for self-completion. A total of 240 questionnaires were administered.
In all three questionnaires, respondents were first asked some background questions about sex, age, income, education, and their current health state using the EuroQol descriptive system (EQ-5D) and the EuroQol VAS. Next, the respondents were asked to value three ill health states that differed in severity, and were chosen from the 243 possible health state scenarios with the EQ-5D, with the standard Measurement and Valuation of Health group (MVH)_A1 scores of, respectively, 0.88 (health state 21211), 0.587 (health state 22221), and −0.043 (health state 33312) to have a large spread in health states (Table 1).
| Health-state | EQ5D code | MVH-A1* score | Description |
|---|---|---|---|
| 1 | 21211 | 0.880 | Some problems in walking, no problems with self-care, some problems performing usual activities, no pain or discomfort, not anxious or depressed. |
| 2 | 22221 | 0.587 | Some problems in walking, some problems with self-care, some problems performing usual activities, moderate pain or discomfort, not anxious or depressed. |
| 3 | 33312 | −0.043 | Confined to bed, extreme problems with self-care, not able to perform usual activities, no pain or discomfort, moderate anxious or depressed. |
- * MVH, Measurement and Valuation of Health group.
In the study of Krol et al. [17], respondents were asked to value the same three health states on a VAS. Now, respondents were asked to value the health states with the use of the TTO method. The respondents were asked to state how many years of full health they considered to be equally good as living 10 years in the given ill health state. Respondents also had the opportunity to choose not to give up life-years and live the full 10 years in the given ill health state. Respondents were informed that the ill health state would remain the same the full 10 years and that life would end after these 10 years in ill health or the (less than 10) years chosen in full health. Respondents were instructed to make choices considering themselves and reflecting their own opinion.
In version 1 of the questionnaires, respondents did not receive directions on whether to incorporate or to ignore the effects of ill health on income while making the trade-offs. The respondents were asked whether they had included income effects, after the valuation process. If they indicated they had included these effects, they were asked to value the same three health states again, but now with the explicit instruction to ignore the effects of ill health on income by assuming that their income would not change due to ill health. In version 2, people were explicitly instructed upfront to assume that income would not change due to ill health, and thus to ignore the effects of ill health on income. In version 3, respondents were oppositely instructed to incorporate the possible effects of ill health on income. In all questionnaires, respondents were asked whether they thought being in the different health states would decrease their income and if so, to what extent. This was always done after the valuation process in order not to “contaminate” the outcomes.
In versions 2 and 3, respondents were furthermore asked whether they had included the possible effects of ill health on leisure time in their valuations and if they thought the three health states presented would affect their leisure time. The latter aspect was asked because the differences in the extent to which respondents believe that the three ill health states will affect their leisure time can be of influence on their valuations. If, for instance, all respondents expect that the health states will have no effect on their leisure time, it could be expected that including or excluding leisure time considerations in the valuation process will have no effect on subsequent valuations. In version 1 (with the double HSV), the leisure time questions were not included to limit the size of the questionnaire.
The questionnaire was designed to confirm the following hypotheses, based on theoretical debate and the results of previous studies.
- 1
Without receiving directions on including or excluding the effects of ill health on income or leisure time, some respondents to HSV will and some will not include these effects in their valuations [11,15,17,18].
- 2
Respondents of HSV who spontaneously include the effects of ill health on income do not significantly value health states different from respondents who do not automatically include these effects [17,18].
- 3
Respondents of HSV who include the effects of ill health on income without instructions, will value the same health states higher the second time when asked explicitly to exclude these effects [15,17].
- 4
Respondents of HSV who are explicitly asked to include the effects of ill health on income value health states not significantly different from respondents who are explicitly asked to exclude the effects [17].
- 5
Respondents of HSV who include the effects of ill health on leisure time, value health states lower than respondents who do not include these effects [18].
Categorical variables were compared with the chi-square test, ordinal variables with the Mann–Whitney U-test (for two groups) or the Kruskal–Wallis test (for more than two groups). Continuous variables are compared with the Student's t test (two groups) and with one-way analysis of variance (two or more groups). Simple linear regression models and linear mixed effects (LME) models were used to estimate the association between the valuations of the health states and other variables such as the version of the questionnaire administered and characteristics of the respondents as age, sex, and income. The data were analyzed with use of SPSS Statistical Software Package 13.0 (SPSS, Chicago, IL) and S-Plus 6.2 Professional (Insightful S-Plus, Seattle, WA).
Results
Sample Population
A total of 222 respondents completed the questionnaire. Eighteen of the administered questionnaires were not returned or returned blank. Twelve questionnaires were excluded from further analysis based on comments of the respondents that they did not understand the TTO method or based on evidence that instructions were not followed (e.g., respondents who did not trade-off healthy years, but instead wanted more than 10 years in full health in return). A total of 210 questionnaires were used for further analysis (version 1; n = 72, version 2; n = 75, version 3; n = 63).
As seen in Table 2, the average age of the respondents was 34.77 years and did not differ between the versions of the questionnaire. Fifty percent of the respondents were female. The average self-reported health of the respondents was 0.82 reported on a VAS. Higher educated people and those with higher incomes were overrepresented in our sample. The respondents with higher incomes were significantly older (P < 0.001). The respondents of the different versions of the questionnaire did not differ in income, education, sex, employment, or self-reported health.
| Version 1 (n = 72) | Version 2 (n = 75) | Version 3 (n = 63) | All respondents | |
|---|---|---|---|---|
| Females | 0.54 | 0.44 | 0.54 | 0.50 |
| Age | 36.03 (14.43) | 35.12 (14.43) | 32.94 (14.17) | 34.77 (14.34) |
| Employment | ||||
| No paid work | 0.36 | 0.23 | 0.29 | 0.29 |
| Full-time employment | 0.31 | 0.43 | 0.40 | 0.38 |
| Part-time employment | 0.33 | 0.34 | 0.31 | 0.33 |
| Education | ||||
| Lower education | 0.11 | 0.13 | 0.21 | 0.15 |
| Medium education | 0.39 | 0.39 | 0.41 | 0.40 |
| Higher education | 0.50 | 0.48 | 0.38 | 0.46 |
| Income | ||||
| Lower income | 0.26 | 0.21 | 0.25 | 0.24 |
| Medium income | 0.26 | 0.34 | 0.33 | 0.35 |
| Higher income | 0.47 | 0.45 | 0.41 | 0.45 |
| Self-reported health (VAS) | 0.82 (0.14) | 0.83 (0.12) | 0.82 (0.15) | 0.82 (0.14) |
- VAS, visual analog scale.
Effects of Ill Health on Income
As seen in Table 3, 28% of the respondents stated that ill health state 1 (the “mildest” state) would decrease their income. Respectively, 71% and 89% expected income loss in health state 2 or 3. Although especially for less severe health states explicit instructions may have induced more respondents to expect income changes, these differences were never significant at the 5% confidence level. Respondents who thought that the health states would decrease their income were younger than those who did not expect income influence (P < 0.01 for all health states). Respondents who stated that health state 2 or health state 3 would not change their income were more often unemployed (P < 0.01). Respondents who stated that health state 3 would decrease their income were better educated (P = 0.05) as opposed to those who did not.
| Health-state 1 decreases income | Health-state 2 decreases income | Health-state 3 decreases income | Included income | Included leisure-time | |
|---|---|---|---|---|---|
| Version 1 (n = 72) | 0.21 | 0.61 | 0.88 | 0.64 | — |
| Version 2 (n = 75) | 0.32 | 0.76 | 0.91 | — | 0.90 |
| Version 3 (n = 63) | 0.32 | 0.77 | 0.86 | — | 0.85 |
| Mean | 0.28 | 0.71 | 0.89 | — | 0.88 |
| Equality of means* | 0.228 | 0.077 | 0.758 | — | 0.371 |
| Expected income loss (€) | 284 (205) | 421 (330) | 787 (461) | — | — |
- * chi-square test.
The respondents who expected income loss to occur expected on average that health state 1 would lead to a loss of 284 euros per month, while this amount was 421 and 787 for health states 2 and 3, respectively. We used regression techniques to see which variables are associated with expected income loss. The valuations of respondents were not significantly associated with the expected amount of income loss. A higher income was associated with a higher expected income loss, but only in the worst health state 3 (P = 0.05). Other demographic variables such as age, sex, and education turned out not to be significant for all the three health states.
The Effects of Instructions
No instructions. Confirming our hypothesis [1] that without instructions, some respondents will and some respondents will not include the effects of ill health on income in HSV, 64% of the respondents of version 1 automatically included the possible effects of ill health on income in their HSV and 36% did not. This percentage of inclusion is higher than those found in previous studies, where it never exceeded 40% [15–18].
The respondents who automatically included the effects of ill health on income did not differ from those who did not, in education, income, employment, sex, age, or self-reported health.
Although Meltzer et al. [16] and Sendi and Brouwer [15] report differences in HSV based on different considerations about negative effects of ill health on income (assessed without explicit instruction), our overall results do not show a significant influence of including or excluding of the effects of ill health on income without instructions on the subject (see Table 4). These findings support the hypothesis [2] that respondents of HSV who spontaneously include the effects of ill health on income do not significantly value health states differently than respondents who do not spontaneously include these effects. These findings are similar to the findings in the studies of Krol et al. [17] and Brouwer et al. [18].
| Valuations | |||||||
|---|---|---|---|---|---|---|---|
| Version | Group | HS1 | n | HS2 | n | HS3 | n |
| The effects of instructions | |||||||
| 1 | All (no instructions) | 0.88 | 72 | 0.72 | 72 | 0.39 | 71 |
| 2 | All (excluding income) | 0.91 | 74 | 0.74 | 74 | 0.45 | 73 |
| 3 | All (including income) | 0.91 | 62 | 0.72 | 62 | 0.38 | 60 |
| P-value* (1 vs. 2 vs. 3) | 0.329 | 0.820 | 0.279 | ||||
| P-value† (1 vs. 2) | 0.201 | 0.537 | 0.188 | ||||
| P-value† (1 vs. 3) | 0.209 | 0.867 | 0.911 | ||||
| P-value† (2 vs. 3) | 0.985 | 0.676 | 0.181 | ||||
| Spontaneous inclusion/ exclusion of income | |||||||
| 1 | Including income | 0.89 | 45 | 0.72 | 45 | 0.41 | 45 |
| 1 | Excluding income | 0.86 | 25 | 0.70 | 25 | 0.35 | 24 |
| P-value† | 0.494 | 0.752 | 0.364 | ||||
| Excluding income after spontaneous inclusion of income | |||||||
| 1 | 1st valuation (+ income) | 0.88 | 34 | 0.73 | 34 | 0.39 | 34 |
| 1 | 2nd valuation (− income) | 0.87 | 34 | 0.75 | 34 | 0.42 | 34 |
| P-value‡ | 0.343 | 0.226 | 0.189 | ||||
| Income affected and income included/excluded | |||||||
| 1 | Income affected§ | 0.79 | 14 | 0.70 | 41 | 0.40 | 60 |
| 1 | Income not affected | 0.90 | 56 | 0.74 | 28 | 0.34 | 9 |
| P-value† | 0.008 | 0.498 | 0.508 | ||||
| 1 | Income included + affected§ | 0.78 | 6 | 0.71 | 24 | 0.42 | 40 |
| 1 | Income excluded + affected§ | 0.79 | 8 | 0.70 | 17 | 0.38 | 20 |
| P-value† | 0.955 | 0.908 | 0.579 | ||||
| 2 | Income affected§ | 0.93 | 23 | 0.72 | 56 | 0.44 | 64 |
| 3 | Income affected§ | 0.87 | 19 | 0.68 | 43 | 0.34 | 50 |
| P-value† | 0.173 | 0.433 | 0.055 | ||||
| Leisure-time (LT) affected and leisure-time included/excluded | |||||||
| 2 & 3 | Including LT | 0.90 | 113 | 0.71 | 113 | 0.39 | 110 |
| 2 & 3 | Excluding LT | 0.94 | 16 | 0.84 | 16 | 0.48 | 16 |
| P-value† | 0.242 | 0.037 | 0.235 | ||||
| 2 & 3 | LT affected¶ | 0.88 | 85 | 0.70 | 113 | 0.40 | 122 |
| 2 & 3 | LT not affected | 0.96 | 45 | 0.86 | 14 | 0.60 | 5 |
| P-value† | 0.000 | 0.021 | 0.123 | ||||
| 2 & 3 | LT included + LT affected¶ | 0.88 | 74 | 0.69 | 97 | 0.38 | 103 |
| 2 & 3 | LT excluded + LT affected¶ | 0.89 | 8 | 0.81 | 13 | 0.45 | 15 |
| P-value† | 0.866 | 0.091 | 0.387 | ||||
- * One-way ANOVA.
- † Student's t test for equality of means.
- ‡ paired sample Student's t test.
- § The valuations of respondents that think the health-state will influence income. The valuations of respondents who do not think the particular health-state will affect income were removed.
- ¶ The valuations of respondents that think the health-state will influence LT. The valuations of respondents who do not think the particular health-state will affect leisure-time were removed.
- P = 0.05 are indicated in bold.
In Table 4, the average TTO valuations of the three health states of the respondents of the three different questionnaires are presented. Our respondents valued the health states higher than the corresponding MVH_A1 scores: 0.90 versus 0.88, 0.73 versus 0.59, and 0.41 versus −0.04 for health state 1 to 3, respectively. The respondents in our sample were reluctant to trade-off life-years: 58.2%, 30.8%, and 12.0% of the respondents were unwilling to trade-off life-years for health states 1 to 3, respectively. There were no differences between the versions in percentages of respondents who would not make the trade-off. Respondents who would not give up time were significantly older than those who would in all the three health states. Furthermore, the respondents who would not give up years in health states 1 and 2 had significantly higher incomes than those who did make the trade-off. This latter difference was not found in health state 3.
Instruction after silence. As mentioned, the respondents of version 1 who included income in their valuations were asked to valuate the same health states again, now assuming these health states would not affect their income. Although other studies [15,17,18] already show that a part of the respondents does not react to a change in instructions by changing their valuations, it was expected that overall, the respondents would value the same health states significantly higher the second time.
As seen in Table 4, we found no significant differences between the first and second valuations of the respondents. These findings are similar to the findings of Brouwer et al. [18] and do not support the hypothesis [3] that respondents change their valuations if they are asked to ignore a negative aspect of ill health they had included before.
Explicit instructions. Table 4 shows that the respondents of version 2 (who were explicitly asked to exclude the effect of ill health on income), did not value the health states significantly higher than the respondents of version 3 (who are explicitly asked to include the effects), similar to the findings of Krol et al. [17]. The valuations of the respondents without instruction (version 1) moreover did not differ from those in the other two groups. These results, thus, seem to imply that instruction does not matter in the sense that it does not change the mean TTO valuations. Only considering respondents (in versions 2 and 3) who actually expected the presented health state to affect income in our analyses, regardless of the instruction given, increases the differences in mean valuations of the respondents in the expected direction (respondents of version 2 give higher values to the health states). These differences, however, are still insignificant, as shown in Table 4.
Multivariate analysis. A linear regression model was used to estimate the association between the HSV of all respondents and the version administered and whether respondents expect a health state to affect income, regardless of whether income effects were included (interaction term included), corrected for age, sex, income, and self-reported health.
As shown in Table 5, income had a significant positive association with the valuations of health states 1 and 2. In health state 1, the interaction term of the version of the questionnaire and whether respondents expect that the health state would actually decrease income turned out to be significant. Respondents who were handed version 1 or version 3 of the questionnaire and expected the relevant health state to decrease income therefore valued that health state lower than the respondents in version 2. These interaction terms were not significant for the valuation of health state 2; however, for health state 3 the interaction term of version 3 and expecting income to be affected was again significant. These results imply that the different instructions do seem to affect respondents' valuations, but only if respondents expect that the health states presented would actually decrease their income. Note that the linear regressions have little explanatory power.
| Valuation HS 1. | Valuation HS 2. | Valuation HS 3. | ||||
|---|---|---|---|---|---|---|
| Beta | P | Beta | P | Beta | P | |
| Constant | 8.057 | 0.000 | 6,403 | 0.000 | 2.445 | 0.174 |
| Sex (male = 1, female = 0) | −0.005 | 0.544 | 0.020 | 0.202 | 0.024 | 0.207 |
| Age | −0.235 | 0.240 | −0.661 | 0.052 | −0.704 | 0.080 |
| Net income | 0.232 | 0.004† | 0.374 | 0.007† | 0.089 | 0.578 |
| Self-reported health | 0.557 | 0.438 | −0.339 | 0.779 | 1.533 | 0.288 |
| Version 1* (dummy) | 0.051 | 0.854 | −0.457 | 0.527 | −1.489 | 0.291 |
| Version 3* (dummy) | 0.421 | 0.153 | 0.743 | 0.387 | 2.552 | 0.094 |
| Does HS 1, HS 2, HS 3 affect income (1 = yes, 0 = no) | 0.405 | 0.255 | −0.378 | 0.574 | −0.053 | 0.964 |
| Dummy version1*income affect HS 1, 2, 3 | −1.691 | 0.003† | 0.093 | 0.915 | 1.101 | 0.461 |
| Dummy version3*income affect HS 1, 2, 3 | −1.063 | 0.043† | −1.256 | 0.201 | −3.594 | 0.027† |
| Adjusted R2 | 0.094 | 0.098 | 0.094 | |||
- * Version 2 = excluded variable.
- † P < 0.05.
Next, we focused on the effects of explicit instructions (as given in versions 2 and 3 of the questionnaire). To that end, an LME model was used and respondents of version 1 of the questionnaire were excluded from the analysis. The LME model was used to estimate the association between the valuation of the health states and explicit instructions, stratified by whether respondents thought income would be affected by the respective health state, adjusted for age and sex. Respondents valuing the three health states were treated as random effects to account for within respondent variability and correlation using an unstructured correlation matrix.
Because the interaction term “version*income-affected*health-state-valuations” in the LME model turned out to be significant (P = 0.048), the effect of explicit instruction on HSV was analyzed separately in each subgroup of respondents who expected income to be affected in, respectively, health states 1, 2, or 3. A linear regression model was used to separately test whether the valuations of the three health states were affected by the explicit instructions in versions 2 and 3. Covariates were added to the respective univariate model to improve the estimate and precision of the effect estimate in case the univariate model showed a P-value of 0.1 or less. Furthermore, in all models, a multivariate model was also used as a starting point and the model was then simplified to assess the effect of including/excluding covariates to the respective model on the main effect estimate of interest. Because the leisure time questions were only asked in versions 2 and 3, respectively, they are also included in the multivariate model. The multivariate model included the variables age, sex, consideration of income and leisure in HSV, and own VAS.
As shown in Table 6, in the subgroup expecting the health state to affect their income, the explicit instruction variable was significantly associated with the valuation of health state 3, with lower scores for respondents including income. Also the variables sex, expecting the health state to affect leisure, and self-reported health were significantly associated with the valuation of health state 3.
| HS | Income affected | variable | Univariate model | Multivariate model* | ||||
|---|---|---|---|---|---|---|---|---|
| Coefficient | SE | P-value | Coefficient | SE | P-value | |||
| 1 | Yes | Version 3 vs. 2 | −0.6201 | 0.4465 | 0.1726 | |||
| 1 | No | 0.4271 | 0.2869 | 0.1403 | ||||
| 2 | Yes | −0.3871 | 0.4918 | 0.4331 | ||||
| 2 | No | 0.3942 | 0.8655 | 0.6524 | ||||
| 3 | Yes | Version 3 vs. 2 | −0.9750 | 0.5216 | 0.0642 | −1.0621 | 0.4995 | 0.0358† |
| Sex | −1.4789 | 0.4986 | 0.0037† | |||||
| Leisure affected | −6.5067 | 2.6061 | 0.0141† | |||||
| Self-reported health | 4.1158 | 2.0523 | 0.0474† | |||||
| 3 | No | Version 3 v 2 | 1.7381 | 1.9602 | 0.3942 | |||
- * R 2 0.16, P-value of multivariate model = 0.0009.
- † P < 0.05.
The results of the subgroup analysis of versions 2 and 3 imply, similar to the analyses of the three versions together, that explicit instructions may have an effect on valuations of health states, however only in respondents who believe their income will be affected by this health state. In other cases, however, explicit instruction did not lead to changes in valuations. This means that the findings of Krol et al. [17] are largely but not completely reproduced in this study and that hypothesis [4] is for most situations, but not for all, confirmed. The picture is less uniform than in previous studies therefore.
Leisure-Time
Eighty-eight percent of the respondents of versions 2 and 3 included the effects of ill health on leisure time while trading-off time versus health. Even though respondents are assumed to include leisure time [9,10,12], 12% of the respondents in our convenience sample did not. This supports hypothesis 1, that without instructions, some respondents will and some will not include these effects. The percentage of respondents who included leisure time is comparable with the 84% of the study of Krol et al. [17]. It is, however, higher than the percentage found in previous studies by Sendi and Brouwer [15] (75%) and Brouwer et al. (61%) [18].
The 88% of the respondents who included leisure time in their valuations had a lower income than the 12% who did not include leisure time (P = 0.03). As shown in Table 4, respondents including leisure time gave lower TTO values to the three health states than respondents who had not included leisure time. The differences, however, were only significant for health state 2. These results can therefore not completely support the hypothesis [5] that respondents who include the effects of leisure time will value the health states lower than those who do not include these effects. These findings are similar to the findings of Brouwer et al. [18], who also found significant differences for only one health state (the worst health state in that case).
As mentioned, we asked the respondents whether they expected the three health states presented to affect their leisure time. Sixty-five percent, 89%, and 96% of the respondents believed that, respectively, health states 1 to 3 would affect their leisure time. We excluded the respondents who thought that the health states would not affect their leisure time. Next, we tested again whether including or excluding leisure time leads to differences in valuations to see if the differences would have increased. Opposite from what we expected, the differences between the valuations of respondents including and excluding income decreased after correcting for thinking a health state would or would not affect leisure time, as shown in Table 4. The correction causes the difference earlier found in health state 2 to lose its significance. Interestingly, whether a respondent thinks the health state will affect leisure time is (regardless of whether a respondent included or excluded leisure time effects) of significant influence on respondents' valuations for health state 1 and health state 2, though not for health state 3. The latter result, also given that the difference between the groups grows with the severity of the health state, is probably due to lack of power given the low number of respondents thinking that leisure would not be affected. Indeed, in the study of Brouwer et al. [18], the fact that leisure was affected was most influential as well, although, after correction for other variables, it was only for the most severe health state that a significant difference was found between respondents who believed leisure to be affected and those who did not (in the expected direction).
Discussion and Conclusion
This article has reported on the first study aimed at investigating the effects of explicit instructions regarding the inclusion or exclusion of income and leisure in HSV using TTO.
In this study, five hypotheses were sought to be confirmed. In line with the results from previous studies, our study confirmed the first hypothesis that without instructions, respondents do not consistently include or exclude the effects of ill health on income. Given the high inclusion of income in this study relative to previous studies, the TTO method may be more sensitive to inclusion of income effects than the VAS. Scale compatibility may be an explanation for this [22]. On the other hand, Meltzer et al. [16] used TTO questions (and even triggered respondents to consider income effects by giving specific information about disability payments) but report lower percentages of inclusion of income by respondents. Differences in sample population, country, and methodology may have contributed to these differences.
Our second hypothesis was also confirmed. The respondents spontaneously including the effects of ill health on income did not value the three health states significantly different when compared to respondents not including these effects. Surprisingly, for all three health states, respondents including income gave higher valuations, although these differences were insignificant.
Hypothesis 3 was not confirmed. Respondents who, without instructions, included income effects did not value the three health states significantly higher the second time when asked explicitly to exclude these effects. Nevertheless, the valuations of the more severe health states 2 and 3, where potential effects of differences in including or excluding income effects would be strongest, did change in the expected direction.
Results regarding the fourth hypothesis were mixed. In most cases, explicit instructions on including or excluding income had no significant effect on HSV; although the group instructed to include income elicited lower valuations of health states 2 and 3. The effect of explicit instruction on the HSV was only significant in some cases where respondents actually expected an income effect.
For the final hypothesis, results were also mixed. Respondents spontaneously including leisure time effects gave lower HSV than respondents excluding leisure. The differences in valuation were however only significant for one of the three health states.
Before discussing the implications, some important limitations of this study need to be stressed. First of all, our study is based on a relatively small convenience sample of respondents. Some of the differences in HSV between groups may have been insignificant because of lack of statistical power, especially in cases where differences in the expected direction were observed but did not reach conventional levels of significance (e.g., for hypotheses 3, 4, and 5). Therefore, repeating this study in a larger sample remains important. Using a wider variation in valued health states in such studies may also be informative. Second, the format of our TTO questions might have lead some respondents to believe that the minimum trade-off they could make was 1 year. The respondents indeed did only trade-off whole or half years. This may have caused respondents willing to trade-off some time, but less than a year, to choose to not give up years at all. The respondents in our sample indeed were reluctant to trade-off life-years, although the reluctance to trade-off life-years is common [23]. Potential bias caused by our TTO framing would be strongest in the mildest health state. It is however not expected to affect potential differences in valuations between the three health states and our focus was not on finding the “right” TTO values, but on finding the effects of instructions on income in HSV. Third, the results of our study may be country-specific. In countries with a less comprehensive social security system than The Netherlands, respondents might be more inclined to consider income and ill health may affect income more strongly, leading to more respondents including the effects of ill health on income or perhaps even to significant differences in HSV between respondents including and excluding income effects. This could be investigated further. A fourth limitation of our study is that we did not ask the respondents afterwards whether they acted according to the instructions given. It could be that a part of the respondents ignored the instructions. Furthermore, it is possible that through the instruction to ignore possible income effects, respondents are actually triggered to do think of income (“the do not think of a pink-elephant effect”[9]). It is also possible that any explicit instruction places too much emphasis on income, causing respondents to overweigh possible income effects in the HSV. This forms a potential bias if we, in daily practice, consider instructing respondents to HSV on including or excluding income. Finally, we asked the respondents the extent to which they believed that the ill health states would affect their income. It, however, remains questionable whether respondents can make realistic estimations of income losses due to illness [18]. This idea is strengthened by the high number of missing values when a specific amount of expected income loss is asked (missing values health state 1: 35%, health state 2: 33%, and health state 3: 40%). One way to circumvent this lack of knowledge is to include explicit information on reasonable estimates of income losses due to certain health states. A similar procedure was followed by Meltzer et al. [16].
In conclusion, our results are generally in line with earlier studies and indicate that silence regarding inclusion of income and leisure does not ensure consistency in this respect in HSV. Some respondents will include these effects while others exclude them. Moreover, the extent to which respondents expect health states to affect their leisure and income varies substantially across respondents. Still, as found in some previous studies, spontaneous inclusion or exclusion of income effects did not result in different valuations of health states. The influence of explicit instructions regarding inclusion of income effects appears to be limited, but still, in contrast to earlier findings does appear to matter. Because we do find some effects of including income after an explicit instruction on the matter, the TTO method may be more sensitive in picking up income effects than the VAS. Especially when respondents believe that income is indeed affected (which is more often the case in severe health states), explicit instruction to exclude income may lead to higher valuations. For leisure, the percentage of respondents including leisure time was relatively high in our study. Still, like in other studies, a substantial part of the respondents do not include leisure time in their valuations although they are normally expected to do so. We find no clear evidence that inclusion as such leads to differences in valuations. Expecting leisure to be affected seems more influential than the indicated inclusion of leisure in HSV, which may indicate simply a difference in perception of the severity of the health states between respondents.
What do these results imply then for productivity costs and leisure? First of all, it is important to realize that, given the limitations of our study and the previous studies conducted, it is premature to draw definite conclusions. There is a clear need for more research in this area using larger samples to further explore this important topic.
Notwithstanding this, it is clear that currently, most HSV methods include no instructions to the respondent regarding the inclusion or exclusion of income effects. In this case, respondents spontaneously include or exclude income. As we have seen, spontaneous inclusion or exclusion of income changes does not appear to result in noticeable differences in HSV even when such changes are expected to be large. If this finding is confirmed in future research, this casts empirical doubts on recommendations to include income changes in health-related quality of life, next to theoretical objections one may have against it (e.g., 8). In that case, it would imply that it is best to include productivity costs on the cost-side of the CE ratio, where they do have a measurable effect. If one actively wants to preclude income effects in HSV, explicit instructions to exclude income changes due to ill health may be considered appropriate. It must be noted, however, that such instructions may involve potential biases, like overweighting income, due to an active, although negative, emphasis on income considerations. Rather than telling respondents what not to consider, and thus risking that they will consider it, it may be better to tell them what to consider; i.e., they should consider that they are covered by full health insurance and salary continuation insurance [14]. More research here would clearly be useful. For now, silence on the matter may be considered the best solution, because it is unclear whether a move away from silence, and what kind of a move, is for the better or the worse.
Although in all the empirical studies on leisure time in HSV (including ours) small samples were used, it seems clear that most people include this in HSV. Not all respondents, however, automatically include these effects, and therefore, it may be necessary to explicitly instruct respondents to include these effects. But even then, more research in this area seems worthwhile because it remains unclear whether including leisure results in an adequate valuation of lost leisure. Silence may not be golden therefore, but how to break it best remains open for debate.
We are grateful to the anonymous reviewers for their useful comments on earlier drafts of this article. The usual disclaimer applies.
Source of financial support: None.
