Performance Comparison of Likert and Binary Formats of SF-36 Version 1.6 Across ECRHS II Adults Populations

Introduction

Among different questionnaires for assessing health status, the 36-item Short-Form Health Survey (SF-36) is probably the most widely used. It was developed more than a decade ago, and translated into about 50 languages successively [1–6]. SF-36 has proven useful in comparing general and specific population, measuring the health deficit of different diseases, or the health benefits of different treatments, and for screening individual patients. It is able to synthesize eight concepts with only 36 questions, and it is structured on three levels: items, multi-items scales, and scale clusters. Twenty-eight items are in ordinal form following the Likert format, seven items are in binary form, and a further item concerns reported health transition over the past year. Four multi-items scales regard the physical health concepts (physical functioning—PF, role limitations due to physical health problems—RP, bodily pain—BP, general health—GH); other four multi-items scales concern the mental health concepts (vitality—VT, social functioning—SF, role limitations due to personal or emotional problems—RE, and mental health perceptions—MH). Two summary measures of the SF-36 are referred to as physical component summary 36 (PCS-36) and as mental component summary 36 (MCS-36) scales. During years, SF-36 has been revised, rectified, and proposed in more specific and synthetic versions (SF-36v2^TM, SF-12®, SF-12v2^TM, SF-8^TM, cf. http://www.sf-36.org), finding a better application in clinical trials.

The SF-36 scales (in all versions) are scored using Likert's method of summated ratings [7]. In this method, a number (weight) is assigned to each item response category, usually 1, 2, 3 for low, middle, high responses, respectively, or 1, 2, 3, 4, 5 for excellent, very good, good, fair, poor responses, respectively, and so on; weights for some items need to be recalibratedso that all items are weighted in the same direction. The scale score is then computed by summing the weights assigned to item responses and by linearly transforming scores to 0–100, where higher scores represent a better level of quality of life (QoL). Summated rating scales have the advantage of simplicity, and can achieve high levels of reliability and validity.

Nevertheless, the item weighting presupposes the equidistance (linearity) between numerical response labels assigned to Likert points. It is a well-known fact that the nonmetric proprieties of ordered categorical data imply that such arithmetic operations are not strictly appropriate for ordinal data [8]. The number of response categories also precludes the chance to administer the questionnaire in an easy and fast way (i.e., telephone research and very ill patients) [9,10], setting a burden on both patients and investigators. A solution may be to recode the 28 items of SF-36 questionnaire from Likert to binary, so to reduce all SF-36 items in binary format.

The European Community Respiratory Health Survey (ECRHS) [11,12] is an international longitudinal population-based study of more than 10,000 young adults, initially aged between 20 and 44 years from 1991 to 1993, randomly selected, and followed up 9 years later using the same standardized protocol, in all centers. The ECRHS was carried out in response to the rapid increase in asthma prevalence, which pointed to environmental factors being important in the development of the disease. At Follow-up, validated QoL questionnaires, including the SF-36, were administered. Although the primary aim of the Quality of Life Working Group is to appraise the burden of allergic conditions across Europe, in large populations with different asthma prevalence and risk factors, the settings of the study makes the ECRHS a unique opportunity for developing methodological work in the field of QoL assessment.

We therefore performed the present analysis based on the ECRHS data to investigate the following specific aims: 1) to aggregate ordinal responses in binary ones, using an approach that takes account of the nonmetric proprieties of the SF-36 data; 2) to evaluate the overall performance of the SF-36 binary recoding, regarding reliability and validity, scaling assumptions by ordered correlation/regression methods; and 3) to compare the equivalence of Likert and binary rescaled SF-36 questionnaire outcomes, and their implications for interpreting research findings.

Methods

Likert versus Binary

Likert scales and binary recordings of the original scales were compared to evaluate their equivalence regarding overall performance of the scoring system, scaling assumptions, reliability, and validity. Descriptive statistics were computed to characterize score distributions, and the percentage of responses on anchor points (extremes) was examined for each scale to detect floor or ceiling effects. Spearman–Brown formula of the Cronbach's alpha coefficient was used to determine the reliability/internal consistency of the scales [16]. High internal consistency has been suggested if alpha > 0.70 for group comparison, and alpha > 0.90 for individual comparison. To explore the heterogeneity/homogeneity among the Likert and binary forms of the SF-36 items across the 25 ECRHS centers (average cluster size was 354), the intraclass correlation coefficients (ICC) were also computed [17].

To check correlations across Likert, binary, and Likert-binary sets, the Pearson's correlation coefficients, polychoric correlations coefficients, and biserial correlations coefficients [18] were computed, respectively. Further, to identify the two health components (physical, mental) underlying the SF-36 questionnaire, both Pearson's correlation matrix (for Likert form scales) and polychoric correlations matrix (for binary form scales) were explored by means of PCA. The number of components was determined on the basis of eigenvalues of the correlation matrix greater than 1, and by looking for sharp breaks in the size of the eingenvalues using a scree plot [19]. Varimax rotation, and scale–component correlations, i.e., “factor loading,” greater than 0.40 in absolute value was chosen to identify a simple component structure, i.e., component with nonoverlapping clusters of SF-36 scales.

To get a single number describing the relationship between Likert and binary matrices output, we used the vector correlation coefficient (RV), a generalization of the Pearson determination coefficient, R²[20]. As the determination coefficient, also RV is bounded between 0 and 1, and Good, Strong, Excellent agreement between the two matrices has been suggested if RV > 0.50, RV > 0.70, RV > 0.90, respectively.

Finally, clinical (criterion-based) validity of the Likert scales and the binary recoding scales were assessed by means of know-groups comparison. Subjects were assigned to mutually exclusive groups differing in self-reported asthma-like symptoms, long-term limiting illness, and depression conditions. It was expected that physical component scales (PF, RP, BP) profiles would score worse in groups with asthma like symptoms or long-term limiting illness; that mental component scales (MH, SF, RE) profiles would score worse in groups with depression; and that GH and VT profiles score worse on groups with all negative conditions. Two-level random intercept regression models [21] with level 1 units (subjects) nested in level 2 units (ECRHS II centers) were fitted to rank the group differences on the SF-36 scales. Considering Likert scales and PF binary recoded scale as continuous responses, linear regression models were fitted. Vice versa, for other binary recoded scales linear or ordered logistic regression models were fitted, considering the binary forms scales as continuous or ordinal responses, respectively. The P-values of the regression parameter estimates were evaluated by t-test (= parameter/standard error) using robust standard errors; the significance level was set at P < 0.05, two-sided.

Descriptive data analyses and MCA were performed using SPSS software, version 11.5 (SPSS Inc., Chicago, IL, USA: http://www.spss.com), while polychoric/polyzerial correlations and two-level regression models were provided by Mplus software, version 3.13 (Muthén & Muthén, Los Angeles, CA, USA: http://www.statmodel.com).

Results

Binary Recoding

The internal criterion index (Guttman's eta) of the MCA scaling ranged from 0.56 (GH) to 0.88 (BP), while the Cronbach's alpha coefficient varied from 0.80 (GH; and SF) to 0.92 (PF), indicating an excellent optimal scaling.

The transformation plots of the optimal weights for each Likert scale are displayed in Figure 1. Generally, PF items seemed to describe a straight line, even if the optimal values of the second and the third answers of A and D items had a similar weight; the A and C items of the GH scale seemed to describe better a curve than a straight line having equal weights in the first and second answers, besides the fourth answer of the B item had an higher optimal weight than the fifth one; in VT scale the item points described better a double curve not respecting the linearity; in SF scale the first A item answer had a value lower than the second one; and finally, no one item described a straight line in MH scale. Therefore, only in BP scale the equidistance assumption was respected, while in the other scales this assumption was rejected.

The MCA optimal weights (normal type) and the binary recoding (bold type) of the SF-36 questionnaire are summarized in Table 2. Columns show the several Likert options, while rows show the different item number for each scale. Negative (positive) optimal weight values were recoded as 0 (1) for PF, BP, VT, and MH scales, while there was an opposite recoding for GH and SF scales. For example, the first answers of each PF scale item had always negative values, thus these were recoded with 0, while the third answers, being positive for the 10 items, were recoded as 1. The recode 0 represents the tendency toward a low QoL level, and vice versa 1 represents high QoL level.

Table 2. MCA optimal weights and binary (0/1) recoding of the SF-36 Likert format items

Binary recoding	Items	Options
		1		2		3		4		5		6
PF	a	1.63	0	−0.02	1	−0.35	1
	b	4.02	0	1.17	0	−0.28	1
	c	3.78	0	1.05	0	−0.27	1
	d	3.24	0	0.38	0	−0.31	1
	e	4.99	0	2.04	0	−0.21	1
	f	3.19	0	0.73	0	−0.27	1
	g	4.02	0	1.01	0	−0.26	1
	h	5.12	0	2.14	0	−0.21	1
	i	5.41	0	3.27	0	−0.15	1
	j	4.90	0	2.58	0	−0.14	1
BP	a	−0.74	1	−0.29	1	0.28	0	1.12	0	2.46	0	3.20	0
	b	−0.59	1	0.48	0	1.45	0	2.55	0	3.33	0
GH	a	1.70	0	1.56	0	1.13	0	0.46	0	−0.51	1
	b	−0.86	1	0.12	0	0.89	0	1.43	0	0.62	0
	c	0.65	0	0.77	0	0.53	0	0.09	0	−0.75	1
	d	−1.11	1	−0.10	1	0.90	0	1.47	0	1.58	0
	x	−1.17	1	−0.50	1	0.35	0	1.61	0	2.21	0
VT	a	−1.10	1	−0.60	1	0.02	0	0.74	0	1.78	0	1.98	0
	e	−1.12	1	−0.71	1	−0.17	1	0.52	0	1.42	0	1.99	0
	g	2.45	0	1.65	0	0.91	0	−0.02	1	−0.66	1	−1.13	1
	i	2.51	0	1.99	0	1.38	0	0.36	0	−0.37	1	−0.76	1
SF	a	−0.59	1	0.78	0	1.67	0	2.41	0	2.59	0
	b	1.54	0	2.34	0	1.47	0	0.36	0	−0.67	1
MH	b	2.01	0	1.97	0	1.40	0	0.53	0	−0.19	1	−0.68	1
	c	1.78	0	2.85	0	2.38	0	1.50	0	0.49	0	−0.50	1
	d	−1.01	1	−0.57	1	0.02	0	0.85	0	1.66	0	1.26	0
	f	1.97	0	2.48	0	2.08	0	0.98	0	−0.08	1	−0.73	1
	h	−0.88	1	−0.52	1	0.05	0	0.89	0	1.72	0	1.48	0

• BP, bodily pain; GH, general health; MCA, Multiple Correspondence Analysis; MH, mental health; PF, physical functioning; SF, social functioning; VT, vitality.

Discussions

The scoring of a scale involves two arbitrary elements: the relative importance of each item in the weighing system, and the scoring system of options within each item. We have used a two-step approach for data analysis: 1) at the item-level, MCA method, that takes account of the nonmetric properties of the data considering ordinal responses as nominal responses; and 2) at the scale-level, ordered correlation/regression analysis, that takes account of the limited digits = n items + 1 (maximum 6 levels of GH and MH) in the scale since the polytomous representation of the binary items sum. These statistical methods were unaffected when changing the set of ordered labels.

The SF-36 questionnaire is a multi-item multidimensional questionnaire using well-accepted Likert standards, but little attention is paid to the specific properties of the ordered categorical data recorded. According to Likert models of parallel items, a construct is often regarded as being latent continuous, is operationalized to be measured by equally important items to increase the reliability and improve precision. The summative Likert scoring system is motivated by the scaling assumptions of highly correlated items and by use of arbitrary numbers, which indicate the ordered structure of the response alternative. It is also assumed that the precision increases with the number of digits in the scale [16].

For analytical convenience, there is a well-established tradition in regarding ordinal data as numeric (continuous) data. Nevertheless, there is negligible scientific evidence to support equidistance in any type of discrete ordinal scaling [22,23]. Ordinal data, irrespective of the type of labeling, thus contain information about ordering only and not about magnitude or distance. There is no reason to assume that response alternatives such as 1 = “definitely true,” 2 = “mostly true,” 3 = “don't know,” 4 = “mostly false,” and 5 = “definitely false” of items a–d of GH scale will yield equal interval scales for individual responses. It means that a set of numerical labels does not represent a mathematical value but serves as a convenient labeling device for ordinal data. These rank-invariant properties of ordinal data include a lack of equidistance between the categories, and that statistical treatment of the data should not be affected when changing the set of ordered labels [8].

One of the major conceptual limitations of Likert item is testing of equidistance (linearity) of option points response. Linearity assumption among response points is often not respected in SF-36 items, and it is necessary to recalibrate the individual answers even by arbitrary decimal values. From here our proposal is to recode Likert values with a method that must be easy to calculate and simple in its interpretation, as the binary format.

Considering SF-36 version 1.6, the constructs of role concerning the perceived limitations due to physical and emotional health problems (RE and RP) are already assessed by binary (yes/no) questions, and we extended the binary coding to the other SF-36 items. The coding 0/1, +/–, A/B is always linear, and the sums of the binary items representing the number of items answered by 1, +, A, which indicates a lack of problems, could be an appropriate rule of scoring.

The MCA processes ordinal data as nominal ones, and transforms them in continuous form by optimal quantifications. The MCA solution allows calculation of the optimal weights of the item options and their ranking, independently by an a priory recoding, enabling an optimal grading for each category response of the questionnaire. In this way, MCA allows to manage the nonlinear information contained in data. In addition, the direction (positive or negative) of optimal weights recodes the data in binary form making item answers linear, and with the same weight (0/1).

Despite the change of the score from summative Likert to cumulative binary, the SF-36 has the same characteristics in the score range of 0–100. Both scores are monotone approximations of underline latent variable constructs. The summative score using binary items is an ordinal scale with limited points, whereas the sum of Likert items is a continuous scale with a large number of points. Consequently, these bring to a low/high variability level, and floor and ceiling challenge.

Nevertheless, binary response format of the SF-36 items would bring various advantages. First, if the item number below the construct is low, i.e., less than 6 or 5, the relation is applicable between response probabilities for a polytomous item and response probabilities for a set of dependent binary item [24]. This perspective points to interesting route of considering the sum score of binary questions as the ordinal score on a polytomous variable, and if this way is taken, it may be possible to examine polychoric and biserial correlations across the binary form scales or to examine odds ratios of know-group comparison by processing ordinal logistic regression models.

Second, the ordinal scale would suggest easy cutoffs that should allow a smart interpretation of QoL differences coming from clinical studies. Considering regression coefficients (or mean differences), it is easier to understand unit change of ordinal score of a SF-36 scale obtained by binary format than unit change of continuous score of the same scale but in original format. For example, in GH scale the ordinal score has always an increasing step point of +20 in the range of 0–100, while using the continuous GH score has an increasing step point of +1 in the range of 0–100. Consequently, the GH group differences are more evident if measured in ordinal score than in continuous score. All this would permit a smart use of reliable change index to evaluate clinical significance in SF-36 outcomes [25].

Third, our study shows that the binary recoding of the original format provide similar results. The findings of this study, based on the ECRHS data, support that the psychometric questionnaire qualities would be fully respected by a binary model. The internal binary consistency shows comparable values with the Likert ones, and these are always higher than the minimum suggested (0.70). The hypothesized Principal Component structure of PCS and MCS of health underlying the SF-36 items was also observed in the binary recoding. Clinical validity produces results that support hypotheses suggested and are compatible with those of the Likert version.

A binary questionnaire would give a practical usefulness for physicians who administer and use it. Binary format would reduce the time to draw up, because it would not be necessary any more to recalibrate the opposite interpretation items, and the score scale would need only the simple sum of the positive (yes) answers. The binary score is better suited than summative Likert score for communicating survey results to practitioners, because it has an interpretation closely related to solving problems (positive or negative), whereas summative Likert has a more complicated interpretation in terms of number of points. Binary score is quick and simple, and allows immediate feedback to those testing.

From the patient point of view that fills in the SF-36, the binary format has an easier interpretation. It has only two possible choices for each question (yes and no, positive and negative, better and worst, etc.) avoiding errors due to misunderstanding and answer interpretation, because some patients may have difficulty interpreting Likert questions. In question 9 (MH scale), subjects appeared to recode the response choices as “bad” to “good” rather than “all of the time” to “none of the time.” More items b and c have an opposite interpretation of item h: subjects may respond “none of the time” to questions b and c regarding filling of anxiety and depression and they would respond “none of the time” again to question h “have you been an happy person” while saying “yes I am an happy person”[26].

Often clinical researches require short questionnaire of faster compilation, and SF-36 has showed to be long for administering; therefore, it is better to use a shorter version as SF-12 or SF-8. These ones, proposed until today, have not a regular application if there are missing data. In fact, a missing answer causes the not substitution possibility and so the score calculation could not be processed. In addition, the information lost reduces the statistical power causing selection bias [27]. Another negative feature of shorter versions is reduced possibility to define the summary measures, but it is not possible to define values regarding single scales causing a further loss of information. All of this is satisfied by the binary version that, with all the 36 original items, allows a good identification both of two health components and of the eight health concepts, pursued by the Likert. Besides, new version structure, if compared with the original format, should substantially reduce time spent to administer and fulfill the questionnaire, and its applying provides essentially the same prognostic information reducing the resources required.

The binary recoding and the scoring system are obtained on ECRHS data, a combination of random sample and symptomatic sample, nerveless if the optimal quantifications of SF-36 questionnaire performing HOMALS within samples were similar (data not shown), the binary scores were different across samples, indicating an homogeneous recoding, and a sensible scoring instrument. Although no computer data simulation was used to verify the findings of the present study, the results seem encouraging about the feasibility of downsizing Likert format offering a plausible alternative for purposes of monitoring the health status in general and in specific populations, even if binary version loses something of reliability and of general information.

Finally, the binary responses offer the chance to order unambiguously the items by frequency of positive (= 1) response for testing a Guttman's scalogram analysis, or for fitting of binary Item Response Theory [28] with parametric models (Rasch Model), or nonparametric models (Mokken model). These models are probabilistic versions of the deterministic Guttman scaling model, and many fit statistics in both the Rasch and Mokken models measure the closeness of the solution to the perfect Guttman scale. Rasch models known under names as the rating scale model, the graded scale response model, the partial credit model consider Likert data, but a reduction of computational burden and simple parameters interpretation provide practical and superior advantages of the dichotomous data with respect to polytomous data.

Conclusions

Despite the loss of information due to the reduction of response's possibility, our results indicate that the SF-36 binary recoding ensure the underlying structure of the original SF-36 is not jeopardized. In addition, it meets at least the same required standards giving the possibility to propose a new version of smarter and easier methodology of administration, compilation, score calculation, and data processing. Consequently, binary recoding may be a valid alternative to the already existing shorter versions, suitable in administering both in clinical setting and clinical trials, in subjects with serious diseases, and by telephone (reduction of times and costs).

The disadvantages of the new binary format are at present the lack of questionnaire testing, and comparability with other studies; and the lack of the interface with the well-established tradition of handling ordinal data as if they were interval too. Assuming quantitative properties simplifies the data analysis and such an approach will not be questioned, while the methods related to ordinal data analysis (polychoric and polyzerial correlations and ordered regression models) are not well-known in medical research.

Source of financial support: This study was partially supported by grants from Ministry of University and Research, University of Pavia.