Development and validation of a primary sclerosing cholangitis–specific patient-reported outcomes instrument: The PSC PRO

A. Concept Elicitation

Following the literature review, which did not return any comprehensive disease-specific PRO instrument, concept elicitation research with 20 PSC patients (recruited from four clinical sites in Florida and North Carolina and PSC Partners Seeking a Cure) was conducted by a trained qualitative interviewer (A.S.) to document the symptom experience of patients with PSC, capture the patients' descriptions of their symptoms, identify the impact of symptoms on their lives, and gather information about the frequency of the symptoms and impacts (unpublished report). As a result, a draft version of PSC PRO was developed (Supporting Table S1). It included three modules: module 1: PSC Chronic, Everyday Symptoms (a nine-item daily assessment of PSC symptoms); module 2: PSC Acute Symptoms (a seven-item measure capturing periodic assessment of PSC symptoms associated with acute episodes of cholangitis); and module 3: PSC Impacts from Symptoms (a 29-item functional impact measure with seven domains). That draft version was then subjected to content validity assessment in a series of cognitive interviews with an in independent cohort of PSC patients.

B. Cognitive Interviews

For the three rounds of cognitive interviews, participants were recruited from four clinical sites in Colorado, Florida, North Carolina, and Washington and from the PSC Partners Seeking a Cure Patient Registry (http://pscpartners.org/patient-registry/) in March-December 2015; the interviews were conducted by two trained interviewers for both telephone (round 1, paper version only) and in-person (rounds 2 and 3, electronic platform and paper versions) interviews. The sample included participants with PSC with or without IBD, without history of LT or CCA, or any other potentially confounding comorbid disease; thus, the study sample is supposed to be representative of the symptomatic PSC patient population expected to be enrolled in upcoming clinical trials. During the cognitive interviews, participants reviewed the instructions, item content, recall period, and response options for the draft PSC PRO instrument. All participants in the electronic PRO (ePRO) cognitive interviews reported a positive impression of the electronic device format and found it easy to move from screen to screen, despite the fact that nearly half reported being unfamiliar with technology, and none had previously used an ePRO device. There was no evidence that the ePRO formatting of either module changed the way respondents interpreted or responded to the instructions or questions.

After each round of cognitive interviews, the data were reviewed by the study team, and recommendations for changes in the instrument modules and individual items were made with additional review by a translation expert. Throughout the qualitative research work (both concept elicitation and cognitive interviews), a steering committee consisting of four expert hepatologists was closely involved in the research process and decision making. This committee provided expertise on the PSC condition and worked closely with investigators to review the PSC PRO item development process. The qualitative data were coded by two members of the study team with review by a senior reviewer (A.S.) and analyzed using ATLAS.ti software.9

A. Patient Population

A validation study was conducted to assess the reliability and validity of the PSC PRO tool in a separately enrolled group of patients with an established diagnosis of PSC (enrolled from November 2016 to June 2017); required sample size was calculated using the data from the cognitive interviews. Patients were ages 18-70 years, of both sexes, all ethnicities, with or without IBD, with or without cirrhosis, and without any other potentially confounding comorbidities. After approval of the study by the Institutional Review Board, patients were screened, verbally consented and interviewed by phone, and then asked to complete a medical history questionnaire followed by the PSC PRO and four other PRO questionnaires (the Short Form-36 version 2 [SF-36v2], the Chronic Liver Disease Questionnaire [CLDQ], the Primary Biliary Cirrhosis-40 [PBC-40], and the 5-D Itch Scale9-13 using a secure ePRO website. Patients had unlimited time to answer each question, but for the purpose of calculating the total time to complete the battery, the maximum time to answer one question was capped at 5 minutes given that longer time would likely be attributed to an interruption. The first 50 patients were additionally asked to complete PSC PRO within 3 months after the initial administration for the purpose of test-retest reliability assessment.

B. Internal Consistency and Test-Retest Reliability

Internal consistency of PSC PRO was quantified by Cronbach's alpha coefficients for its domains without and then with one-item removal, as well as item-item and item-total correlations. The distributions of the domain scores were also evaluated for skewness and floor-ceiling effects.

For the purpose of test-retest reliability assessment, a subgroup of patients without any evidence of clinical change (determined by the interviewer) completed PSC PRO twice within approximately 3 months. Correlations between the two administrations were calculated. The distributions of differences in scores between multiple administrations were evaluated, and the median differences were compared to zero by a nonparametric test for matched pairs. Additionally, the intraclass correlation coefficients (ICCs; ICC_(3,1) by Shrout-Fleiss14), which is another indicator of a repeated measure reliability, were calculated for the PSC PRO domains; a general linear model with subject ID and the administration ID (first or second) as two predictors of an outcome (the domain score) was used.

C. Validity

Construct validity was assessed in a round of confirmatory factor analysis, by discriminant validity (correlations with own domains vs. other domains), and by associating the PSC PRO scores with potentially relevant demographic and clinical parameters (known-groups validity). The known-groups validation parameters were age, sex, the presence and severity of cirrhosis, history of depression, the presence of IBD, ulcerative colitis, and Crohn's disease, and other clinically relevant parameters with a prevalence of at least 10%; those were also qualitatively used to appreciate the magnitude of the minimal clinically important difference (MCID) for the domains of PSC PRO. Wilcoxon's nonparametric test was used to compare the PSC PRO scores between the groups of patients listed above. Additionally, in a round convergent validity assessment, which used the data from a few validated generic and disease-specific PRO instruments (SF-36v2, CLDQ, PBC-40, and the 5-D Itch Scale) administered at the same time as PSC PRO, correlations between the most related domains of PSC PRO and of those instruments were expected to be the highest (such as between the Emotional Impact domain of PSC PRO and mental health–related domains of SF-36, Emotional or Worry domains of CLDQ, and Emotional domain of PBC-40; or between Physical Function of PSC PRO and Physical Functioning of SF-36, Activity/Energy or Fatigue or CLDQ, and Fatigue of PBC-40).

All quantitative analyses were made in SAS software (version 9.3; SAS Institute Inc., Cary, NC). The study was approved by the Western Institutional Review Board, and a waiver of signed consent was obtained. For subjects who were eligible for the study, a verbal consent by telephone was required.

A. Internal Consistency

Both first and second administrations of PSC PRO were used for assessment of internal consistency (N = 155 entries). A round of confirmatory factor analysis validated the original design with seven domains for module 2: 99.8% of the sample variance could be explained by exactly seven factors. Furthermore, an excellent internal consistency was detected in all eight PSC PRO domains—Cronbach's alphas ranged from 0.84 to 0.94 (Table 2). Additionally, the changes in Cronbach's alphas after sequential one-item exclusions did not exceed 0.09 (Table 2), therefore suggesting that the items from the same domains were neither too correlated with nor too different from one another. The greatest variability in Cronbach's alphas was found in the Activities of Daily Living domain (–0.09 after excluding Item 20—“running errands,” +0.04 after excluding Item 21—“avoiding certain foods”). The item-to-own domain correlations were above 0.7 for all but one item from the module 2: Impact of Symptoms; the only exception with correlation of 0.488 was the abovementioned Item 20. Those correlations, however, were substantially lower for the module 1: PSC Symptoms (minimum, 0.28; maximum, 0.74; median, 0.64), suggesting a greater independence of individual symptoms of PSC from each other (Table 2).

The distribution of the PSC PRO domain scores suggested skewness toward lower values, which would reflect a better health status (Fig. 1A-H). Indeed, the proportion of values of less than 3.0 (on a 1-5 scale) exceeded 70% for six of seven domains of module 2: Impact of Symptoms, while less than 10% of patients had domain scores of 4.0 or greater (on a 1-5 scale used for the domains of that module); the only less-skewed domain was Emotional Impact, which had a mode value of 2.0 rather than 1.0 (Fig. 1F). Furthermore, despite the potential maximum of 120 (all the symptoms present at their worst), the maximal observed value for the module 1: PSC Symptoms score did not exceed 70, whereas 78% of patients had that domain score of less than 30 (Fig. 1A). In that context, ceiling effect was not significant given that few or none at all of the patients had the highest possible scores in any of the items or domains (<10% for 36 of 40 scored items, 10%-20% for the remaining 4 items, and <3% for the domain scores). At the same time, the floor effect was more profound for both items and domains: 35 of 40 the items were scored so that >30% of patients had the lowest possible values (of the remaining five items, three belong the Emotional Impact domain and two others are physical and mental tiredness symptoms from module 1), whereas 26 of 40 items had >40% of patients with the lowest scores and 15 of 40 had >50%. Furthermore, from 24% to 36% patients also had the lowest possible module 2 domain scores, with the only exception of the Emotional Impact domain, where that proportion was 5.8%.

B. Validity

Discriminant validity analysis has shown that 41 of 42 items had the highest absolute correlations with their own domains; the only exception was Item 18 (“completing household chores”), which correlated with the domain of Physical Function, insignificantly higher than with its own domain of Activities of Daily Living (0.84 vs. 0.87; P = 0.32).

Known-groups validity of PSC PRO with reference to demographic and clinical parameters is summarized in Table 3 and Supporting Table S7. No difference in PSC PRO scores between patients of different age groups, patients with IBD, ulcerative colitis, aphthous mouth ulcers, or Crohn's disease was found (all P > 0.04). On the other hand, male PSC patients had significantly better scores in comparison to females in five of seven module 2: Impact of Symptoms domains. Furthermore, as expected, patients with cirrhosis, complications of cirrhosis and, in particular, history of weight loss or malnutrition and patients with a history of depression or anxiety had significantly impaired PSC PRO scores.

From the magnitude of difference in the PSC PRO scores observed in this study between the clinically meaningful separate groups, particularly patients with and without IBD (Table 3 and Supporting Table S7), we suggest the MCID for PSC PRO scores to be 4 for module 1: Symptoms and 0.3 for domains of module 2: Impact of Symptoms, until a systematic MCID determination study for this instrument is performed.

Correlations of PSC PRO domains with domains of other validated PRO instruments are shown in Table 4. Although all correlations were statistically significant, the highest correlated domains for the module 1: PSC Symptoms were those related to fatigue and vitality. Role Physical of SF-36 and Fatigue of CLDQ were the highest or close to the highest correlated domains for Physical Function, Activities of Daily Living, Work Productivity, Role Function, and Quality of Life domains of PSC PRO. On the other hand, the Emotional Impact PSC PRO domain was expectedly the most correlated with the Mental Health domain of SF-36, Worry domain of CLDQ, and Emotional domain of PBC-40. Furthermore, the Social/Leisure Impact domain was the most correlated with Social Functioning domain of SF-36, Fatigue domain of CLDQ, and Social domain of PBC-40. Finally, of all PSC PRO domains, the 5-D Itch Scale was the most correlated with module 1: PSC Symptoms score (R = 0.60; P < 0.0001; Table 4), and of module 1 items, the highest correlation of 0.84 (P < 0.0001) was expectedly found for Item 4 (“Over the past 24 hours, how bad was your itching at its worst?”).

C. Test-Retest Reliability

Fifty-three patients completed the PSC PRO instrument for a second time. The median interval between administrations was 37 days (IQR, 27-47). In test-retest analysis of PSC PRO items, all but one item was highly significantly correlated between the two administrations (P < 0.02); the only exception was one of the PSC symptoms (fever) included in module 1. The correlations between repeated measurements of the domain scores ranged from 0.70 (Work Productivity) to 0.85 (Activities of Daily Living; all P < 0.0001; Table 5). Furthermore, all the score differences, except for the module 1: PSC Symptoms score, had a median of 0, and none of the module 2: Impact of Symptoms domains returned a statistically significant difference between the two administrations (all P > 0.05; Table 5). Finally, ICC values ranged from 0.71 to 0.86 for all of the PSC PRO domain scores (Table 5).