2.1 Study Design

The EPAT-64 and its short form, the EPAT-16, were developed and tested as part of the ASPIRED study (Assessment of Patient-Centeredness Through Patient-Reported Experience Measures) [13], a 6-year mixed-methods research project on the measurement of patients' perspectives on PC. Here, we report on the item selection process and psychometric evaluation of the EPAT-16, using the same data set that was used for the development and evaluation of the EPAT-64 [5]. Since no specific guidelines exist for reporting psychometric evaluations of PREMs, we followed the COSMIN guidelines for reporting patient-reported outcome measures (PROMs) where applicable (see Appendix 1).

2.2 The EPAT Questionnaire

For psychometric testing, we conducted a cross-sectional survey that included all preliminary EPAT items. We developed these items based on the integrative model of PC [6, 7] shown in Figure 1, informed by a literature review, key informant interviews with experts and focus groups with patients with chronic diseases [8]. The preliminary questionnaire assessed 16 dimensions of PC, comprising 121 items for inpatients and 120 items for outpatients. Our goal was to create a generic instrument that would be applicable across different diseases. Participants rated the EPAT items on a 6-point Likert scale of agreement (1 = completely disagree, 2 = strongly disagree, 3 = somewhat disagree, 4 = somewhat agree, 5 = strongly agree, 6 = completely agree), with an additional ‘does not concern me’ option for each item. Higher scores indicated a greater degree of PC, except for nine items, which we reverse-coded before analysis. We originally developed and tested the EPAT in German. For this paper, we translated the EPAT-16 into English using the TRAPD process, which involves a team-based approach consisting of translation, review, adjudication, pretesting and documentation [14].

2.3 Data Collection and Participants

We used the same sample for the EPAT-16 item selection that we had used for the EPAT-64. We collected data through both paper-pencil questionnaires and an online survey conducted via LimeSurvey (LimeSurvey GmbH, Hamburg, Germany). For the paper-pencil data collection, we used consecutive sampling at 10 inpatient and 14 outpatient healthcare institutions (e.g. hospitals, primary care centres) in the metropolitan area of Hamburg, Germany, from June 2020 to February 2022. Outpatients received the questionnaire before their appointment and were asked to report on their experiences in the past 4 weeks. Inpatients received the questionnaire shortly before discharge and were asked to report on their entire inpatient stay. For the online survey, we recruited participants using community-based strategies, including social media and patient organizations from across Germany. Online participants were asked to refer to their last outpatient visit or inpatient stay within the past month. We included adults (18 years or older) with at least one of the following four medical conditions: cardiovascular diseases, cancer, musculoskeletal diseases and mental disorders. Inclusion was based on self-report. As outlined in our study protocol, we aimed for a sample size of 250 participants in each of the two settings (outpatient and inpatient) for each of the four medical conditions, resulting in a target sample size of 2000 participants [13].

The full questionnaire for data collection included the preliminary version of the EPAT, the first item of the German version of the 12-item Short Form Survey [15], the German version of the 8-item Client Satisfaction Questionnaire (ZUF-8) [16], the European Health Literacy Questionnaire (HLS-EU-Q16) [17] in German as well as questions on demographic characteristics and health status. We also added two control items (e.g. ‘To show that you are reading attentively, please check “somewhat disagree”/”somewhat agree”’). Data collection was anonymous. Participants gave explicit informed consent (either written before receiving the paper–pencil questionnaire or by active click at the start of the online survey) and could opt to receive a €10 incentive for completing the questionnaire.

2.4 Data Analyses

3.1 Sample

Details of the sample were already reported in the primary publication on EPAT-64 [5]. The participant flow chart and the table with all sample characteristics are replicated in Appendix 2. The final analyses included n = 1092 outpatient and n = 931 inpatient questionnaires. Of the outpatients, 59.1% were female, 38.5% male and 0.5% identified as diverse. Their mean age was 52.8 years (standard deviation [SD] = 17.4, range 18–95). Regarding medical conditions, 25.4% were treated for cardiovascular diseases, 25.0% for cancer, 19.9% for musculoskeletal diseases and 25.0% for mental disorders. In the inpatient sample, 41.3% were female, 55.5% male and 0.6% identified as diverse. The mean age was 55.5 years (SD = 17.4, range 18–92). Of the inpatients, 30.7% were treated for cardiovascular diseases, 37.7% for cancer, 9.9% for musculoskeletal diseases and 21.7% for mental disorders.

3.2 Item Characteristics

The items selected for EPAT-16 are shown in Table 1 with their corresponding dimension of the EPAT-64, their English translation and their item characteristics. None of the final EPAT-16 items had to be reversed.

Regarding relevance, in the outpatient sample three items had ‘does not concern me’ response rates above 30% (highest rate was for the ‘physical support’-item with 54.7%). In the inpatient sample all items had ‘does not concern me’ response rates below 30% (highest rate was for the item ‘involvement of family and friends’ with 29.6%). Acceptance was good for all items (highest rate of missing responses was 1.4% for outpatients and 1.5% for inpatients). Regarding item difficulty, 12 and 11 items in the outpatient and inpatient sample, respectively, were within the recommended range. The remaining items showed ceiling effects. Item total correlations ranged from 0.403 to 0.748 in the outpatient sample and from 0.489 to 0.742 in the inpatient sample. Item characteristics per medical condition are shown in Appendix 3.

3.3 Psychometric Analyses of Final EPAT-16

Standardized factor loadings are shown in Table 1. They ranged from 0.407 to 0.829 in the outpatient sample and from 0.444 to 0.803 in the inpatient sample. Except for SRMR, all model fit indices slightly differed from the recommended range (see Table 2). We further explored this misfit by calculating residual correlations between items and modification indices (see Appendix 4). The highest residual correlation was found between the items for ‘Integration of medical and nonmedical care’ and ‘Involvement of family and friends’ (0.249 for outpatients and 0.278 for inpatients). The average variance explained of the unidimensional model was 94.7% for the outpatient sample and 94.5% for the inpatient sample. McDonald's Omega indicated a high reliability of the unidimensional model (0.908 for the outpatient version and 0.906 for the inpatient version). Regarding measurement invariance, the model fit indices exhibited only slight changes for configural invariance and weak invariance when compared to the CFA of the full data set. For strong invariance, all model fit indices worsened noticeably. Yet, likelihood ratio tests for model comparisons (configural invariance vs. weak invariance, and weak invariance vs. strong invariance) showed a significant decline of fit (p < 0.001) for all comparisons in both samples.

Regarding construct validity, all hypotheses were confirmed. In the outpatient sample, the EPAT-16 sum score had a pooled correlation of 0.757 with satisfaction with care and of −0.113 with general health status. In the inpatient sample, those correlations were 0.794 and −0.050, respectively.

EPAT-16 sum scores had mean = 72.6 (SD = 15.7) in the outpatient sample and mean = 73.6 (SD = 15.2) in the inpatient sample (maximum possible range is 16 [lowest degree of PC] to 96 [highest degree of PC]). Distribution of EPAT-16 sum scores is shown in Figure 2. Mean, SD and percentiles for the EPAT-16 sum scores for the overall samples and per medical condition can be found in Appendix 5.

4.1 Summary

We reported on the development and item selection for the EPAT-16, a short form designed to assess the overall degree of PC. We also evaluated the psychometric properties of the final EPAT-16 within the item selection sample. All items of the EPAT-16 demonstrated high acceptability and good corrected item-total correlations. While all items showed high relevance in the inpatient sample, three items were marked as ‘does not concern me’ by more than 30% of the outpatients. Approximately three-quarters of the items displayed appropriate difficulty levels, while the remaining items exhibited ceiling effects. The sum score showed a good degree of variance, although most scores were concentrated in the upper half of the possible range. Regarding structural validity, most model fit indices for the unidimensional model were slightly outside the recommended range. However, the model explained a substantial portion of the item variance, and McDonald's Omega indicated high reliability. Even though the model fit indices changed only slightly when imposing the same factor loadings, based on the likelihood ratio tests the EPAT-16 did not achieve weak measurement invariance. Finally, all hypotheses regarding construct validity were confirmed.

The EPAT-16 offers an efficient and concise method for assessing PC while ensuring representation of all 16 dimensions in the integrative model of PC [6, 7]. This was achieved by an item selection strategy that, next to item characteristics, took into account to which extent an item reflected the dimension definition. While the EPAT-64 allows to explore detailed questions about specific dimensions of PC, the EPAT-16 serves as a potentially valuable tool when the focus is on an overall PC score and resources are constrained. Reducing the number of items to a quarter makes the EPAT-16 more feasible in routine care, where time and personnel are limited, and patients might have low cognitive abilities or energy due to their medical conditions and treatment. The EPAT-16 can be utilized in routine care for continuous feedback on the overall degree of PC or for quality improvement. It can be used in large-scale multivariate studies assessing the relationship of a large number of constructs and their relationship to PC. It also enables large-scale collection of patient feedback across healthcare institutions, facilitating comparisons and reporting of PC levels.

Although other short questionnaires assessing PC exist [31, 32], EPAT stands out due to its rigorous item development and selection process, which ensures a strong theoretical foundation and a comprehensive evaluation of PC [33], even with a reduced number of items.

4.2 Limitations and Future Research

The results presented in this study are limited by the fact that all psychometric analyses were conducted in the same sample as had been used for item selection [12]. It is also the same sample which we used to develop and evaluate the long version, EPAT-64. To confirm that the psychometric properties hold true when participants respond to only the 16 items, the EPAT-16 should be administered to a new, independent sample in future studies. Future research should specifically ask patients to complete both the EPAT-16 and EPAT-64 separately, allowing for an accurate calculation of the correlation between the two questionnaires [12]. We did not report this correlation here, as using the same sample would have resulted in overestimation due to using responses on the EPAT-16 items on both sides of the calculation [12].

Although the items were selected based on both content and psychometric characteristics—a key strength of the EPAT-16—this process could have been more transparent by incorporating formal content rating and more representative by involving a larger group of experts in the selection process [12].

Furthermore, most model fit indices were slightly outside the recommended range, suggesting that exploring alternatives to the unidimensional model may be worthwhile. The residual correlations and modification indices presented in Appendix 4, in conjunction with theoretical considerations, can guide the development of such alternative models. An example for how such a model might be developed can be found in Appendix 4. Wording of the EPAT-16 items was designed to be applicable for all patients regardless of their medical condition. Yet, measurement invariance between medical conditions was not supported in the data analysed here. Hence, while it can be applied generically to different healthcare institutions, comparability of results between different medical conditions is limited. Appendix 5 provides results of EPAT-16 sum scores per setting and medical condition to allow users to compare results of similar samples. Future research should assess the psychometric properties of the EPAT-16, including its factor structure and how it changes, in patient groups with further medical conditions that were not included here. In general, before using the EPAT-16 in different patient populations, its psychometric properties first should be tested in comparable samples. Adaptations of the EPAT-16 to specific populations might be required, in particular where items show high rates of ‘does not concern me’ responses, indicating limited relevance for the respective population. Furthermore, ceiling effects of some items reduce the ability of the EPAT to differentiate between healthcare facilities with high PC ratings. This might be addressed by changing the response scale (e.g. frequency-based scales rather than agreement-based scales) or incorporating additional discriminating items to enhance sensitivity.

In addition, the strong association of the sum score with the results from the Client Satisfaction Questionnaire deserves further attention, as the construct measured by the EPAT is likely to be a complex mixture of subjective and objective experiences [5]. This should be further explored in future studies. Finally, validating the English version of the EPAT-16, along with translating and testing it in other languages, would expand access to more patients and facilitate research on PC across different cultures, countries and healthcare systems.