1. Introduction

Many cancer survivors are not adequately supported to deal with the consequences of cancer and cancer treatment [1], which impacts physical, psychological, and social wellbeing of survivors [2, 3]. These problems lead to reduced quality of life [4] and work limitations, which can negatively impact income and social connectedness and come with substantial societal costs [5–8]. Rogers, McCabe and Dowling [9] indicated that survivors lack support after initial cancer treatment is completed. The needs of cancer survivors go beyond diagnosis and treatment, with people needing surveillance and care for the late and long-term effects of cancer treatment and comorbidities [10]. Their needs vary in intensity, frequency and complexity of care and the required management [11]. Therefore, care must be personalised to meet the needs of cancer survivors and reduce societal impact.

The biopsychosocial model is a helpful lens for understanding the consequences of cancer and cancer treatment [12]. This model was developed as health does not solely depend on biological factors but is also influenced by psychological and social influences, which demonstrate the need for a holistic approach. A holistic approach recognises a person as a whole and acknowledges the interdependence of biological, social, psychological and spiritual aspects [13]. To provide tailored care, Dekkers and Hertroijs [14] developed patient profiles for diabetes and joint replacement using self-reports. Wijlens et al. [15] developed a holistic patient profile for fatigue after breast cancer which investigates patient experience of CRF and its perpetuating factors (factors responsible for the persistence of fatigue) [16]. However, this is not yet operationalised into a holistic assessment questionnaire. This holistic assessment questionnaire could be used to personalise towards the needs of individual cancer survivors in order to provide tailored care as perpetuating factors are investigated. A holistic assessment questionnaire for late and long-term effects of cancer should be available to be able to personalise care to the needs of cancer survivors to reduce the impact of these effects in their daily life.

For cancer-specific instrument development, the following three steps are performed: (1) definition of construct [17–20], (2) item generation [17, 19–22] and (3) item reformulation [17–19, 21, 23–26] before pilot testing, validation and reliability is evaluated [18]. For holistic assessment questionnaires, the considered construct refers to the concept that is the objective to study, which is multidimensional and complex [17]. As a result, the development process of a holistic assessment questionnaire might differ from cancer-specific instrument development. Several holistic questionnaires have been developed [27–32]; however, the development process of these questionnaires is not described in detail [29, 30]. Rothrock, Kaiser and Cella [33] described the development of Patient-Reported Outcome (PRO) Measure that provide patients’ perspective on functioning, symptoms, wellbeing, and experiences with treatment. With regard to cancer-specific instrument development, validated questionnaires that assess (aspects of) the construct are preferably combined in the item generation step. The number of validated questionnaires that need to be included would be immense and many items would be irrelevant because of the multidimensionality and complexity of a holistic construct. Therefore, it is not possible to combine generic and disease-specific instruments to develop a holistic assessment questionnaire. It is important to have a detailed construct definition, to be able to operationalise a holistic construct into a compact holistic questionnaire. Instructions to develop this are lacking, which emphasizes the need for a method to develop holistic assessment questionnaires. This method is expected to have a thorough first step concerning the construct definition.

A holistic assessment questionnaire could be helpful for cancer-related fatigue (CRF) as it impacts quality of life [34, 35]. CRF is multidimensional and is one of the most common and debilitating symptoms experienced during and after cancer, but standards to select the treatment that is most appropriate for individual patients are lacking [36–38]. CRF is unlikely to decrease without treatment when experienced years after cancer treatment as perpetuating factors cause persistence of CRF [39]. Several types of treatments have been proven to be beneficial in reducing CRF. These treatments have different concepts and are, on average, effective in research settings but not for all patients [37]. The individual should be holistically assessed so that the right treatment can be selected because of CRF’s multidimensionality and significant emotional, physical, and social impact, which is not limited to cancer survivors themselves [1]. Therefore, the aims of this study are to present a method for the development of holistic assessment questionnaires including its challenges. This process will be illustrated by the operationalisation of a holistic patient profile into a holistic assessment of CRF questionnaire for CRF in breast cancer as a case [15].

This paper is organised as follows: the questionnaire development and process are described, illustrated with the example in CRF combining the methods and results, followed by the discussion. Within the description of questionnaire development, an introduction is given after which the three steps of questionnaire development follow.

2. Questionnaire Development

The proposed method to develop a holistic assessment questionnaire is illustrated in Figure 1, and consists of (1) construct definition on the element level, (2) item selection, and (3) item reformulation. For the construction definition (1), the construct should be defined based on thorough research, such as interviews with relevant stakeholders. Applying a holistic approach results in a broad construct definition, so we advise starting with the construct definition, composed of themes with a description on element level of each theme (construct > theme > element). Elements are the components of a theme. After the construct definition, item selection (2) is an iterative process of searching for validated questionnaires that together are able to assess the full construct and redefinition of the construct. A construct description (1) that is too broad will result in many irrelevant items on the validated questionnaires (2) because the construct of the questionnaires does not fully correspond to the construct definition. Our suggestion would be to blend, going back and forth between construct definition and validated questionnaires, and to narrow the construct description to the construct themes’ elements to be able to select items from questionnaires. A balance needs to be found between gathering as much relevant information as possible and the expected time investment of filling in the questionnaire. Item selection requires a considerable amount of work including discussion with experts and future users [21]. During item reformulation (3), the selected items are rephrased to be comprehensible for most of the target audience with the help of expert consultation. These experts should have experience in the field of the intended use of questionnaire. Each phase is described in more detail in the following and text boxes are illustrative for the example of the holistic assessment of the CRF questionnaire development process.

2.1. Construct Definition—Theme Level

A construct definition is an essential first step to select what to include in the construct [40]. When the construct definition is too narrow, important elements may fail to be included, resulting in underrepresentation of the construct. Alternatively, a too broadly defined construct can result in construct-irrelevant variance as extraneous elements are included in the construct. Preferably, constructs should be grounded in a theoretical framework. The construct definition for our example is shown in Textbox 1.

2.1.1. Textbox 1: Construct Definition on the Theme Level for Holistic Assessment of the CRF Questionnaire

Healthcare professionals and breast cancer patients and survivors were interviewed to determine the relevant aspects for monitoring CRF holistically. This qualitative information was used to develop a holistic patient profile for CRF after breast cancer, as well as relevant literature and is described in more detail [15]. The holistic patient profile for CRF consist of the five themes: (a) experience of CRF, (b) impact and consequences, (c) coping, (d) personality, and (e) CRF treatment. A theoretical framework for CRF was not found. Therefore, themes corresponding to perpetuating factors their conceptual description or assessment method were searched for.

For an adequate construct definition of CRF, it is also important to determine which qualitative themes are all suitable for operationalisation into a questionnaire. The differences between the construct elements and holistic patient profile are explained. Dysregulation of sleep [41–46] and dysregulation of activities [41–44, 46] are perpetuating factors that were not included as a theme in the holistic patient profile but were mentioned in quotes as part of several coping styles. Therefore, sleep and activity are included in the questionnaire as elements of the theme ‘day pattern’ so that CRF (construct) > day pattern (theme) > sleep + activity (elements). Summarising, the initial themes of the operationalised questionnaire are CRF dimensions, general CRF, day pattern, coping, personality and social health. See questionnaire themes and elements in Figure 2, which also provides an overview of the construct definition process from the holistic patient profile towards the description on the element level. This corresponds to step 1, construct definition, of Figure 1.

2.2. Item Selection

It is advised to use existing validated questionnaires if possible for item selection, as development and validation of a questionnaire is a costly and time-consuming process [17, 22, 40]. Using a holistic approach, the construct is likely to consist of several themes which can be divided into elements. For the development of the holistic assessment questionnaire, it is advised to search for validated questionnaires that are able to assess the construct’s themes.

Validated questionnaires can be found via qualitative research, questionnaires used in literature and snowballing method assessing the references of the articles. Guidelines, for example, found on guideline.gov, might also be excellent sources which summarise relevant literature. In addition, one can search in scientific libraries as well as search engines. If articles discuss suitable questionnaires, they often do not include the questionnaire items, especially in languages other than English. If a questionnaire is not open access, other means should be used such as questionnaire repositories (general or disease specific) as well as contacting the authors directly.

An exhaustive list of items is a primary requisite and critical for the development of a good questionnaire [21]. Evaluation of such a list can be performed in several ways such as expert judgement, Delphi study and cognitive interviews with end users [17]. Choices need to be made to select appropriate items for the construct. A priority list, containing the questionnaire requirements, can be helpful for item selection. This priority list could relate to language, costs, scoring method, length, answer options of items, time period, target group and questionnaire use of validated questionnaires. For example, with regard to answer options, closed-ended item are easier to administer and analyse compared with open-ended items [24]. Criteria relevant for the aim of other studies could be to align with purpose of the questionnaire (diagnosing symptom, measuring symptom change and symptom inventory). The time period should align with the intended use of the instrument as well as with the temporal characteristics of the construct [47].

The item selection procedure with regard to our CRF example is illustrated in Textbox 2.

2.2.1. Textbox 2: Item Selection

For the development of a holistic assessment of CRF questionnaire, our search aimed at finding validated questionnaires that are able to assess the questionnaires’ themes of Figure 2. Validated questionnaires followed from various sources consisting of the questionnaires mentioned by the patients and healthcare professionals who were involved in the qualitative study, as well as the questionnaires used in literature (including [48–51]), and their references (snowballing method) were consulted. The guidelines from the National Comprehensive Cancer Network (NCCN) were also consulted for fatigue questionnaires [52]. Repositories for instruments in general (e.g., meetinstrumentenzorg.nl, healthmeasures.net and richtlijnendatabase.nl) and disease specific (for cancer, e.g., facit.org and kankernazorgwijzer.nl) were searched. The total search resulted in 110 validated questionnaires (generic and disease specific) that are able to assess (part of) a theme or multiple themes (Figure 3).

As the 110 relevant questionnaires of Figure 3 resulted in over three-thousand questionnaire items, a priority list was necessary to assist in selecting the correct items. Table 1 shows the priority list used for the holistic assessment of the CRF questionnaire. This list was constructed by the first author and was prioritised with all authors.

Table 1. Priority list to select questionnaires indicating the minimal, preferred and alternative requirements.

	Preferably	Alternatively	Minimally
Language	Dutch	Translation
Costs	Free	Copyright
Scoring	—
Target group	Breast cancer	Cancer	Other diseases
Length	Maximally 15 min per week for the whole questionnaire
Answer options	Likert	Multiple choice	Yes/no
Time period	Last week	Today	Last 4 weeks
Questionnaire use	Entire questionnaire		Relevant item

3. Discussion

Tailoring care to an individual, requires a holistic overview of this individual and their context. With this article, we propose a method to develop a holistic assessment questionnaire. The development process consists of construct definition on the element level, item selection and item reformulation. Development of a holistic assessment questionnaire is time consuming, and this was exemplified by the operationalisation of the holistic patient profile for CRF into a holistic assessment questionnaire, which resulted in including items of twenty-one different validated questionnaires. This was only possible after an immense questionnaire search, composing a priority list to select the relevant items, and evaluation with experts and patient advocates. It is advised to determine the needed items on the element level instead of the theme level in contrast to the cancer-specific instruments developed method. The construct definition on the element level was based on interviews with healthcare professionals and breast cancer patients and survivors [15]. The current healthcare system consists of specialised departments, which increases the expertise on a symptom or specific disease but does not consider the patient holistically nor use the biopsychosocial model [12]. The experience and impact of CRF are also perpetuated by psychological and social influences; therefore, the holistic assessment of the CRF questionnaire contains the themes “coping” and “social health.” Future research is needed to determine if the developed holistic assessment of the CRF questionnaire based on qualitative research for breast cancer, could be extrapolated to other types of cancer. Although the development process of a holistic assessment questionnaire is time-consuming, more PROs such as late and long-term effects of cancer are multidimensional and could benefit from a holistic approach in their assessment to enable personalised care and thereby improve quality of life and reduce societal impact.

Newly developed questionnaires must be validated when adapted items or parts of questionnaires are included, resulting in the opportunity to adapt the items to better suit the target population. Validation consists of several different methods, such as face, content, construct, convergent and criterion validation [17, 20, 22, 81, 82]. Face validation assess if the questionnaire measures the intended measure via qualitative research with experts. Regarding content validity, quantitative measures are used to determine if the items of the questionnaire address all the relevant items of the construct [22, 82]. Construct validity aims to show strong correlations between the same constructs and weak correlations with different constructs whereas convergent validity determines the correlation between a scale and conceptually similar (sub)scales [22, 82]. Criterion validity evaluates the instrument compared with the golden standard [82, 83]. As a result of the holistic approach, construct or convergent validity will be unfeasible because of the number of questionnaires that need to be included which also would not resemble the construct description. This also implies that it is not possible to have used factor analysis in the development of the questionnaire. Therefore, face and content validity are the preferred next step to validate the holistic assessment questionnaire. Subsequent research depends on the purpose of the holistic assessment questionnaire (diagnosing symptom, measuring symptom change, symptom inventory).

The holistic assessment questionnaire measures a construct (in our example CRF) consisting of themes (e.g., experience of CRF) that are divided into elements (e.g., cognitive fatigue, emotional fatigue, and physical fatigue) so that construct > themes > elements. A possible disadvantage of this approach is the interpretation, as usually the score of a questionnaire represents the construct as the sum of the underlying themes. This approach implies that adding the themes would result in the construct. However, one can wonder if the sum of the themes always represents the construct. With regard to our example, three themes are about perpetuating factors while the other concerns the experience of CRF. The themes could also be considered as latent variables as they represent an underlying concept measured indirectly (e.g., social health), usually represented as subscales of a questionnaire [21]. In this case, one could score per theme. If the themes (in our example, CRF experience, day pattern, social health and coping) are considered as individual constructs, CRF should have a hierarchical higher level and could, therefore, be something abstract like a network. CRF as the questionnaire’s construct is most logical since fatigue is a tangible concept. For validation, this need to be clearly stated as construct validity is normally researched on the construct level and thus the questionnaire level. If the themes are considered separate constructs, construct validity should be performed on the subscale or theme level. This issue also could arise for other holistic assessment questionnaires developed in the future. Concerning the abovementioned, we propose to have one construct for the holistic assessment questionnaires which could consist of several themes. To validate the holistic assessment questionnaire, face validation and content validation are possible [21].

A validated questionnaire could provide questions for more than one construct theme. This also indicates that only combining generic and cancer-specific instruments to holistically assess late and long-term effects of cancer does not work as shown for CRF. Because of the holistic approach, this would result in a misalignment in the constructs of concept of existing validated questionnaires and the developed questionnaire.

Several resources were used to search for validated questionnaires that could be of use in the development process. As a result, an overview of existing questionnaires related to our construct was created. However, it is challenging to find the items and for some questionnaires, the individual items are not publicly accessible. This is a limitation since this restricted the consulted questionnaires and it could be that other relevant items were missed because of unavailability of questionnaires. However, as the number of questionnaires that were used in our example was still relatively large, we do not believe this affected the findings. Finding repositories with an overview of instruments reduced the number of unavailable questionnaires, for example, the Dutch validation of the International Physical Activity Questionnaire (IPAQ). The article of van Poppel et al. [84] does not contain the questionnaire but it is available at meetinstrumentenzorg.nl [62]. Developed questionnaires should, therefore, be published open access to avoid this limitation for future research. It is understandable that questionnaires are not published openly as protecting intellectual property can be challenging and hinder implementation.

4. Conclusion

In this study, we proposed a method to develop a holistic assessment questionnaire to be able to personalise towards the needs of individual cancer survivor. The development of a holistic assessment questionnaire consists of a construct definition with assessing the requirements on the element level, item selection and item reformulation. We illustrated these steps using the development of a holistic assessment questionnaire for CRF after breast cancer. To develop a holistic assessment questionnaire, several adjustments with regard to cancer-specific instruments development were proposed, whereafter we presented a strong and novel method to develop holistic assessment questionnaires. This method could serve as a blueprint for the development of other holistic assessment questionnaires to tailor care and thereby improve quality of life of cancer survivors and reduce societal impact.

In the future, open access questionnaire repositories would limit the time burden of the development process of a holistic assessment questionnaire. Preferably, these repositories also contain information about the language availability, scoring procedure, intended use and target population in which the questionnaire is validated and reliable.

Ethics Statement

This is a methodological article for which no ethical approval is required.

Consent

Informed consent was obtained from all participants.

Conflicts of Interest

The authors declare no conflicts of interest.

Author Contributions

The study was conceptualised by K.W., A.W., L.B., S.S., M.V. and C.B. Data were collected, analysed, visualised and written into the first draft of the manuscript by K.W. under the supervision of A.W., M.V. and C.B. All the authors critically revised various versions of the manuscript and read and approved the final manuscript.

Funding

This work was supported by Queen Wilhelmina Fund (KWF) Kankerbestrijding and Dutch Research Council (NWO) Domain AES, as part of their joint strategic research programme: Technology for Oncology II. The collaboration project is cofunded by the PPP Allowance made available by Health∼Holland, Top Sector Life Sciences & Health, to stimulate public–private partnerships.