The review search strategy was executed in four databases from 2009–August 2019. Two reviewers screened abstracts/manuscripts. Candidate QIs and characteristics were extracted using a tailored abstraction tool and assessed for scientific soundness. To prioritize candidate indicators, a modified Delphi consensus process was conducted. Consensus was sought over two rounds; (1) multidisciplinary expert workshops to identify relevance to Australian CRC MDTs, and (2) an online survey to prioritize QIs by clinical importance.

Results

A total of 93 unique QIs were extracted from 118 studies and categorized into domains of care within the CRC patient pathway. Approximately half the QIs involved more than one discipline (52.7%). One-third of QIs related to surgery of primary CRC (31.2%). QIs on supportive care (6%) and neoadjuvant therapy (6%) were limited. In the Delphi Round 1, workshop participants (n = 12) assessed 93 QIs and produced consensus on retaining 49 QIs including six new QIs. In Round 2, survey participants (n = 44) rated QIs and prioritized a final 26 QIs across all domains of care and disciplines with a concordance level > 80%. Participants represented all MDT disciplines, predominantly surgical (32%), radiation (23%) and medical (20%) oncology, and nursing (18%), across six Australian states, with an even spread of experience level.

Conclusion

This study identified a large number of existing CRC QIs and prioritized the most clinically relevant QIs for use by Australian MDTs to measure and monitor their performance.

1 INTRODUCTION

Colorectal cancer (CRC) is one of the main causes of cancer-related death worldwide and is the third most commonly diagnosed cancer in Australia accounting for 11% of Australian cancer cases.^{1, 2} There were 15,206 cases of CRC diagnosed in 2017 and 5,255 deaths from CRC in 2019.² The CRC care continuum is complex and involves multiple modalities of treatment delivered by health professionals (HPs) from multiple specialist disciplines including CRC surgery, radiation oncology, medical oncology, radiology, anatomical pathology, nursing, and palliative/supportive care. Multidisciplinary team (MDT)-based care is now widely accepted as a standard of care in cancer.^3-7 There are ongoing advances in the various treatment modalities available with rapidly evolving evidence pertaining to their use. Therefore, good communication, collaboration, and coordination among MDT members are needed.^{6, 8, 9} To ensure consistent delivery of high-quality care, it is important to measure and monitor the delivery of care by MDTs to identify any unwarranted variations.¹⁰

An important tool for monitoring care delivery and patient outcomes is a set of clinically relevant, evidence-based quality indicators (QIs).^{9, 11} QIs are measurable elements of practice performance supported by evidence for assessing the quality of care.¹² They can be categorized as structure, process, and outcome measures.¹³ Whilst variability in outcomes for CRC patients may be attributable to patient factors, significant variation exists in processes and structures of care within healthcare organizations.¹⁴ An ideal measurement program should therefore employ multiple QIs to capture a holistic view of patient care delivery. Effective implementation of QI measurement and monitoring can encourage clinical practice reflection, clarify team responsibilities, and focus improvement efforts.¹⁵ Defining QIs in a tumor stream can facilitate the creation of benchmarks for standardization of care across healthcare services.⁹ QI development can be evidence-based and derived from scientific literature or, when scientific evidence is lacking, determined by an expert panel of HPs in a consensus process.¹⁶ Ideally, a QI is both selected from research data and supplemented by expert opinion.¹⁷

Given the wide-reaching implications of quality measurement, QIs should be developed and selected systematically, in alignment with patient-centered MDT-based care delivery, and engage HPs in their selection.¹⁸ However, current cancer care quality measurement efforts remain heavily directed towards meeting the administrative requirements of service providers, and credentialing of services as determined by discipline-specific colleges, accrediting bodies, and government agencies. Furthermore, the utility of QIs has been limited by the availability of and accessibility to quality datasets.^{19, 20} For these reasons, many of the existing QIs have been selected by consensus and convenience and are discipline focussed not patient-centered MDT focussed. The aim of this study is to identify an evidence-based set of QIs and clinically prioritize them for use in Australian CRC MDTs.

2 METHODS

We conducted a systematic review of the literature describing existing CRC QIs to identify QIs from diagnosis through treatment to survivorship or palliative care and assess the extent to which these QIs have been developed and validated. Following this, a two-step modified Delphi consensus process was undertaken to prioritize the extracted QIs into a final set based on their clinical importance to Australian CRC MDTs.

2.1 Systematic review

The search strategy was guided by the PICO Framework.²¹ Studies were included if they described the development, selection, validation, or review of CRC QIs from diagnosis, treatment follow-up, and survivorship or palliative care. On August 14, 2019, string terms (Supplementary Material—Search Strategy) were executed in four databases (MEDLINE, EMBASE, CINAHL, and PsychInfo) for the dates January 1, 2009–August 14, 2019. String terms were developed using MeSH and free-text terms referring to key concepts; CRC (e.g., colorectal neoplasms, rectal tumor, and colon carcinoma) and QIs (e.g., quality measures or performance indicators). The search was restricted to studies in English. A citation search and a hand search of the reference lists of relevant papers were also conducted. To retrieve supplementary endorsed and validated QIs in the grey literature, a manual search of the pertinent quality measurement and cancer agency websites and publications was conducted. Citations retrieved from the searches were imported into EndNote for de-duplication, then imported into Covidence for screening.

Two authors (Candice Donnelly and Puma Sundaresan) screened the titles and abstracts against the exclusion criteria (Supplementary Material—Search Strategy) to identify potentially relevant studies. Full-text manuscripts were also screened by two reviewers (Candice Donnelly and Michelle Or), and justifications for inclusion or exclusion were confirmed by a third reviewer (Puma Sundaresan). It was anticipated that the QI literature would include studies of different methodological designs. Therefore, a quality assessment instrument was chosen that had been used in other QI reviews to assess the methodological quality of a range of study types and extracted QIs.²² The quality assessment was conducted by one reviewer (Candice Donnelly). The grading system scored the included manuscripts from the highest quality level (i.e., A1: systematic reviews or two independent prospective cohort studies) to the lowest quality level (i.e., C: non-comparative studies and D: expert opinion)²³. This assessment contributed to grading the level of supporting evidence (scientific soundness) for each QI extracted on a scale of “1”, the highest level of evidence to “4”, the lowest level of evidence. Each QI was rated by calculating how many of the graded studies supported the QI combined with the quality of those studies. For example, a QI with a score of “1” was one that had the support of either a study scored as level A1 or two independent studies of level A2.

QIs were extracted by using a tailored template implemented in Microsoft Excel v.16.34. The tailored version included elements of QIs suggested in previous systematic and narrative reviews.^24-26 One reviewer (Candice Donnelly) piloted the form on the first 10 articles and only minor refinements were required. Two reviewers (Mathushan Thevaraja and Candice Donnelly) extracted reported QIs from each study and where available characteristics of those QIs including; numerator, denominator, exclusion criteria, study design, period of measurement, and any additional details, where available (i.e., data source, benchmark, risk adjustment and detected differences in care). To synthesize the data, each extracted QI and characteristic were grouped with those addressing the same measurement, then tabulated under a single QI description. Once de-duplicated, the QIs were divided into domains across the continuum of cancer care including diagnosis, staging, clinical aspects of surgery, radiotherapy and chemotherapy, follow-up, and supportive care, aligned with the Cancer Council Australia Colorectal Cancer Guidelines.³ Subsequently, QIs were categorized as structure, process, or outcome measures, according to the Donabedian Model.¹⁶ Clinical disciplines contributing to the outcome of each QI were also detailed.

2.2 Delphi consensus process

A modified Delphi technique was used to refine the extracted QIs and prioritize the most clinically important QIs. This consensus approach has been widely used and validated within other quality measurement and health services research.^27-29 The consensus was sought over two rounds; a multidisciplinary expert workshop and an online survey. Participants were a purposive sample of multidisciplinary HP experts by organization affiliation. This method of group Delphi can enable participants to provide contextual justifications for deviating judgments.^{30, 31} Given the candidate QIs were extracted from international literature, workshop participants assessed the QIs for inclusion/exclusion or modification based on relevance to modern Australian CRC care delivery. Participants were encouraged to discuss potential gaps and the development of additional QIs. The QIs retained in this process were included in the second Delphi round.

The second round was conducted by an online survey hosted on REDCap³² open to HP members of CRC MDTs across Australia. Participants were recruited by email invitation through professional networks and public sources of information, and study adverts through professional medical societies. Participants rated QIs from the first round on “clinical importance” on a 4-point Likert scale. Free-text responses were available to provide an opportunity for comments. Clinical importance was defined as “the relevance of this information to clinical decision making and provides the opportunity for the MDT to change practice to improve the quality of care delivered”. The level of concordance was measured by inferential analysis. Each QI that met a percentage agreement of >80% recorded as “important” or “critically important”, was included in the final set of QIs. Descriptive statistics were used to describe participants’ demographic characteristics and subgroup analysis.

3 RESULTS

3.1 Systematic review

The literature search identified 708 records from January 1, 2009, to August 14, 2019 (presented in Figure 1).³³ After de-duplication 619 abstracts and titles were screened and 454 records were excluded as they did not have data on the development, selection, validation, or review of CRC QIs or did not meet all of the inclusion criteria. A total of 165 manuscripts were included for full-text review. Group consensus was required for 16/165 (9.7%) of articles to resolve screening disagreements. A total of 118 studies were included. Using the grading system,²² among the included QI studies, the majority were of levels B, C, and D methodological quality; with 85 retrospective comparative studies (72%), six non-comparative studies (5.1%), and 21 expert opinion studies (17.8%), usually based on a consensus process. There were limited A2 studies and none of A1 level; with six prospective cohort studies (5.1%).

Details are in the caption following the image — **FIGURE 1**
Open in figure viewer PowerPoint

PRISMA flow diagram: results of systematic review search

A total of 93 unique QIs and related characteristics were extracted. The number of reported QIs in individual articles varied significantly, with many articles focussed on just one QI and others discussed sets of QIs (range, n = 1–52). The majority of extracted QIs were process indicators (n = 59, 63.4%), followed by outcome indicators (n = 32, 34.4%) and structure indicators (n = 2, 2.2%). The majority of outcome indicators were surgical QIs (26/32, 81.3%). As shown in Figure 2. QIs were disproportionately spread across the Domains of Care. The Domain of Care with the most QIs was “surgical treatment for primary CRC” (n = 29, 31.2%), with the “management of metastatic cancer” domain recording the second most QIs (n = 17, 18.3%) which were also predominantly surgical (n = 8, 47%). The number of QIs measuring supportive care (n = 6, 6.5%) and neoadjuvant therapy (n = 6, 6.5%) was relatively low.

The grey literature search identified seven QI sets endorsed by international quality improvement and cancer organizations, National Health Service Scotland,³⁴ American Society of Clinical Oncology (ASCO)/National Comprehensive Cancer Network,³⁵ ASCO National Initiative on Cancer Care Quality,³⁶ Association of Surgeons of the Netherlands, Dutch Surgical Colorectal Audit,³⁷ Colorectal Surgical Society of Australia and New Zealand,³⁸ Commission on Cancer National Accreditation Program for Rectal Cancer,³⁹ and Queensland Cancer Alliance.⁴⁰ Only one additional QI that was not identified through the peer-reviewed literature search, ‘mortality after chemotherapy or radiotherapy treatment’ in neoadjuvant and adjuvant therapy³⁴ was identified. All other QIs identified in the grey literature, overlapped with those in the extracted QI set from the peer-reviewed literature.

A range of identified disciplines was responsible for contributing to each QI as demonstrated in Figure 3, including; gastroenterology (n = 9), radiology (n = 9), pathology (n = 3), colorectal surgery (n = 73), medical oncology (n = 32), radiation oncology (n = 26) and nursing (n = 11). Approximately half of QIs were multidisciplinary (n = 49, 52.7%), where more than one discipline contributed to its outcome. The evidence supporting each QI was highly variable. Only three QIs were given a scientific soundness score of “1”. These three QIs that have been studied and published extensively included (1) “pre-operative visualization of the colon”, (2) “resected lymph nodes ≥12”, and (3) “post-operative mortality”. Surgical CRC QIs had the highest number of QIs with an evidence score of either “1” or “2”, with 15 QIs supported by a number of retrospective or prospective comparative studies. Management of metastatic CRC QIs accounted for the fewest studies and lowest level of supporting evidence.

3.2 Delphi consensus process

Workshop participants (n = 12) represented a core CRC MDT membership including surgery (n = 3), medical and radiation oncology (n = 4), pathology (n = 1), radiology (n = 1), and nursing (n = 3). Participants assessed the 93 QIs for relevance to modern Australian CRC care delivery. A total of 49 QIs were retained; 24 were excluded due to advances in current clinical practice or technology since publication of those QIs, lack of relevance to Australian clinical practice, basic practice, and not relevant for measurement, or beyond the control of MDT members. Only four QIs were modified to include the latest technique or guideline, and 31 QIs were merged into 11 QIs, for example, workshop participants suggested the combination of specific post-operative complications be grouped into one QI. Some of the QIs that were specific to a treatment or tumor location or grade were modified to be applied more broadly, for example, some QIs for the management of metastatic CRC were extracted from an article that focussed specifically on CRC patients undergoing hepatic resection⁴¹ so participants amended the original wording of these QIs to include oligometastatic disease beyond the liver. One new QI was developed to address the delivery and timing of stoma follow-up care.

Participants of the second Delphi round (n = 44) included surgeons (n = 14, 32%), radiation oncologists (n = 10, 23%), medical oncologists (n = 9, 20%), nurses (n = 8, 18%), gastroenterologists (n = 1, 2%), diagnostic radiologist (n = 1, 2%), and an anatomical pathologist (n = 1, 2%). The majority of participants were CRC MDT members from New South Wales (n = 31, 70%), with the rest from Victoria, Northern Territory, Western Australia, South Australia, and Queensland. Participants had an even spread of clinical experience level; <5 years (n = 12, 27%), 5–15 years (n = 17, 39%), and >5 years (n = 15, 34%). There were 26 QIs with a level of concordance >80%. This prioritized set included QIs within each Domain of Care as demonstrated in Figure 2. Table 1 presents the final set of prioritised CRC QIs and their characteristics, categorized into the Domains of Care. Almost all QIs in the final set were from the literature review, with the new indicator identified in Round 1 of the Delphi also prioritized. All three QIs with the highest scientific soundness score were retained in the final set. All disciplines contribute to QIs selected in the priority set. There are a mix of process and outcome QIs in the final set, and only one structure QI. The free text comments included suggestions to re-word QIs or debate the benchmark in timeliness measures. None of the QIs that were prioritized and included in the final set were subjects of debate.

TABLE 1. Final set of colorectal cancer (CRC) quality indicators (QIs) and their characteristics. This table was created by the authors.

Indicator description	Numerator	Denominator	Donabedian category	Health professionals	Evidence grading	Related references^a	Concordance
Diagnosis, staging, and treatment planning (five QIs)
Pre-operative visualization of the total colon	Number of CRC patients who had preoperative visualization of total colon consisting of a complete colonoscopy to the caecum or distal ileum (or in case of incomplete colonoscopy, CT colonography as next line)	Number of CRC patients who underwent surgery	Process	Gastroenterologist Colorectal Surgeon Radiologist	1	26, 34, 37, 42-52	93%
Imaging prior to definitive treatment	Number of colon cancer patients who had CT (chest, abdomen, and pelvis examinations completed prior to definitive treatment	Number of colon cancer patients who received definitive treatment	Process	Gastroenterologist Colorectal Surgeon Radiologist	2	26, 34, 37, 39, 43, 45, 48, 51-57	100%
Imaging prior to definitive treatment	Number of rectal cancer patients who had high-resolution MRI examination prior to definitive treatment	Number of rectal cancer patients who received definitive treatment	Process/*s	Colorectal Surgeon Radiation Oncologist Radiologist	2	26, 34, 37, 39, 43, 45, 48, 51-57	100%
Staging completed prior to definitive treatment	Number of CRC patients who were staged prior to definitive treatment	Number of CRC patients who received definitive treatment	Process	Radiologist Colorectal Surgeon	2	26, 39, 42, 45, 48-50, 56, 58-63	100%
A comprehensive and standardized surgical pathology report	Number of CRC patients with a completed surgical pathology report according to the ANZ RCP Colorectal Cancer Structured Reporting Protocol (2016)	Number of CRC patients who underwent surgery	Structure/*p	Pathologist Gastroenterologist Colorectal Surgeon	3	26, 36, 37, 39, 44, 48, 52, 55-57, 61, 64-67	98%
Radiation oncology consult for rectal cancer patients	Number of stage II-IV rectal cancer patients who were referred for a consultation with a Radiation Oncologist	Number of stage II-IV rectal cancer patients	Process	Colorectal Surgeon Gastroenterologist	4	36, 49, 56, 61, 68	89%
Neoadjuvant and adjuvant Therapy (six QIs). *For the purpose of this table, the neo-adjuvant and adjuvant domains were combined as there were two QIs common to both domains.
Neoadjuvant chemoradiotherapy for local regionally advanced rectal cancer patients	Number of local regionally advanced rectal cancer patients who received neoadjuvant chemoradiotherapy	Number of local regionally advanced rectal cancer patients who underwent surgery	Process	Colorectal Surgeon Medical Oncologist Radiation Oncologist	2	26, 34, 36, 42, 46-48, 50, 52, 55, 60, 63, 69, 70	98%
Timeliness of neoadjuvant therapy	Number of CRC patients who received neoadjuvant chemoradiotherapy within 6 weeks of diagnosis(54	Number of CRC patients who received neoadjuvant therapy	Process/*s	Colorectal Surgeon Medical Oncologist Radiation Oncologist	3	40, 49, 54, 63-65	84%
Adjuvant chemotherapy for stage II-III CRC patients	Number of stage II-III CRC patients who received adjuvant chemotherapy	Number of stage II-III CRC patients	Process	Colorectal Surgeon Radiation Oncologist	2	26, 36, 47, 48, 50, 52, 58, 63, 69, 71-75	84%
Timeliness of adjuvant chemotherapy	Number of stage II-III CRC patients who received adjuvant chemotherapy within 8 weeks^{26, 36, 43, 49, 50, 54, 56, 64, 65, 76} following surgical resection OR within 4 months following stage III colon cancer diagnosis³⁵ and 9 months following stage II-III rectal cancer diagnosis³⁵	Number of stage II-III CRC patients who received adjuvant chemotherapy	Process	Medical Oncologist Nurse Care Coordinator	2	26, 35, 36, 43, 48-50, 54, 56, 58, 64, 65, 76-78	80%
Evidence-based regimen of chemotherapy	Number of CRC patients who received a chemotherapy regimen (consistent with published dose/cycles guidelines	Number of CRC who received chemotherapy	Process	Medical Oncologist	3	36, 49, 64, 65	95%
Mortality following chemotherapy or radiotherapy	Number of CRC patients who died following chemotherapy or radiotherapy who died within 30 and 90 days of treatment	Number of CRC patients who received chemotherapy or radiotherapy	Outcome	Medical Oncologist Radiation Oncologist	4	34	84%
Perioperative optimization (two QIs)
Stoma care follow-up appointment	Number of CRC patients who attended a stoma care follow-up appointment within 2 weeks of surgery involving stoma creation	Number of CRC patients who underwent surgery that involved stoma creation	Process	Stoma Nurse Colorectal Surgeon	3	*This indicator was developed in Round 1	84%
Thromboembolic prophylaxis	Number of CRC patients who underwent surgical resection with correct thromboembolic prophylaxis	Number of CRC patients who underwent surgery	Process	Colorectal Surgeon	2	41, 69, 79	84%
Surgery (nine QIs)
Complete resection (incl. CME/TME)	Number of CRC patients who underwent surgery with complete resection (R0 resection, circumferential resection margin, distal margin, completeness/intact TME plane)	Number of CRC patients who underwent surgical resection	Outcome	Colorectal Surgeon	2	26, 34, 37, 38, 47-51, 55, 58, 60, 61, 69, 70, 78, 80-87	95%
Adequate lymph node retrieval	Number of CRC patients with ≥12^{26, 34, 35, 43, 44, 47, 50, 52, 70, 88-94} resected lymph nodes	Number of CRC patients who underwent surgical resection	Process	Colorectal Surgeon	1	26, 34, 35, 37, 38, 42-48, 50, 52, 54, 55, 58-60, 64-66, 70, 71, 74, 75, 77, 88-114	89%
Intraoperative tumor perforation, puncture, or laceration	Number of CRC cancer patients with intraoperative local tumor perforation, puncture, or laceration resulting in a deviation from the planned resection or involving an additional procedure	Number of CRC patients who underwent surgery	Outcome	Colorectal Surgeon	2	48, 55, 60, 70, 115-117	89%
Unplanned re-operation	Number of CRC patients who returned to theatre after surgery for an unplanned reoperation (washout of abdomen, small bowel resection, further colorectal resection, drainage of intra-abdominal abscess, division of adhesions, stoma formation or operation on a stoma, and wound complications) within 30 days following surgery^{34, 118-120}	Number of CRC patients who underwent surgery	Outcome	Colorectal Surgeon	2	34, 38, 45, 48, 51, 55, 56, 79, 118-121	82%
Unplanned re-admission	Number of CRC patients who were re-hospitalized within 30 days⁹⁵ of hospital discharge	Number of CRC patients who underwent surgery	Outcome	Colorectal Surgeon	2	95, 118, 121-124	80%
Post-operative complications	Number of CRC patients who have postoperative complications (i.e., anastomotic leak, infection of the surgical site, abdominal wall dehiscence, deep vein thrombosis, pneumonia, sepsis, hemorrhage or hematoma, blood transfusion, metabolic derangements, urinary dysfunction, bowel dysfunction, fever (>38°C, duration > 2 days, multiple organ failure, neurological/psychiatric complications, cardiac complications, catheter-related bloodstream infection, transfusion reaction) within 30 days following surgery	Number of CRC patients who underwent surgery	Outcome	Colorectal Surgeon	2	34, 37, 38, 45, 48, 51, 52, 55, 56, 59, 60, 66, 69, 70, 74, 79, 111, 116-118, 122, 123, 125-133	98%
Post-operative mortality	Number of CRC patients who died within 30 days^{34, 37, 40, 52} following surgery	Number of CRC patients who underwent surgery	Outcome	Colorectal Surgeon	1	26, 34, 37, 38, 44-48, 51, 52, 54, 55, 59, 60, 66, 70, 72, 81, 86, 108, 111, 116, 118, 119, 121, 123, 124, 128, 132, 134-142)	93%
Overall survival	Number of CRC patients who have survived at (t following surgery *t = 1 year¹⁴³, 2 years¹⁴³, 5 years^{40, 41, 48, 54, 56, 60, 86, 143}	Number of CRC patients who underwent surgery	Outcome	Colorectal Surgeon	2	40, 41, 48, 54-56, 60, 86, 125, 142-144	84%
Recurrence	Number of CRC patients who have had a recurrence within (t following surgery *t = 2 years⁶⁶, 5 years^{48, 54-56, 60, 70, 74}	Number of CRC patients who received surgery	Outcome	Colorectal Surgeon	2	48, 54-56, 60, 66, 70, 74, 144	89%
Post-treatment surveillance (two QIs)
Colonoscopy surveillance	Number of curatively treated CRC patients who received post-operative colonoscopy within 1 year of surgical treatment	Number of CRC patients who received curative treatment	Process	Colorectal Surgeon	2	55, 58, 59, 68	80%
Colonoscopy surveillance	The number of CRC patients who received postoperative colonoscopy within 6 months after an incomplete preoperative colonoscopy (incl. obstructing lesion at the time of presentation	Number of CRC patients who had an incomplete preoperative colonoscopy	Process	Colorectal Surgeon	3	48, 50	80%
Documented follow-up history and physical	Number of CRC patients with documented history/physical examination, and postoperative visits every 3 months for 2 years, then every 6 months in years 2–5 (total of 5 years)	Number of CRC patients	Process	Colorectal Surgeon Radiation Oncologist	3	50	91%
Metastatic (one QI)
Oligometastatic liver/lung resection or ablative therapy	Number of CRC patients with oligometastatic liver/lung cancer who underwent resection or ablative therapy	Number of CRC patients with oligometastatic liver/lung cancer	Process	Liver/Colorectal Surgeon Medical Oncologist Cardiothoracic Surgeon Radiation Oncologist	4	145	81%
Supportive Care (one QI)
Discharge letter details the treatment summary, follow-up, and monitoring	Number of CRC patients with a discharge letter stating details of treatment summary, follow-up, and long-term monitoring, provided to the patient/carer on the day of discharge⁶¹ and referring physician within 4 weeks³⁹	Number of CRC patients	Process	Colorectal Surgeon	4	61	80%

^a References identified in the literature discuss the development, selection, validation, and review of the quality indicator.
Process/*p.
Structure/*s.
Outcome/*o.
t = specified time period.

4 DISCUSSION

This study identified a large number of existing CRC QIs that have been studied internationally and has prioritized the most clinically relevant QIs for use by Australian MDTs to measure and monitor their performance. Previous literature has described systematic reviews of multidisciplinary CRC QIs conducted in European settings, where one study included a consensus process.^{25, 26} In addition to updating the literature, the current study has addressed gaps presented in the two previous reviews and has focussed on quality measurement in the Australian MDT setting. Bianchi et al.’s²⁶ review and two-step Delphi consensus process with a 12-person Swiss expert panel aimed to develop QIs for use in a population based setting. The working group in Bianchi et al.’s study proposed new QIs prior to the consensus process. Only 10 QIs identified in Bianchi et al.’s²⁶ paper were also prioritized in the final set of 26 QIs for this study. Keikes et al.’s²⁵ review included 41 studies and found a total sum of 349 QIs across these studies but didn't specify the number of unique indicators. In contrast, our review included 118 studies and found a total sum of 383 QIs, with a total of 93 unique QIs. This highlighted the importance of the second component of this study, in particular, the multidisciplinary involvement in the modified-Delphi. Neither study rated quality indicators and noted an urgent need to develop a validated system for this purpose, whilst our study utilized a grading system to calculate each QI's scientific soundness.

The majority of QIs identified in this systematic review were categorized as process indicators, although many of the QIs identified were multidimensional. It is recognized that many of the process indicators are significantly influenced by structural factors including the availability of the physical facility, equipment, and human resources. For example, “timeliness of CT and/or an MRI for staging”, which is commonly categorized as a process indicator may be influenced by a number of structural indicators including, the availability of staff and MRI machines. The relationships among structural, process, and outcome indicators are important considerations in the accurate interpretation of quality measurement. A significant number of QIs do not have high-quality studies to support their validity and impact on quality improvement, particularly in neoadjuvant therapy, palliative management of metastatic CRC, and supportive care. Many QIs currently used by organizations have been implemented without a high level of evidence. It is possible that these QIs have been selected on the basis of consensus, local practice guidelines, and the convenience and availability of data. However, some studies debated the reliability of extensively endorsed QIs, finding no association between the QI and improved patient outcomes or that QI results are influenced by patient/hospital level factors and should be used with caution.^{43, 92, 93, 95} This study presents the current state of evidence supporting the use of CRC QIs and highlights important QIs for further feasibility and validity testing.

The current review found, as did Keikes et al.,²⁵ that most QIs were surgical measures. This may be indicative of the number of well-established national and internationally co-ordinated agencies dedicated to collecting analyzing standardized surgical data.^{37, 38} Another contributing factor may be that assessment of outcomes pertaining to surgery is well-defined and can often be measured immediately following the surgery whilst those pertaining to other therapies may take much longer to determine and may address outcomes that are harder to measure (delayed side effects, quality of life outcomes and patient experience). In areas such as ‘supportive care’ where there are a very small number of QIs, the data required to calculate additional QIs may not be routinely collected by health services or require patient reported data. As electronic medical records evolve, the integration of patient reported outcome measures will be fundamental to capturing and standardizing the reporting of patient health status.¹⁴⁶ Further research is needed to identify priority quality of life indicators for patients and how these are most effectively analyzed and translated into quality improvement by local MDTs.

Comparing the core treatment modalities, there was a lack of radiation therapy-specific QIs. This may be due to the non-tumor-specific nature of radiation therapy quality assurance activities that are typically undertaken at a service level. However with the evolution in the roles of radiation therapy (in rectal cancer) and modern technologies, there is an opportunity for disease-specific radiation therapy QIs, particularly in the areas of treatment waiting times, radiation therapy target volume delineation, dose prescription, plan evaluation, and image guidance during treatment delivery. There may also be an opportunity to develop QIs in diagnostic disciplines (radiology, nuclear medicine, and anatomical pathology) to facilitate accurate staging as well as in the areas of informed clinical treatment decisions by patients and HPs, and timely access to treatment. These gaps could be addressed by developing QIs from nationally and internationally published and endorsed clinical guidelines from both cancer organizations and discipline-specific guidelines.

Despite a large number of surgical QIs, over half of QIs in the final set pertain to care delivery by multiple disciplines and HPs thus highlighting the multidisciplinary nature of CRC care, recognizing that an MDT focus on service quality improvement is important. The involvement of HP experts from each discipline delivering CRC care is a strength and should maximize the applicability of the final QI set presented. Future work will focus on the examination of the feasibility and construct validity of the QI as these will be critical to implementation. Furthermore, it will be important to establish and test local benchmarks.

4.1 Limitations

A limitation of extracting QIs from the literature is the time lag from QI development and validation to publication. This may have resulted in missing QIs that describe more recent diagnostic and treatment techniques. Whilst a comprehensive list of quality and safety agencies was purposefully selected for the search, it may not encompass all possible organizations internationally. This study specifically prioritized QIs based on clinical importance and relevance to the Australian MDT setting. Other common measures in Delphi surveys such as ‘feasibility’ were not included in this process. The current feasibility of measuring QIs will be determined in the next project phase with real-world clinical data as opposed to participant opinions of feasibility from different Australian states and data systems. Furthermore, a clinically important QI set may form a basis for which data points should be considered standardization and collection in the future, rather than constructing QIs on the basis of data already collected.

5 CONCLUSION

This study has identified a priority set of evidence-based and clinically important QIs that could be used by CRC MDTs to measure, describe and monitor the quality of care they deliver. Our findings highlight that an MDT-based approach to quality measurement and monitoring addresses the full spectrum of the CRC patient pathway and experience. Future work will focus on the feasibility of implementation and examining the utility of this priority set, which has the potential to facilitate the monitoring of CRC patient care delivery at the tumor stream level, identify potential variation in care, provide standards of care to benchmark against through self-regulation, direct quality improvement activities and embed a culture of quality improvement within the service.

AUTHOR CONTRIBUTIONS

Candice Donnelly, Puma Sundaresan, Kim-Lin Chiew, and Shalini Vinod developed the search strategy. Candice Donnelly, Puma Sundaresan, Michelle Or, and Mathushan Thevaraja completed the components of the systematic review screening and analysis. Candice Donnelly and Puma Sundaresan conducted the consensus process with contributions from Shalini Vinod, Tim Shaw, and Anna Janssen. Candice Donnelly conducted the Delphi analysis. Candice Donnelly drafted the manuscript with contributions from Puma Sundaresan, Michelle Or, James Toh, Shalini Vinod, Anna Janssen, Nimalan Pathma-Nathan, Tim Shaw, and Paul Harnett.

ACKNOWLEDGMENTS

Candice Donnelly is undertaking a PhD and would like to thank her supervision team (Tim Shaw, Shalini Vinod, Paul Harnett, and Anna Janssen) for their guidance. The research team would like to acknowledge the Delphi participants. This work was funded by the Radiation Oncology Network, Western Sydney Local Health District, and the Sydney West Transitional Cancer Research Centre (SWTCRC). The SWTCRC was funded by The Cancer Institute NSW under the Translational Cancer Research Centres (TRC) [15/TRC/1-01]. Funding provided salary support for Candice Donnelly.

Open access publishing facilitated by The University of Sydney, as part of the Wiley - The University of Sydney agreement via the Council of Australian University Librarians.

CONFLICT OF INTEREST

The authors declare that they have no conflict of interest.

ETHICS STATEMENT

Permission to conduct this study was granted by the University of Sydney Human Research Ethics Committee [HREC 2021/778].

Open Research

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author, Candice Donnelly, upon reasonable request.

Supporting Information

REFERENCES

1 Cancer Australia. National cancer control indicators. Available from: https://ncci.canceraustralia.gov.au/cancer-types/colorectal-cancer
Google Scholar
2 Australian Institute of Health and Welfare. Cancer data in Australia Canberra: AIHW. 2021. Available from: https://www.aihw.gov.au/reports/cancer/cancer-data-in-australia
Google Scholar
3 Cancer Council Australia. Clinical Practice Guidelines for the Prevention, Early Detection and Management of Colorectal Cancer. Cancer Council Australia; 2017.
Google Scholar
4Australi Cancer, a. All about multidisciplinary care Available from: https://www.canceraustralia.gov.au/clinical-best-practice/multidisciplinary-care/all-about-multidisciplinary-care
Google Scholar
5 National Institute for Health and Care Excellence. Colorectal Cancer: NICE Guideline United Kingdom: NICE; 2020. Available from: www.nice.org.uk/guidance/ng151
Google Scholar
6Selby P, Popescu R, Lawler M, Butcher H, Costa A. The value and future developments of multidisciplinary team cancer care. Am Soc Clin Oncol Educ Book. 2019; 39: 332-340.
10.1200/EDBK_236857
PubMed Google Scholar
7Lan Y-T, Lin J-K, Jiang J-K. Effects of a multidisciplinary team on colorectal cancer treatment. Formos J Surg. 2015; 48(5): 145-150.
10.1016/j.fjs.2015.07.003
Google Scholar
8Specchia ML, Frisicale EM, Carini E, et al. The impact of tumor board on cancer care: evidence from an umbrella review. BMC Health Serv Res. 2020; 20(1): 73.
10.1186/s12913-020-4930-3
PubMed Web of Science® Google Scholar
9Pai S, Blaisdell D, Brodie R, et al. Defining current gaps in quality measures for cancer immunotherapy: consensus report from the Society for Immunotherapy of Cancer (SITC) 2019 Quality Summit. J Immunother Cancer. 2020; 8(1):e000112.
10.1136/jitc-2019-000112
Web of Science® Google Scholar
10Menon M, Cunningham C, Kerr D. Addressing unwarranted variations in colorectal cancer outcomes: a conceptual approach. Nat Rev Clin Oncol. 2016; 13(11): 706-712.
10.1038/nrclinonc.2016.94
PubMed Web of Science® Google Scholar
11Mainz J. Quality indicators: essential for quality improvement. Int J Qual Health Care. 2004; 16: i1-i2.
10.1093/intqhc/mzh036
PubMed Web of Science® Google Scholar
12Donabedian A. The quality of care. How can it be assessed? JAMA. 1988; 260(12): 1743-1748.
10.1001/jama.1988.03410120089033
CAS PubMed Web of Science® Google Scholar
13Donabedian A. Evaluating the quality of medical care. Milbank Q. 1966; 44(3): 166-206.
10.2307/3348969
Web of Science® Google Scholar
14 Cancer Institute NSW. Cancer Control in NSW: Statewide Report 2019. Cancer Institute NSW; 2020.
Google Scholar
15Chazapis M, Gilhooly D, Smith A, et al. Perioperative structure and process quality and safety indicators: a systematic review. Br J Anaesth. 2018; 120(1): 51-66.
10.1016/j.bja.2017.10.001
CAS PubMed Web of Science® Google Scholar
16Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care. 2003; 15(6): 523-530.
10.1093/intqhc/mzg081
PubMed Web of Science® Google Scholar
17Wollersheim H, Hermens R, Hulscher M, et al. Clinical indicators: development and applications. Neth J Med. 2007; 65(1): 15-22.
CAS PubMed Web of Science® Google Scholar
18Albert JM, Das P. Quality assessment in oncology. Int J Radiat Oncol Biol Phys. 2012; 83(3): 773-781.
10.1016/j.ijrobp.2011.12.079
PubMed Web of Science® Google Scholar
19Meyer GS, Nelson EC, Pryor DB, et al. More quality measures versus measuring what matters: a call for balance and parsimony. BMJ Qual Saf. 2012; 21(11): 964.
10.1136/bmjqs-2012-001081
PubMed Web of Science® Google Scholar
20Copnell B, Hagger V, Wilson SG, Evans SM, Sprivulis PC, Cameron PA. Measuring the quality of hospital care: an inventory of indicators. Int Med J. 2009; 39(6): 352-360.
10.1111/j.1445-5994.2009.01961.x
CAS PubMed Web of Science® Google Scholar
21Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007; 7(1): 1-6.
10.1186/1472-6947-7-16
PubMed Web of Science® Google Scholar
22van den Bosch CM, Hulscher ME, Natsch S, Gyssens IC, Prins JM, Geerlings SE. Development of quality indicators for antimicrobial treatment in adults with sepsis. BMC Infect Dis. 2014; 14(1): 345.
10.1186/1471-2334-14-345
PubMed Web of Science® Google Scholar
23CBO: Kwaliteitsinstituut voor de Gezondheidszorg CBO hvwhwcnte-b-w-r-pr.
Google Scholar
24Chiew K, Jalaludin B, Sundaresan P, Vinod S. P3. 15-07 a literature review and assessment of lung cancer quality indicators. J Thorac Oncol. 2018; 13(10): S993.
10.1016/j.jtho.2018.08.1883
Web of Science® Google Scholar
25Keikes L, Koopman M, Tanis PJ, Lemmens V, Punt CJA, van Oijen MGH. Evaluating the scientific basis of quality indicators in colorectal cancer care: a systematic review. Eur J Cancer. 2017; 86: 166-177.
10.1016/j.ejca.2017.08.034
PubMed Web of Science® Google Scholar
26Bianchi V, Spitale A, Ortelli L, Mazzucchelli L, Bordoni A. Quality indicators of clinical cancer care (QC3) in colorectal cancer. BMJ Open. 2013; 3(7):e002818.
10.1136/bmjopen-2013-002818
PubMed Web of Science® Google Scholar
27Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995; 311(7001): 376-380.
10.1136/bmj.311.7001.376
CAS PubMed Web of Science® Google Scholar
28Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000; 32(4): 1008-1015.
10.1046/j.1365-2648.2000.t01-1-01567.x
CAS PubMed Web of Science® Google Scholar
29Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011; 6(6):e20476.
10.1371/journal.pone.0020476
CAS PubMed Web of Science® Google Scholar
30De Villiers MR, De Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005; 27(7): 639-643.
10.1080/13611260500069947
PubMed Web of Science® Google Scholar
31Ives J, Dunn M, Molewijk B, et al. Standards of practice in empirical bioethics research: towards a consensus. BMC Med Ethics. 2018; 19(1): 1-20.
10.1186/s12910-018-0304-3
PubMed Web of Science® Google Scholar
32Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2): 377-381.
10.1016/j.jbi.2008.08.010
PubMed Web of Science® Google Scholar
33Moher D, Liberati A, Tetzlaff J, Altman D. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009; 339:b2535.
10.1136/bmj.b2535
PubMed Web of Science® Google Scholar
34 NHS Scotland. Colorectal cancer – clinical quality performance indicators. In: Scottish Cancer Taskforce. v3.0 ed. Healthcare Improvement Scotland; 2017; 1-33.
Google Scholar
35Desch C, McNiff K, Schneider E, et al. American Society of Clinical Oncology/National Comprehensive Cancer Network quality measures. J Clin Oncol. 2008; 26(21): 3631-3637.
10.1200/JCO.2008.16.5068
PubMed Web of Science® Google Scholar
36Malin J, Schneider E, Epstein A, Adams J, Emanuel EJ, Kahn KL. Results of the National Initiative for Cancer Care Quality: how can we improve the quality of cancer care in the United States? J Clin Oncol. 2006; 24(4): 626-634.
10.1200/JCO.2005.03.3365
PubMed Web of Science® Google Scholar
37Van Leersum N, Snijders H, Henneman D, et al. The dutch surgical colorectal audit. Eur J Surg Oncol. 2013; 39(10): 1063-1070.
10.1016/j.ejso.2013.05.008
CAS PubMed Web of Science® Google Scholar
38Dagher H, Ahern S, Salimi F, et al. The bi-national colorectal cancer audit report 2019. Colorectal Surgical Society of Australia and New Zealand; 2019.
Google Scholar
39 Commission on Cancer . The National Accreditation Program for Rectal Cancer Standards Manual. American College of Surgeons; 2017.
Google Scholar
40Walpole E, Theile D, Philpot S, Youl P. Development and implementation of a cancer quality index in Queensland, Australia: a tool for monitoring cancer care. J Oncol Pract. 2019; 15(7): 636-643.
10.1200/JOP.18.00372
Web of Science® Google Scholar
41Dixon E, Armstrong C, Maddern G, et al. Development of quality indicators of care for patients undergoing hepatic resection for metastatic colorectal cancer using a Delphi process. J Surg Res. 2009; 156(1): 32-38.
10.1016/j.jss.2009.03.084
PubMed Web of Science® Google Scholar
42van der Geest LG, Krijnen P, Wouters MW, et al. Improved guideline compliance after a 3-year audit of multidisciplinary colorectal cancer care in the western part of the Netherlands. J Surg Oncol. 2012; 106(1): 1-9.
10.1002/jso.23038
PubMed Web of Science® Google Scholar
43McConnell Y, Inglis K, Porter G. Timely access and quality of care in colorectal cancer: are they related? Int J Qual Health Care. 2010; 22(3): 219-228.
10.1093/intqhc/mzq010
PubMed Web of Science® Google Scholar
44Porter G, Urquhart R, Kendell C, Bu J, McConnell Y, Grunfeld E. Timely access and quality of care in colorectal cancer: a population-based cohort study using administrative data. BMC Res Notes. 2013; 6: 355.
10.1186/1756-0500-6-355
PubMed Google Scholar
45Gooiker GA, Kolfschoten NE, Bastiaannet E, et al. Evaluating the validity of quality indicators for colorectal cancer care. J Surg Oncol. 2013; 108(7): 465-471.
10.1002/jso.23420
PubMed Web of Science® Google Scholar
46Kolfschoten NE, Gooiker GA, Bastiaannet E, et al. Combining process indicators to evaluate quality of care for surgical patients with colorectal cancer: are scores consistent with short-term outcome? BMJ Qual Saf. 2012; 21(6): 481-489.
10.1136/bmjqs-2011-000439
CAS PubMed Web of Science® Google Scholar
47Spitale A, Ortelli L, Mazzucchelli L, Bordoni A. Quality indicators of colorectal cancer care in southern Switzerland: results from a population-based study. Swiss Med Wkly. 2017; 147(4748).
PubMed Web of Science® Google Scholar
48Ludt S, Urban E, Eckardt J, et al. Evaluating the quality of colorectal cancer care across the interface of healthcare sectors. PLoS One. 2013; 8(5):e60947.
10.1371/journal.pone.0060947
CAS PubMed Web of Science® Google Scholar
49Malafa MP, Corman MM, Shibata D, Siegel EM, Lee JH, Jacobsen PB. The Florida initiative for quality cancer care: a regional project to measure and improve cancer care. Cancer Control. 2009; 16(4): 318-327.
10.1177/107327480901600406
PubMed Web of Science® Google Scholar
50Jackson GL, Zullig LL, Zafar SY, et al. Using NCCN clinical practice guidelines in oncology to measure the quality of colorectal cancer care in the veterans health administration. J Natl Compr Canc Netw. 2013; 11(4): 431-441.
10.6004/jnccn.2013.0058
PubMed Web of Science® Google Scholar
51Jannasch O, Udelnow A, Romano G, et al. International quality assurance project in colorectal cancer—unifying diagnostic and histopathological evaluation. Langenbecks Arch Surg. 2014; 399(4): 473-479.
10.1007/s00423-014-1176-8
CAS PubMed Web of Science® Google Scholar
52Mathoulin-Pelissier S, Becouarn Y, Belleannee G, et al. Quality indicators for colorectal cancer surgery and care according to patient-, tumor-, and hospital-related factors. BMC Cancer. 2012; 12: 297.
10.1186/1471-2407-12-297
PubMed Web of Science® Google Scholar
53Camps C, Albanell J, Anton A, et al. Quality indicators to assure and improve cancer care in Spain using the Delphi technique. J Natl Compr Canc Netw. 2016; 14(5): 553-558.
10.6004/jnccn.2016.0063
PubMed Web of Science® Google Scholar
54Chung K, Chen L, Chang Y, Chang Y, Lai M. Application of the analytic hierarchy process in the performance measurement of colorectal cancer care for the design of a pay-for-performance program in Taiwan. Int J Qual Health Care. 2013; 25(1): 81-91.
10.1093/intqhc/mzs070
PubMed Web of Science® Google Scholar
55Demetter P, Ceelen W, Danse E, et al. Quality of care indicators in rectal cancer. Acta Gastroenterol Belg. 2011; 74(3): 445-450.
CAS PubMed Web of Science® Google Scholar
56Khare SR, Batist G, Bartlett G. Identification of performance indicators across a network of clinical cancer programs. Oncol. 2016; 23(2): 81-90.
CAS Google Scholar
57Brady J, Xu Z, Scarberry K, et al. Evaluating the current status of rectal cancer care in the US: where we stand at the start of the Commission on Cancer's National Accreditation Program for Rectal Cancer. J Am Coll Surg. 2018; 226(5): 881-890.
10.1016/j.jamcollsurg.2018.01.057
PubMed Web of Science® Google Scholar
58Hayman A, Chang E, Molokie R, Kahng L, Prystowsky JB, Bentrem DJ. Assessing compliance with national quality measures to improve colorectal cancer care at the VA. Am J Surg. 2010; 200(5): 572-576.
10.1016/j.amjsurg.2010.07.008
PubMed Web of Science® Google Scholar
59Helewa RM, Turner D, Wirtzfeld D, et al. Does geography influence the treatment and outcomes of colorectal cancer? A population-based analysis. World J Surg Oncol. 2013; 11: 140.
10.1186/1477-7819-11-140
PubMed Web of Science® Google Scholar
60Stelzner S, Hellmich G, Haroske G, Puffer E, Jackisch T, Witzigmann H. Practicability of quality goals for the treatment of rectal cancer. Int J Colorectal Dis. 2010; 25(9): 1093-1102.
10.1007/s00384-010-0965-y
PubMed Web of Science® Google Scholar
61Young JM, Masya LM, Solomon MJ, Shepherd HL. Identifying indicators of colorectal cancer care coordination: a Delphi study. Colorectal Dis. 2014; 16(1): 17-25.
10.1111/codi.12399
CAS PubMed Web of Science® Google Scholar
62Leal TB, Holden T, Cavalcante L, et al. Colon cancer staging in vulnerable older adults: adherence to national guidelines and impact on survival. Ann Hematol Oncol. 2014; 1(3).
PubMed Google Scholar
63Corbellini C, Andreoni B, Ansaloni L, et al. Reliability and validity assessment of administrative databases in measuring the quality of rectal cancer management. Tumori. 2018; 104(1): 51-59.
10.5301/tj.5000708
PubMed Web of Science® Google Scholar
64Siegel EM, Jacobsen PB, Lee JH, et al. Florida initiative for quality cancer care: improvements on colorectal cancer quality of care indicators during a 3-year interval. J Am Coll Surg. 2014; 218(1): 16-25.
10.1016/j.jamcollsurg.2013.09.008
PubMed Web of Science® Google Scholar
65Siegel EM, Jacobsen PB, Malafa M, et al. Evaluating the quality of colorectal cancer care in the state of Florida: results from the Florida initiative for quality cancer care. J Oncol Pract. 2012; 8(4): 239-245.
10.1200/JOP.2011.000477
PubMed Google Scholar
66Vergara-Fernandez O, Swallow CJ, Victor JC, et al. Assessing outcomes following surgery for colorectal cancer using quality of care indicators. Can J Surg. 2010; 53(4): 232-240.
PubMed Web of Science® Google Scholar
67Ruffo G, Barugola G, Rossini R, Sartori CA. Colorectal surgery in Italy. Criteria to identify the hospital units and the tertiary referral centers entitled to perform it. Updates Surg. 2016; 68(2): 123-128.
10.1007/s13304-016-0372-1
PubMed Web of Science® Google Scholar
68Krzyzanowska MK, Barbera L, Elit L, et al. Identifying population-level indicators to measure the quality of cancer care for women. Int J Qual Health Care. 2011; 23(5): 554-564.
10.1093/intqhc/mzr043
PubMed Web of Science® Google Scholar
69Habib MR, Solomon MJ, Young JM, Armstrong BK, O'Connell D, Armstrong K. Evidence-based and clinical outcome scores to facilitate audit and feedback for colorectal cancer care. Dis Colon Rectum. 2009; 52(4): 616-623.
10.1007/DCR.0b013e31819edb7d
PubMed Web of Science® Google Scholar
70Merkel S, Klossek D, Gohl J, Papadopoulos T, Hohenberger W, Hermanek P. Quality management in rectal carcinoma: what is feasible? Int J Colorectal Dis. 2009; 24(8): 931-942.
10.1007/s00384-009-0736-9
PubMed Web of Science® Google Scholar
71McColl RJ, McGahan CE, Cai E, et al. Impact of hospital volume on quality indicators for rectal cancer surgery in British Columbia, Canada. Am J Surg. 2017; 213(2): 388-394.
10.1016/j.amjsurg.2016.07.007
PubMed Web of Science® Google Scholar
72Raoof M, Zafar SN, Ituarte PHG, Krouse RS, Melstrom K. Using a lymph node count metric to identify underperforming hospitals after rectal cancer surgery. J Surg Res. 2019; 236: 216-223.
10.1016/j.jss.2018.11.040
PubMed Web of Science® Google Scholar
73Samuel CA, Zaslavsky AM, Landrum MB, Lorenz K, Keating NL. Developing and evaluating composite measures of cancer care quality. Med Care. 2015; 53(1): 54-64.
10.1097/MLR.0000000000000257
PubMed Web of Science® Google Scholar
74Schneider PM, Vallbohmer D, Ploenes Y, et al. Evaluation of quality indicators following implementation of total mesorectal excision in primarily resected rectal cancer changed future management. Int J Colorectal Dis. 2011; 26(7): 903-909.
10.1007/s00384-011-1155-2
PubMed Web of Science® Google Scholar
75Spain P, Teixeira-Poit S, Halpern MT, et al. The National Cancer Institute Community Cancer Centers Program (NCCCP): sustaining quality and reducing disparities in guideline-concordant breast and colon cancer care. Oncologist. 2017; 22(8): 910-917.
10.1634/theoncologist.2016-0252
PubMed Web of Science® Google Scholar
76Yu S, Shabihkhani M, Yang D, et al. Timeliness of adjuvant chemotherapy for stage III adenocarcinoma of the aolon: a measure of quality of care. Clin Colorectal Cancer. 2013; 12(4): 275-279.
10.1016/j.clcc.2013.08.002
PubMed Web of Science® Google Scholar
77Mason MC, Chang GJ, Petersen LA, et al. National Quality Forum colon cancer quality metric performance: how are hospitals measuring up? Ann Surg. 2017; 266(6): 1013-1020.
10.1097/SLA.0000000000002003
PubMed Web of Science® Google Scholar
78Massarweh NN, Haynes AB, Chiang YJ, et al. Adequacy of the national quality forum's colon cancer adjuvant chemotherapy quality metric: is 4 months soon enough? Ann Surg. 2015; 262(2): 312-320.
10.1097/SLA.0000000000000859
PubMed Web of Science® Google Scholar
79Soria-Aledo V, Angel-Garcia D, Martinez-Nicolas I, Rebasa Cladera P, Cabezali Sanchez R, Pereira Garcia LF. Development and pilot study of an essential set of indicators for general surgery services. Cir Esp. 2016; 94(9): 502-510.
10.1016/j.ciresp.2016.06.009
PubMed Web of Science® Google Scholar
80Comber H, Sharp L, Timmons A, Keane FB. Quality of rectal cancer surgery and its relationship to surgeon and hospital caseload: a population-based study. Colorectal Dis. 2012; 14(10): 692-700.
10.1111/j.1463-1318.2012.03145.x
PubMed Web of Science® Google Scholar
81Denost Q, Saillour F, Masya L, et al. Benchmarking trial between France and Australia comparing management of primary rectal cancer beyond TME and locally recurrent rectal cancer (PelviCare Trial): rationale and design. BMC Cancer. 2016; 16: 1-7.
10.1186/s12885-016-2286-1
Web of Science® Google Scholar
82Gietelink L, Henneman D, van Leersum NJ, et al. The influence of hospital volume on circumferential resection margin involvement: results of the Dutch surgical colorectal audit. Ann Surg. 2016; 263(4): 745-750.
10.1097/SLA.0000000000001009
PubMed Web of Science® Google Scholar
83Merkel S, Weber K, Matzel KE, Agaimy A, Gohl J, Hohenberger W. Prognosis of patients with colonic carcinoma before, during and after implementation of complete mesocolic excision. Br J Surg. 2016; 103(9): 1220-1229.
10.1002/bjs.10183
CAS PubMed Web of Science® Google Scholar
84Atkinson S, Daly M, Midura E, et al. The effect of hospital volume on resection margins in rectal cancer surgery. J Surg Res. 2016; 204(1): 22-28.
10.1016/j.jss.2016.04.041
PubMed Web of Science® Google Scholar
85Denost Q, Faucheron J, Lefevre JH, et al. French current management and oncological results of locally recurrent rectal cancer. Eur J Surg Oncol. 2015; 41(12): 1645-1652.
10.1016/j.ejso.2015.09.019
CAS PubMed Web of Science® Google Scholar
86Massarweh NN, Hu CY, You YN, et al. Risk-adjusted pathologic margin positivity rate as a quality indicator in rectal cancer surgery. J Clin Oncol. 2014; 32(27): 2967-2974.
10.1200/JCO.2014.55.5334
PubMed Web of Science® Google Scholar
87Lichliter WE. Techniques in total mesorectal excision surgery. Clin Colon Rectal Surg. 2015; 28(1): 21-27.
10.1055/s-0035-1545066
PubMed Web of Science® Google Scholar
88Govindarajan A, Gonen M, Weiser MR, et al. Challenging the feasibility and clinical significance of current guidelines on lymph node examination in rectal cancer in the era of neoadjuvant therapy. J Clin Oncol. 2011; 29(34): 4568-4573.
10.1200/JCO.2011.37.2235
CAS PubMed Web of Science® Google Scholar
89Dillman RO, Aaron K, Heinemann FS, McClure SE. Identification of 12 or more lymph nodes in resected colon cancer specimens as an indicator of quality performance. Cancer. 2009; 115(9): 1840-1848.
10.1002/cncr.24185
PubMed Web of Science® Google Scholar
90Romanus D, Weiser MR, Skibber JM, et al. Concordance with NCCN colorectal cancer guidelines and ASCO/NCCN quality measures: an NCCN institutional analysis. J Natl Compr Canc Netw. 2009; 7(8): 895-904.
10.6004/jnccn.2009.0059
PubMed Web of Science® Google Scholar
91Wang J, Kulaylat M, Rockette H, et al. Should total number of lymph nodes be used as a quality of care measure for stage III colon cancer? Ann Surg. 2009; 249(4): 559-563.
10.1097/SLA.0b013e318197f2c8
PubMed Web of Science® Google Scholar
92Blaker H, Hildebrandt B, Riess H, et al. Lymph node count and prognosis in colorectal cancer: the influence of examination quality. Int J Cancer. 2015; 136(8): 1957-1966.
10.1002/ijc.29221
CAS PubMed Web of Science® Google Scholar
93Mathis KL, Green EM, Sargent DJ, Delaney C, Simmang CL, Nelson H. Surgical quality surrogates do not predict colon cancer survival in the setting of technical credentialing: a report from the prospective COST trial. Ann Surg. 2013; 257(1): 102-107.
10.1097/SLA.0b013e318260a8e6
PubMed Web of Science® Google Scholar
94Porter G, Urquhart R, Bu J, Johnson P, Rayson D, Grunfeld E. Improving nodal harvest in colorectal cancer: so what? Ann Surg Oncol. 2012; 19(4): 1066-1073.
10.1245/s10434-011-2073-9
PubMed Web of Science® Google Scholar
95Hechenbleikner E, Makary M, Samarov D, et al. Hospital readmission by method of data collection. J Am Coll Surg. 2013; 216(6): 1150-1158.
10.1016/j.jamcollsurg.2013.01.057
PubMed Web of Science® Google Scholar
96Bilchik A, Nissan A, Wainberg Z, et al. Surgical quality and nodal ultrastaging is associated with long-term disease-free survival in early colorectal cancer: an analysis of 2 international multicenter prospective trials. Ann Surg. 2010; 252(3): 467-476.
10.1097/SLA.0b013e3181f19767
PubMed Web of Science® Google Scholar
97McFadyen C, Lankshear S, Divaris D, et al. Physician level reporting of surgical and pathology performance indicators: a regional study to assess feasibility and impact on quality. Can J Surg. 2015; 58(1): 31-40.
10.1503/cjs.004314
PubMed Web of Science® Google Scholar
98Nir S, Greenberg R, Shacham-Shmueli E, White I, Schneebaum S, Avital S. Number of retrieved lymph nodes and survival in node-negative patients undergoing laparoscopic colorectal surgery for cancer. Tech Coloproctol. 2010; 14(2): 147-152.
10.1007/s10151-010-0578-z
CAS PubMed Web of Science® Google Scholar
99Pappas DP, Garbus JE, Feuerman M, Reed WP. Improving uniformity of care for colorectal cancers through National Quality Forum quality indicators at a Commission on Cancer-accredited community based teaching hospital. Surg Oncol Clin N Am. 2011; 20(3): 587-596.
10.1016/j.soc.2011.01.013
PubMed Web of Science® Google Scholar
100Wong JH, Lum SS, Morgan JW. Lymph node counts as an indicator of quality at the hospital level in colorectal surgery. J Am Coll Surg. 2011; 213(2): 226-230.
10.1016/j.jamcollsurg.2011.05.003
PubMed Web of Science® Google Scholar
101Liu JB, Huffman KM, Palis BE, et al. Reliability of the American College of Surgeons Commission on cancer's quality of care measures for hospital and surgeon profiling. J Am Coll Surg. 2017; 224(2): 180-190.
10.1016/j.jamcollsurg.2016.10.053
PubMed Web of Science® Google Scholar
102Chang Y, Chen L, Chang Y, Chung K, Lai MS. Application of propensity score model to examine the prognostic significance of lymph node number as a care quality indicator. Surg Oncol. 2012; 21(2): 75-85.
10.1016/j.suronc.2011.12.003
Web of Science® Google Scholar
103Chien T, Lin Y, Chang C, Tsai MT, Uen YH. Using a bubble chart to enhance adherence to quality-of-care guidelines for colorectal cancer patients. Eur J Cancer Care. 2012; 21(6): 712-721.
10.1111/j.1365-2354.2012.01334.x
PubMed Web of Science® Google Scholar
104Dejardin O, Ruault E, Jooste V, et al. Volume of surgical activity and lymph node evaluation for patients with colorectal cancer in France. Dig Liver Dis. 2012; 44(3): 261-267.
10.1016/j.dld.2011.10.003
PubMed Google Scholar
105Shulman LN, Browner AE, Palis BE, et al. Compliance with cancer quality measures over time and their association with survival outcomes: the commission on cancer's experience with the quality measure requiring at least 12 regional lymph nodes to be removed and analyzed with colon cancer resections. Ann Surg Oncol. 2019; 26(6): 1613-1621.
10.1245/s10434-019-07323-w
PubMed Web of Science® Google Scholar
106Fung-Kee-Fung M, Boushey RP, Watters J, et al. Piloting a regional collaborative in cancer surgery using a “community of practice” model. Curr Oncol. 2014; 21(1): 27-34.
10.3747/co.21.1663
CAS PubMed Web of Science® Google Scholar
107Steele SR, Chen SL, Stojadinovic A, et al. The impact of age on quality measure adherence in colon cancer. J Am Coll Surg. 2011; 213(1): 95-105.
10.1016/j.jamcollsurg.2011.04.013
PubMed Web of Science® Google Scholar
108Adam M, Turner M, Sun Z, et al. The appropriateness of 30-day mortality as a quality metric in colorectal cancer surgery. Am J Surg. 2018; 215(1): 66-70.
10.1016/j.amjsurg.2017.04.018
PubMed Web of Science® Google Scholar
109Stojadinovic A, Nissan A, Wainberg Z, et al. Time-dependent trends in lymph node yield and impact on adjuvant therapy decisions in colon cancer surgery: an international multi-institutional study. Ann Surg Oncol. 2012; 19(13): 4178-4185.
10.1245/s10434-012-2501-5
PubMed Web of Science® Google Scholar
110Marks JH, Valsdottir EB, Rather AA, Nweze IC, Newman DA, Chernick MR. Fewer than 12 lymph nodes can be expected in a surgical specimen after high-dose chemoradiation therapy for rectal cancer. Dis Colon Rectum. 2010; 53(7): 1023-1029.
10.1007/DCR.0b013e3181dadeb4
CAS PubMed Web of Science® Google Scholar
111Sancho-Muriel J, Frasson M, Hervas D, et al. Standard outcome indicators after colon cancer resection. Creation of a nomogram for autoevaluation. Cir Esp. 2017; 95(1): 30-37.
10.1016/j.ciresp.2016.10.001
PubMed Web of Science® Google Scholar
112Nathan H, Shore AD, Anders RA, Wick EC, Gearhart SL, Pawlik TM. Variation in lymph node assessment after colon cancer resection: patient, surgeon, pathologist, or hospital? J Gastrointest Surg. 2011; 15(3): 471-479.
10.1007/s11605-010-1410-9
PubMed Web of Science® Google Scholar
113Lisovsky M, Schutz SN, Drage MG, Liu X, Suriawinata AA, Srivastava A. Number of lymph nodes in primary nodal basin and a “second look" protocol as quality indicators for optimal nodal staging of colon cancer. Arch Pathol Lab Med. 2017; 141(1): 125-130.
10.5858/arpa.2015-0401-OA
PubMed Web of Science® Google Scholar
114Boutanos C, Capdepont M, Svrcek M, et al. National multicentric evaluation of quality of pathology reports for rectal cancer in France in 2016. Virchows Archiv. 2019; 474(5): 561-568.
10.1007/s00428-019-02534-8
CAS PubMed Web of Science® Google Scholar
115Kin C, Snyder K, Kiran RP, Remzi FH, Vogel JD. Accidental puncture or laceration in colorectal surgery: a quality indicator or a complexity measure? Dis Colon Rectum. 2013; 56(2): 219-225.
10.1097/DCR.0b013e3182765c43
PubMed Web of Science® Google Scholar
116Peterson CY, Palazzi K, Parsons JK, Chang DC, Ramamoorthy SL. The prevalence of laparoscopy and patient safety outcomes: an analysis of colorectal resections. Surg Endosc. 2014; 28(2): 608-616.
10.1007/s00464-013-3216-9
PubMed Web of Science® Google Scholar
117Hendren S, Birkmeyer JD, Yin H, Banerjee M, Sonnenday C, Morris AM. Surgical complications are associated with omission of chemotherapy for stage III colorectal cancer. Dis Colon Rectum. 2010; 53(12): 1587-1593.
10.1007/DCR.0b013e3181f2f202
PubMed Web of Science® Google Scholar
118Almoudaris A, Burns E, Bottle A, et al. Single measures of performance do not reflect overall institutional quality in colorectal cancer surgery. Gut. 2013; 62(3): 423-429.
10.1136/gutjnl-2011-301489
PubMed Web of Science® Google Scholar
119Henneman D, Dekker JW, Wouters MW, Fiocco M, Tollenaar RA. Benchmarking clinical outcomes in elective colorectal cancer surgery: the interplay between institutional reoperation- and mortality rates. Eur J Surg Oncol. 2014; 40(11): 1429-1435.
10.1016/j.ejso.2014.08.473
CAS PubMed Web of Science® Google Scholar
120McSorley S, Lowndes C, Sharma P, Macdonald A. Unplanned reoperation within 30 days of surgery for colorectal cancer in NHS Lanarkshire. Colorectal Dis. 2013; 15(6): 689-694.
10.1111/codi.12135
CAS PubMed Web of Science® Google Scholar
121Burns EM, Mamidanna R, Currie A, et al. The role of caseload in determining outcome following laparoscopic colorectal cancer resection: an observational study. Surg Endosc. 2014; 28(1): 134-142.
10.1007/s00464-013-3139-5
PubMed Web of Science® Google Scholar
122Damle R, Macomber C, Flahive J, et al. Surgeon volume and elective resection for colon cancer: an analysis of outcomes and use of laparoscopy. J Am Coll Surg. 2014; 218(6): 1223-1230.
10.1016/j.jamcollsurg.2014.01.057
PubMed Web of Science® Google Scholar
123de la Portilla F, Builes S, Garcia-Novoa A, et al. Analysis of quality indicators for colorectal cancer surgery in units accredited by the Spanish Association of Coloproctology. Cir Esp. 2018; 96(4): 226-233.
PubMed Web of Science® Google Scholar
124Hofstede SN, Ceyisakar IE, Lingsma HF, Kringos DS, Marang-van de Mheen PJ. Ranking hospitals: do we gain reliability by using composite rather than individual indicators? BMJ Qual Saf. 2019; 28(2): 94-102.
10.1136/bmjqs-2017-007669
PubMed Web of Science® Google Scholar
125Gort M, Otter R, Plukker JTM, Broekhuis M, Klazinga NS. Actionable indicators for short and long term outcomes in rectal cancer. Eur J Cancer. 2010; 46(10): 1808-1814.
10.1016/j.ejca.2010.02.044
PubMed Web of Science® Google Scholar
126Ommundsen N, Nesbakken A, Wyller TB, et al. Post-discharge complications in frail older patients after surgery for colorectal cancer. Eur J Surg Oncol. 2018; 44(10): 1542-1547.
10.1016/j.ejso.2018.06.024
PubMed Web of Science® Google Scholar
127Xu T, Makary M, Kazzi E, et al. Surgeon-level variation in postoperative complications. J Gastrointest Surg. 2016; 20(7): 1393-1399.
10.1007/s11605-016-3139-6
PubMed Web of Science® Google Scholar
128Merkow RP, Bentrem DJ, Chung JW, Paruch JL, Ko CY, Bilimoria KY. Differences in patients, surgical complexity, and outcomes after cancer surgery at National Cancer Institute-designated cancer centers compared to other hospitals. Med Care. 2013; 51(7): 606-613.
10.1097/MLR.0b013e3182928f44
PubMed Web of Science® Google Scholar
129Hedrick T, Harrigan A, Sawyer R, et al. Defining surgical site infection in colorectal surgery: an objective analysis using serial photographic documentation. Dis Colon Rectum. 2015; 58(11): 1070-1077.
10.1097/DCR.0000000000000466
PubMed Web of Science® Google Scholar
130Murray AC, Pasam R, Estrada D, Kiran RP. Risk of surgical site infection varies based on location of disease and segment of colorectal resection for cancer. Dis Colon Rectum. 2016; 59(6): 493-500.
10.1097/DCR.0000000000000577
PubMed Web of Science® Google Scholar
131Hou X-T, Pang D, Lu Q, et al. Validation of the Chinese version of the low anterior resection syndrome score for measuring bowel dysfunction after sphincter-preserving surgery among rectal cancer patients. Eur J Oncol Nurs. 2015; 19(5): 495-501.
10.1016/j.ejon.2015.02.009
PubMed Web of Science® Google Scholar
132Hernandez-Boussard TM, McDonald KM, Morrison DE, Rhoads KF. Risks of adverse events in colorectal patients: population-based study. J Surg Res. 2016; 202(2): 328-334.
10.1016/j.jss.2016.01.013
PubMed Web of Science® Google Scholar
133Snijders HS, Henneman D, van Leersum NL, et al. Anastomotic leakage as an outcome measure for quality of colorectal cancer surgery. BMJ Qual Saf. 2013; 22(9): 759-767.
10.1136/bmjqs-2012-001644
CAS PubMed Web of Science® Google Scholar
134Fischer C, Lingsma HF, van Leersum N, Tollenaar RA, Wouters MW, Steyerberg EW. Comparing colon cancer outcomes: the impact of low hospital case volume and case-mix adjustment. Eur J Surg Oncol. 2015; 41(8): 1045-1053.
10.1016/j.ejso.2015.04.009
CAS PubMed Web of Science® Google Scholar
135Gruen RL, Pitt V, Green S, Parkhill A, Campbell D, Jolley D. The effect of provider case volume on cancer mortality: systematic review and meta-analysis. CA Cancer J Clin. 2009; 59(3): 192-211.
10.3322/caac.20018
PubMed Web of Science® Google Scholar
136Henneman D, van Bommel AC, Snijders A, et al. Ranking and rankability of hospital postoperative mortality rates in colorectal cancer surgery. Ann Surg. 2014; 259(5): 844-849.
10.1097/SLA.0000000000000561
PubMed Web of Science® Google Scholar
137Hureibi K, Evans C, Wong L, Ain Q. Time to rethink surgeon-specific outcome data for colorectal surgeons in England: cross-sectional data of 73,842 resections for colorectal cancer. Int J Surg. 2018; 57: 101-104.
10.1016/j.ijsu.2018.08.005
PubMed Web of Science® Google Scholar
138Mainz J, Hjulsager M, Og MT, Burgaard J. National benchmarking between the Nordic countries on the quality of care. J Surg Oncol. 2009; 99(8): 505-507.
10.1002/jso.21204
PubMed Web of Science® Google Scholar
139Osler M, Iversen LH, Borglykke A, et al. Hospital variation in 30-day mortality after colorectal cancer surgery in denmark: the contribution of hospital volume and patient characteristics. Ann Surg. 2011; 253(4): 733-738.
10.1097/SLA.0b013e318207556f
PubMed Web of Science® Google Scholar
140Zheng H, Zhang W, Ayanian JZ, et al. Profiling hospitals by survival of patients with colorectal cancer. Health Serv Res. 2011; 46(3): 729-746.
10.1111/j.1475-6773.2010.01222.x
PubMed Web of Science® Google Scholar
141Hofstede SN, van Bodegom-Vos L, Kringos DS, Steyerberg E, Marang-van de Mheen PJ. Mortality, readmission and length of stay have different relationships using hospital-level versus patient-level data: an example of the ecological fallacy affecting hospital performance indicators. BMJ Qual Saf. 2018; 27(6): 474-483.
10.1136/bmjqs-2017-006776
PubMed Web of Science® Google Scholar
142Theile DE, Philpot S, Blake M, Harrington J, Youl PH. Outcomes following colorectal cancer surgery: results from a population-based study in Queensland, Australia, using quality indicators. J Eval Clin Pract. 2018; 21: 21.
Google Scholar
143Youl P, Philpot S, Theile DE, Cancer Alliance Queensland. Outcomes after rectal cancer surgery: a population-based study using quality indicators. J Healthc Qual. 2019; 22: 90-100.
10.1097/JHQ.0000000000000200
Web of Science® Google Scholar
144Abdelsattar Z, Wong S, Birkmeyer N, et al. Multi-institutional assessment of sphincter preservation for rectal cancer. Ann Surg Oncol. 2014; 21(13): 4075-4080.
10.1245/s10434-014-3882-4
PubMed Web of Science® Google Scholar
145Turner NH, Wong HL, Field K, et al. Novel quality indicators for metastatic colorectal cancer management identify significant variations in these measures across treatment centers in Australia. Asia Pac J Clin Oncol. 2015; 11(3): 262-271.
10.1111/ajco.12355
PubMed Web of Science® Google Scholar
146Horn ME, Reinke EK, Mather RC, O'Donnell JD, George SZ. Electronic health record–integrated approach for collection of patient-reported outcome measures: a retrospective evaluation. BMC Health Serv Res. 2021; 21(1): 1-11.
10.1186/s12913-021-06626-7
PubMed Web of Science® Google Scholar

Citing Literature

Volume20, Issue2

April 2024

Pages 259-274

Filename	Description
ajco13917-sup-0001-FigureS1.docx35.7 KB	Supporting information
ajco13917-sup-0002-FigureS2.docx17.3 KB	Supporting information
ajco13917-sup-0003-FigureS3.docx107.6 KB	Supporting information
ajco13917-sup-0004-SuppInfo.docx16.8 KB	Supporting information

Measurement that matters: A systematic review and modified Delphi of multidisciplinary colorectal cancer quality indicators

Abstract