A systematic review on the effect of routine outcome monitoring and feedback on client outcomes in alcohol and other drug treatment

2.1 Search strategy

This systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Electronic literature searches were performed between July 2020 and November 2021. The databases Medline (via Ovid), PsycInfo (via Ovid) and Scopus were searched for articles published in English with no limitation on year of publication or study type. The search was performed using keywords combined with relevant subject headings. The following combination of search terms was used in each database: (‘feedback informed treatment’ or ‘patient level feedback’ or ‘partners for change outcome management system’ or ‘progress monitoring’ or ‘routine outcome m*’ or ‘routine monitoring’ or ‘client feedback’ or ‘patient feedback’ or ‘measurement based care’) and (‘substance use’ or ‘alcoholism’ or ‘alcohol abuse’ or ‘drug abuse’ or ‘drug addiction’ or ‘drug dependency’) and (‘treatment outcome*’ or ‘outcome*’). We manually searched the bibliographies of identified articles to ensure complete coverage. One additional paper was identified through contact with experts.

2.2 Eligibility criteria

Our eligibility criteria were chosen in accordance with our PICO framework. Our PICO framework consisted of: Population—people with substance use disorders receiving treatment in AOD treatment settings; Intervention—feedback from ROM; Control—control group or condition of no feedback from ROM; and Outcome—AOD treatment outcomes. To be included in the review studies needed to: (i) be English language; (ii) be conducted primarily in AOD treatment settings with participants receiving treatment primarily for SUDs; (iii) examine the effect on client outcomes of feeding back routinely monitored data to clients and/or clinicians; (iv) have feedback from ROM as the study intervention; (v) contain a control group or condition of no feedback from ROM; and (vi) measure AOD treatment outcomes (e.g., substance use, treatment retention). Narrative reviews, systematic reviews, commentaries and research that did not examine the effect on client outcomes of feeding back routinely monitored data to clients and/or clinicians were excluded. Two rounds of screening (title/abstract and full text) were undertaken by one author (IC) using the above inclusion/exclusion criteria, in consultation with a second author (LM). After selection, all 10 included studies were reviewed by two authors (IC and LM) and six studies by three authors ([21, 26, 27] by NL and [28-30] by KM). Records were stored and managed in the software program EndNote [31].

2.3 Data extraction

One author (IC) extracted data from each study and entered it into Tables 1–3. A second author (LM) reviewed Tables 1–3, ensured the data were correct and added any additional relevant data. A third author (KM or NL) then followed the same process for the studies listed above. Data items that were extracted from each study included the author, year of publication, study population, treatment settings, outcomes measured, study design, methods, intervention, control group/condition, overall study quality and results.

2.4 Risk of bias assessment

We based our assessment of evidence quality and risk of bias on the quality assessment tools developed by the National Institutes of Health (see https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools) and outlined in detail in the Systematic Evidence Review from the Blood Pressure Expert Panel, 2013 [32], one of the reports recommended on the website for guidance on how to apply the tool. As stated by the panel, the NIH quality assessment tools were ‘designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study … not … to provide a list of factors comprising a numeric score’ [32; p. 82, appendix A]. Based on the studies returned by our literature review we used the tool for (i) Quality Assessment of Controlled Intervention Studies; and (ii) Quality Assessment Tool for Before–After (Pre–Post) studies with No Control Group.

Two authors (IC and LM) reviewed the quality of each of the papers using the tools described above and assigned an assessment of good (low risk of bias), fair (moderate risk of bias) or poor (high risk of bias) to each. If the ratings differed, the reviewers discussed the article until a consensus was reached.

2.5 Evidence synthesis

There was considerable heterogeneity in the study designs and outcome measures of the included studies, making it difficult to directly compare studies. Due to this heterogeneity, and the small number of studies identified, a formal meta-analysis was not performed. Instead, the evidence was synthesised narratively. We have organised the synthesis of study results firstly by whom the feedback was provided to—clinicians only, clients only and clinicians plus clients—and secondly by which outcomes were measured. We have also reviewed client and treatment characteristics and timing of feedback across studies.

3.1 Study selection

Our PRISMA flowchart is shown in Figure 1. The search strategy yielded 99 papers without duplicates through database searching, citation chaining and manual searching. After screening titles/abstracts, 21 papers remained. Full texts were then assessed for eligibility by applying the inclusion/exclusion criteria listed above. A further 11 papers were excluded: seven did not examine the effect on client outcomes of feeding back routinely monitored data to clients and/or clinicians, three were not conducted primarily in AOD treatment settings and one did not have feedback from ROM as the study intervention. A final group of 10 papers was included in this study, each reviewed by at least two authors. This information is summarised in Tables 1–3.

3.2 Types of study

Of the 10 studies reviewed, five were controlled intervention studies, involving randomisation of participants (whether clients or clinicians or both) to either feedback or control groups, and five were treated as before–after (pre–post) within-subjects studies with a control condition. When performing our quality assessment of the included studies, we assessed the Miller and colleagues [21], McCaul and Svikis [33] and Seitz and Mee-Lee [34] studies based on the criteria for Before–After studies because, while important details of study design and analysis were not reported in these studies, it was clear they involved some kind of repeated-measures design with a within-subjects control condition but had no separate control group.

3.3 Study quality

See Tables S1 and S2 for details of how quality ratings were assigned to each individual study.

3.4 Outcomes

Table 4 shows the outcomes measured by the six good- or fair-quality studies. The cells shaded grey indicate that the differences observed between the feedback and control groups/conditions were significant for the outcome in question. Four out of six studies found a significant improvement in at least one outcome: two out of three clinician-only feedback studies, one out of one client-only feedback studies, and one out of two client and clinician feedback studies. Only one study [27], a clinician-only feedback study, found that feedback was associated with significant improvements (i.e., reductions) in substance use, and this was not overall, but only for those subjects who were deemed to be off-track. Two studies found feedback to be significantly associated with treatment retention, one a clinician-only feedback study [40] and one a clinician and client feedback study [28]. Two studies, both clinician-only feedback studies [27, 40], found feedback to be associated with improvements in global functioning/wellbeing, and one of these studies [40] also found feedback to be positively associated with therapeutic alliance and clinicians' perception of client outcomes. And finally, the sole client-only feedback study found feedback to be significantly associated with reductions in anxiety and stress [43].

3.5 Client and treatment characteristics

Treatment settings and modalities were relatively consistent across the good/fair-quality studies, with all participants attending counselling as outpatients. Client characteristics were also similar, with all studies recruiting adults, except for one whose participants were adolescents and young adults under 25 years [43]. Most studies did not restrict their sample to one primary drug of concern, except one study where all participants were being treated for cannabis dependence [29]. This consistency in setting, modality, age and sampling of clients makes it difficult to determine whether these factors influence outcomes in any way as there are no alternative groups available to compare against.

3.6 Timing of feedback

Most studies provided feedback in a regular timely manner weekly or twice weekly. For the four good/fair-quality studies that provided feedback weekly or twice weekly, two showed significant improvement in some outcomes [27, 43] while two studies did not [26, 29]. Unfortunately, the other two good-quality studies [28, 40] did not specify the frequency of feedback, so although both these studies showed improvement in outcomes it is difficult to draw any conclusions about the role timing of feedback may have played in improving outcomes.

4.1 Interpretation of findings

The results of our review suggest that while there is some encouraging preliminary evidence to suggest that providing AOD clients and/or clinicians with feedback is associated with improved client outcomes, overall there is not enough good quality evidence to draw any firm conclusions.

With the search parameters we used we could only find 10 experimental clinical AOD studies that had addressed the question of whether feedback from ROM improved patient outcomes, and of these only six were of good or fair quality as evidence. Feedback was provided to clinicians only in three of these studies, to clients only in one, and to both clients and clinicians in two.

Perhaps of most concern is the fact that only one good-quality study [27], from among the four that measured it, found that providing feedback was associated with reduced substance use among clients, and this was only among a subset of participants, the off-track clients. Four good-quality studies found feedback significantly improved psychosocial constructs that could be considered precursors to substance use—for example, treatment retention (two studies) [28, 40], global function/wellbeing (two studies) [27, 40], therapeutic alliance (one study) [40] and mood symptoms (one study) [43]. However, while it appears as if feedback may have some effect on these ‘upstream’ constructs, in AOD settings reducing frequency and quantity of substance use is the proximal goal of treatment. The apparent lack of a reliable association between feedback and substance use prevents us from concluding that it is an effective AOD intervention yet.

In addition, two of the good-quality studies failed to find any effect of feedback for any of the outcomes they measured [26, 29]. One of these, Crits-Cristoph and colleagues [26], was by far the largest study—over 1500 clients and 118 clinicians—so the absence of a significant finding across all the outcomes measured seems especially telling. The authors provide several possible reasons for the absence of a result: low levels of substance use among clients at the start of treatment creating a floor effect; a ceiling in therapeutic alliance scores; and aggregated caseload feedback instead of individualised client feedback. This latter methodological limitation was corrected in their follow-up study [27], where individualised client feedback was provided to clinicians. This change appeared to be effective as substance use among the off-track clients in the group of clinicians that received feedback was significantly lower than among the off-track clients whose clinicians did not. This finding supports the theory that clinicians who receive regular timely individualised feedback about their clients' substance use and general functioning are better able to identify when specific clients are not doing well and respond quickly to provide additional care and support. However, once again, more studies are needed.

A wide range of outcomes was measured by the studies we reviewed (see Table 4). Of these, treatment retention and global function/wellbeing have the most support for efficacy of feedback, each with positive findings in two studies; however, once again this is not enough to draw any conclusions. Overall, there was considerable variation in the methods employed and the outcomes measured by the studies we reviewed. Some fed back information to clinicians, some to clients, some to both; some studies measured substance use and some did not; some focused on therapeutic alliance and others on global functioning. The small number of studies, their heterogeneity, and the absence of a positive association between ROM with feedback and substance use means that, at present, we cannot definitively conclude that ROM with feedback leads to better outcomes for AOD clients.

4.2 Relevance and future implications

There is ample evidence that ROM and feedback improve client outcomes in mental health treatment settings [6-14]. An obvious assumption was that what worked in mental health settings would translate well to AOD treatment; however, at present, and until more quality studies are conducted, we cannot conclude whether this is the case. Important questions that still need to be answered are to whom feedback should be delivered (clients or clinicians or both), when feedback should be delivered, what sort of information should be contained in the feedback and what action should be taken after feedback. For example, it is likely that a different set of mechanisms would operate when information is fed back to clients than when it is fed back to clinicians. Future studies should compare three groups—feedback to clients only vs. feedback to clinicians only vs. feedback to both—in the same study.

All the studies we reviewed were conducted in the context of AOD counselling and therapy. As such, the focus in most studies we reviewed was on constructs integral to counselling such as therapeutic alliance, mood and psychosocial functioning. These constructs are of course important but the way we assess the efficacy of AOD ROM and feedback interventions should primarily be via its effect on objective clinical endpoints (e.g., clients' substance use) rather than focusing on potential mechanisms of change (e.g., therapeutic alliance). Furthermore, feeding back client information to clients and clinicians should be researched in other treatment contexts (e.g., during routine case management by doctors or nurses), not just counselling.

Timing of feedback was also relatively consistent across studies, with most studies providing feedback weekly or at every session, however with such a low number of studies and variation in methods across studies, it is difficult to know what specific effect timing of feedback may have had on client outcomes. Future studies should more closely examine how the timing of feedback may affect outcomes, as well as the type of feedback and what information it contains.

Finally, and importantly, the studies we reviewed placed little emphasis on measuring what clinicians or clients actually did with the feedback they received. Determining how clinicians and clients are using feedback will be crucial in identifying the precise mechanisms by which feedback leads to better outcomes, if this proves to be the case.

4.3 Study limitations

This review had several limitations. Although more than one reviewer reviewed each paper and performed the quality assessment, only one reviewer was involved in the study selection process, which increases potential for introduction of selection bias. Only articles written in English were included in the search, potentially reducing the search's yield. A further limitation is that this review was not pre-registered. However, while pre-registration does create transparency in the research process and safeguard against researcher bias, the absence of pre-registration on its own does not indicate the presence of these biases.