2.1 Qualitative Approach and Reflexivity

This was a qualitative study, grounded in a critical realist ontology (which holds that there is a reality to understand but that researchers can only ever partially know it) and contextualist epistemology (data do not directly represent reality, but reality is made sense of by the researcher's interpretative practices) [7, 8]. The study is reported according to the Standards for Reporting Qualitative Research [9]. Reflexivity is the process of critically reflecting on the process of knowledge production [10] and was a key consideration in our study. Reflexivity was explored and incorporated into the methods through the use of a reflexive journal and regular reflective discussions with the whole research team (Supporting Information S1: Material 1).

2.2 Public and Patient Involvement and Engagement (PPIE)

PPIE has underpinned this work from its conception. A Patient Advisory Group (PAG) contributed to the research questions and funding application; commented on drafts of materials for ethics application; co-designed patient information materials; contributed to pilot interviews; co-developed the semi-structured topic guide; and contributed to the interpretation of findings (Supporting Information S1: Material 2). Hughes and Duffy outlined the different operational definitions of PPIE [11]. We used a combination of embedded consultation (members of the public are consulted regularly throughout the research cycle, giving feedback on research ideas and dissemination plans) and collaboration/co-production (members of the public contribute to key decisions and findings as part of a steering group) throughout. We ensured a commitment to clear and agreed meaning and purpose of involvement through regular meetings and correspondence.

2.3 Recruitment and Sampling Strategy

We recruited adults (aged 18 years old and older) with lived experience of depression (excluding those with a history of severe mental illness) and GPs working in the English National Health Service (NHS; partner, salaried or locum GPs). We used purposive sampling [12], aiming for maximum variation in factors that we determined were likely to be most important in impacting on participants' experiences. For people with lived experience of depression, these factors were age, gender, ethnicity and index of multiple deprivation (IMD) of home postcode. For GPs, these factors were number of years working as a GP, contractual status, ethnicity and IMD (practice level). The NIHR Clinical Research Network (CRN) Yorkshire and Humber supported with recruitment through 15 general practices. People with lived experience of depression were recruited through a database search of GP electronic health record (followed by a letter of invitation to participate) or study advertisement through participating practices. GPs were recruited through a combination of study information being shared through participating practices, social media advertisement (Twitter/X) and snowball sampling [12, 13]. Potential participants were asked to express interest in participating by directly contacting the research team.

2.4 Data Collection and Analysis

GPs and people with lived experience of depression were interviewed concurrently between May and October 2022 by a single researcher (A.S.M.). Semi-structured interviews were guided by topic guides (see Supporting Information S1: Material 3), and these were modified iteratively as further data were collected and analysed. Interviews were conducted online and digitally recorded using Microsoft Teams. Transcription was done by an external professional audio-transcription company in the naturalised ‘intelligent’ verbatim; anonymised transcripts were stored on a secured University drive.

Data were analysed using NVivo 13 [14]. We used a form of codebook thematic analysis (TA), in which a coding framework is developed inductively during analysis to guide the generation of themes and researcher subjectivity is recognised as a resource for research rather than a limitation to be controlled for [15, 16]. Throughout analysis, we incorporated principles of constant comparison, whereby new data were systematically compared with previously analysed data, looking for similarities and differences [17]. Analysis was carried out by a single researcher (A.S.M.); to ensure consistency, a random selection of transcripts was coded by two researchers (A.S.M. and C.A.C.-G.), and the coding was regularly reviewed by the whole multidisciplinary team and PAG. Sufficient sample size was determined by principles of data saturation [18] and information power [19]. Through constant comparison [17], we were able to identify the point at which no new themes (new ideas and meanings) were apparent upon collection of further data. We reviewed the richness of the information in the data set and the extent to which this met the aims and objectives of the study at regular multidisciplinary research team meetings. Trustworthiness [20] of analysis was enhanced through triangulation of findings within the multidisciplinary team and PAG; ensuring an adequate amount of rich data; comparisons and contrasts between data from two participant groups (GPs and people with lived experience of depression); and transparency in reporting data collection and analysis methods [21].

2.5 Ethical Considerations

Ethical approval for this study was granted by the NHS's Health Research Authority on 11 March 2022 (REC reference22/WM/0022). All participants provided written consent to participate and a risk protocol was available in the event that a participant disclosed suicidal ideation or intent. All participants were reimbursed for their time in the form of vouchers.

3.1 Summary of Participants

Twenty-three people with lived experience of depression and 22 GPs were interviewed (Table 1).

Illustrative data examples are presented to support the findings from the analysis. Participants are identified by a label indicating type of participant (P for person with lived experience or GP; gender [M or F]; and age).

3.2 Use of Prediction Models in Primary Care: The GP Perspective

This section presents findings from the analysis of interviews with GPs only, because this theme is about how GPs use prediction models and therefore was not relevant to people with lived experience. GPs reported using prediction models in primary care for the purposes of both diagnosis and prognosis. The findings are presented in terms of which models are used; why some models are used over others; and how prediction models are used in general practice.

3.2.2 Why Prediction Models Are Used

Two key themes were generated: some models are perceived to be useful (i.e., GPs are internally motivated to use them) and some models have to be used (GPs are externally motivated to use them, e.g., to meet financial or contractual targets). These two attributes are not mutually exclusive, but if neither applies, then this study suggests that GPs are unlikely to use the model in practice. The analysis also showed some additional key facilitators and barriers to models being routinely used in practice.

3.2.2.1 Some Models Are Perceived to be Useful

Models were perceived to be useful when they guide discussions with patients, particularly when the GP perceives that the patient may not be satisfied with their suggested treatment options. An example of this is using a diagnostic prediction model to provide patients with an objective measure of the likelihood of viral infection rather than bacterial infection, to justify a clinical decision that could lead to difficult discussions with patients:

The FeverPAIN score is really helpful. I don't use that, necessarily, for my own clinical judgement but I use it for my patients, because I think when I'm not prescribing antibiotics it can be a difficult conversation. But actually, when I say ‘there's a really good and clever reason and I can reassure you for that’, then I think patients take it better than saying ‘I'm a really clever doctor and I don't think you need antibiotics’.

GP6-M-33

All GPs reported being more likely to use prediction models when they are perceived to directly influence clinical management. This was particularly the case when they perceived that the model adds something above what their clinical assessment (history and examination) can offer:

I think also if I can see a need for it…For the QRISK, deciding on whether the patient should have a statin, I need that evidence. Because a lot of the times, patient will have a high cholesterol but actually their risk can be one per cent. And I would have started them on a statin unnecessarily and vice versa.

GP2-F-38

GPs generally perceived a model as useful if it is known to be trustworthy and valid. Most GPs were interested in the evidence base for prognostic models, particularly those they used commonly, but said that they usually would not consult or critically appraise the primary evidence themselves. Usually, the fact that prognostic models were endorsed by the NHS or the National Institute for Health and Care Excellence (NICE) assured GPs that the models were sufficiently supported by evidence to warrant using.

If they were on NICE guidelines that gives me a lot of confidence that they are rigorous, as they've been well researched…For things that I'm using all the time, like QRisk, I like to have a better understanding of the population that it was studied…So, things that are more relevant to my population, I think, it's good to be able to explain that to patients.

GP1-F-35

Use of models by specialist colleagues, publicity in the medical press and incorporation into clinical templates also increased GPs' confidence in using tools and gave GPs the impression that they are widely used and accepted.

I suppose ones that you see specialists using…you'd feel a bit more confident because if your local consultants are using it, then you'd probably feel like it's a reasonably validated tool.

GP10-F-44

Most GPs reported being more likely to use models earlier in their career, particularly if they had been a feature of their GP training or formal medical education. All GPs also reported perceiving models as useful if they can be proven to save clinician time overall. Finally, all GPs reported valuing prediction models that they perceive to be potentially medicolegally protective:

In some situations, you're not safe if you don't use it. So, you know, if you're determining a few things, if somebody has DVT you, sort of, have to demonstrate that you made a safe decision by using a Wells score… from a point of view of documentation and medicolegal aspects.

GP18-F-34

In summary, GPs perceived prediction models to be useful for four key reasons: they guide discussions around management with patients; they aid clinical decision-making; they are perceived as valid (trustworthy); or they protect the GP (e.g., medicolegally).

3.2.2.2 Some Models Have to be Used

GPs reported the external motivators that make them think that a model has to be used. These included financial incentivisation (for example, through Quality Outcome Framework [QOF]), Care Quality Commission (CQC) recommendations or inclusion in practice policies and local referral pathways:

If it [prediction model] was really evidenced, cost-effective and worked for patients and QOF got a hold of it and told us that there was a financial incentive to do it, I think that would probably encourage practices to put more emphasis on encouraging their GPs to use it.

GP3-F-32

Being adopted as part of a practice or local policy was another thing that made GPs more likely to use a model.

Sometimes it's a practice protocol and policy, sometimes it's QOF-related, it's just your local sort of protocols and guidelines and things as to what you follow as well as just good practice. So I think it's just a mixture really of things.

GP8-F-52

3.2.2.3 The Time-Limited Consultation

GPs discussed the practical barriers and facilitators to using models in practice (Figure 1). Limited time and resources were viewed as major barriers, with prediction models being one of many competing considerations in a consultation. Other barriers were limited reminders to use the particular tool and limited accessibility of the model (i.e., not integrated into practice systems). All GPs discussed the increasing complexity of the GP consultation generally, with many different demands on that relatively short amount of time and multiple different issues requiring discussion.

I think a lot of this is down to time, isn't it, most appointments are still ten minutes…when you manage someone with mental health issues, ten minutes, even 15 minutes isn't enough at all, and then to then open up a pop-up with lots of questions, it's going to take another five, ten minutes to do, so it's just it's time constraints more than anything else.

GP11-M-40

Given the pressures outlined, most GPs suggested that the outputs and recommendations from models had to be realistic within the clinical context. Where demands arising as a result of models were overly onerous or unrealistic, most GPs did not feel that it was an efficient use of their time to use a model. Other facilitators to use were integration with the electronic clinical record; models that can easily be delegated to other non-GP members of the primary care team (e.g., practice nurses) or filled in by patients directly; and models that do not require additional work to utilise.

So it would need to be like a template that's accessible that people can find on their computer system, via Ardens or whatever, and then kind of recorded into the computer system, that's always preferable. It'd need to be really quite simple because some people are more IT-literate than others.

GP15-F-47

3.3 How to Communicate Results of Clinical Prediction Models

In summary, GPs and people with lived experience of depression reported that numerical information can be useful but ideally, explanations involving numbers should be tailored to individual patients' preferences. Visual representations, such as Cates plots (which use smiley faces to visually represent risks and benefits of a particular treatment), can be helpful for explaining risk to patients.

3.4 Prognostic Models for Depression Recurrence in Primary Care

4.1 Summary of Key Findings

GPs reported using prediction models if they perceive them to be valuable and/or that they have to use them as part of their contractual obligations. If neither of these criteria are met, prognostic models are unlikely to be used in general practice. If these criteria are met, then there are still practical barriers to use (e.g., time constraints in a consultation) that must be overcome and facilitators to use (e.g., integration with the GP IT system) that should be considered in any implementation process.

People with lived experience of depression reflected that they were keen to learn about their outcome risks and there is likely to be value in producing numerical and other kinds of output (e.g., risk categories) and allowing the patient and GP together to decide on the most appropriate information to guide a discussion tailored to the individual patient. There was more scepticism from participants about the use of prognostic models for depression relapse compared with physical health conditions, due to the perceived more ‘subjective’ nature of depression. However, some people with lived experience of depression reported that they would find an approach more in line with physical health to be reassuring. The importance of an appropriate action plan or additional support for higher risk individuals was highlighted by all participants.

4.2 Comparison With Previous Literature

There have been a limited number of previous studies exploring clinician views of prediction models and only one in primary care [4]. One study explored the views of mental health clinicians and healthcare administrators on prognostic models in the context of suicide risk prediction [32]. In that study, clinicians shared the concerns of participants in our study that they would not have the clinical resources to adequately meet the potentially increased demand, as well as having medicolegal concerns. Like in our study, participants thought that, once a clinician is aware of risk information, they feel that they would be personally liable for not acting quickly enough. Currently, there is limited guidance for helping clinicians understand and mitigate the risk inherent in using risk prediction models to guide medical treatment [33].

Our results are further supported by preexisting literature on the value of models that help facilitate discussion with patients around prognosis and guide patient management [34, 35]. Only one preexisting study explored patient perspectives of prognostic models, in the context of communicating prognosis to people with head and neck cancer [36], in which patients were broadly accepting of the use of prognostic models and keen to understand the quantitative results. Our findings suggesting that integration with clinical systems is a facilitator to use are supported by a previous study looking to incorporate the FRAX tool into a nurse-delivered care review in general practice [37].

A recent scoping review looked at the evidence around communication of prognostic model results by healthcare providers to patients [38]. The review reported a lack of evidence on the best ways to communicate prognostic model results. It did, however, highlight the importance of contextualising information when discussing with patients, which is consistent with our findings. This contextualisation specifically referred to defining timeframes over which predictions are applicable, using results in the context of shared decision making and using data visualisation as an explanatory tool (particularly for those with lower health literacy).

The findings from our study suggest that communication of risk benefits from being individualised to each patient in question. This is supported by previous research on the subject of risk communication that has found that either percentages, frequencies or proportions are acceptable for conveying numerical risk to patients; the use of visual representations in multiple formats helps cater for different patient preferences (illuminated with words and numbers where possible); and the importance of initially assuming low numeracy and providing optional additional detail as desired (rather than providing more complex information to start) [39-41]. An additional consideration for further studies of clinician and patient perspectives around clinical prediction models is a better understanding of the communication of uncertainty around risk estimates. There has been some previous research exploring this in the context of prognostic models in cancer, which found heterogeneity in the ways in which uncertainty is both available to and understood by clinicians and communicated with patients [42]. The extent to which there is uncertainty around estimates is of significance to researchers and often measured and reported in the form of 95% confidence intervals. Whether confidence intervals would be useful and easily understood by clinicians and patients in the context of individual risk estimates from prediction models is unclear and warrants further qualitative exploration.

4.3 Strengths and Limitations of the Study

Participants were recruited from populations with a broad range of demographics in both groups. The sample of people with lived experience of depression had fewer people from ethnic minority backgrounds than we aimed for, despite efforts to increase this by purposively approaching practices with higher ethnic diversity in their patient populations. Thus, the perspectives of people from different cultural backgrounds may not be well represented here. Future work should further consider the views of people with lived experience of depression from a wider range of ethnic minority backgrounds. Apart from a random subsample of five transcripts that were coded by two researchers (A.S.M. and C.A.C.-G.), the rest were coded by a single researcher (A.S.M.). This is consistent with the theoretical approach and form of codebook TA used, but is worth taking into consideration when interpreting our findings.

A key strength of this study was the involvement of the PAG, which was involved from study conception. This group contributed to the design of patient information materials and the development of the semi-structured topic guide, to ensure that the topics discussed were relevant and important to patients. They further contributed to the interpretation of findings, adding new insight to the results of these qualitative analyses.

4.4 Implications for Practice and Research

Prediction model researchers should be cognizant of the clinical context, before starting model development. Predictions made in primary care must relate to predictors and outcomes that are important to GPs and patients. Our study suggests that, if GPs do not view the model as useful and are not otherwise externally motivated to use it, the model will not be used in primary care. Researchers are advised to consider developing models that provide a range of outputs (both numerical and visual) to allow discussions between patients and clinicians to be tailored to individual preferences.

Primary care mental health is an area where prediction models may be useful. Models have been developed to predict outcomes in psychosis [43], depression onset [44] and depression outcomes [45], and to guide depression management [46] in primary care with promising predictive performance. This study has demonstrated that predicting relapse and recurrence of depression in primary care would likely have useful applications, although models have demonstrated limited predictive performance thus far [47]. As in previous studies [32], participants in our study highlighted the importance of having an action plan or ability to provide additional support as appropriate, in response to the findings of prediction models. Previous literature suggests that appropriate approaches for people with increased depression relapse risk might include case management and relapse prevention planning [48, 49], as well as more formal relapse prevention interventions [50]. Researchers aiming to develop clinical prediction models in the area of primary care mental health should be aware of potential barriers to model use, including perceived increased complexity and challenges in predicting outcomes when compared with prediction models for physical health conditions. When developing prediction models in this area, we recommend embedded and meaningful public and patient involvement and concurrent qualitative research, ideally as part of a mixed-methods approach.