Material practices for meaningful engagement: An analysis of participatory learning and action research techniques for data generation and analysis in a health research partnership

2.1 Study setting: the RESTORE project

RESTORE (Research into Implementation Strategies to Support Patients of Different Origins and Language Background in a Variety of European Primary Care Settings) was an EU-funded primary health-care research project that ran from 2011 to 2015. The objective of RESTORE was to investigate and support the implementation of guidelines and training initiatives (GTIs) intended to enhance communication in cross-cultural primary care consultations. This qualitative, comparative case study37 involved diverse stakeholders across five primary care settings: Austria, England, Greece, Ireland and The Netherlands (see File S1 for a description of the five settings). A sixth research team in Scotland focused on policy-related implications of the study. The choice of these six countries matched the academic teams who developed the proposal and intentionally included countries with diverse primary health-care systems. Ethical approval was granted by respective national committees.

RESTORE was the overall setting in which we explored and evaluated stakeholders’ and researchers’ experiences of two PLA techniques. These techniques were employed during Stage 2 and were intended to enable stakeholders (migrants, general practice staff, community interpreters, service providers, service planners and others) to work collaboratively and with RESTORE researchers to select a GTI for their local primary care setting.

First, we describe sampling and recruitment for RESTORE—this is the sample for the evaluation reported in this paper. We then describe the two specific PLA techniques employed in RESTORE and the methods used to evaluate stakeholders’ and researchers’ experiences of these techniques. Finally, we present our analysis of the evaluation data.

2.2 Sampling and recruitment

In Stage 2, we used a combination of purposeful and network sampling to identify and recruit 78 stakeholder representatives across five research sites. A geographically defined area (district) was selected in each partner country. Selection was pragmatic, based on researchers’ knowledge of groups working in the district and proximity to the research teams, to facilitate data collection. Eligible organizations/agencies were those involved in primary health-care planning and delivery (eg health-care centres, regional health authorities) as well as those addressing migrant health issues (eg non-governmental organizations focused on migrants). The aim was to identify individuals who were decision-makers (eg health authority service planners and policymakers), service providers (eg general practitioners (GPs), primary care staff, community interpreters) or service users (ie migrants using local primary care services).

In line with standard ethical procedures, stakeholders in all countries were provided with information leaflets and signed consent forms prior to participating in fieldwork sessions.

2.3 PLA data generation and co-analysis in RESTORE

Two members of the RESTORE consortium in the Irish team (practitioner/trainers with over 25 years’ international experience in PLA research and training in diverse cultural and social settings)26, 32, 33, 42-44 led the design of PLA in RESTORE. They provided training and standardized fieldwork protocols which enabled researchers to facilitate PLA in a consistent and rigorous manner across research sites.

The PLA process for RESTORE was based on a PLA mode of engagement and PLA techniques in a PLA-brokered dialogue between stakeholders. One of the striking features of PLA in general is the highly visual nature of the techniques used. Stakeholders work together to generate maps, charts and diagrams which function as powerful reference points (data displays) as they engage in verbal interaction, discussing, questioning, and learning from each other's perspectives, adding new data to maps and charts.8, 32, 44-47 The inherent analytical capabilities of PLA techniques aim to enable stakeholders to assess, correlate, categorize and/or prioritize data they are co-generating. PLA techniques, therefore, have the capacity to facilitate meaningful engagement that automatically incorporates co-generation and co-analysis of data “by” and “with” stakeholders. In a very practical way, then, stakeholders using PLA techniques engage in a structured, integrated, visual-verbal-tangible process of co-generating and co-analysing data which produces visual-tangible results.6, 25, 32, 43 This activity can appeal to a wide range of stakeholder groups, including those where literacy and/or numeracy challenges may feature, as a key aim is to ensure that stakeholders/stakeholder groups do not become disenfranchised during the research process. Data co-generation and co-analysis may occur during a single PLA session, or iteratively and in successive waves of fieldwork and/or data-generation encounters during PLA research, bringing stakeholders’ unique knowledge and insights to bear on emerging findings. This ensures that stakeholders’ perspectives influence the conduct and trajectory of research and research outcomes—a key hallmark of meaningful engagement.3, 25

The two PLA techniques we used in RESTORE in Stage 2 were Commentary Charts and Direct Ranking. A Commentary Chart is an interactive knowledge-exchange, knowledge-enhancement technique which allows stakeholders to exchange differential knowledge, expertise and perspectives. Box 1 provides information on generic steps for facilitating Commentary Charts, based on the expertise of the RESTORE project PLA trainers.

In RESTORE, PLA Commentary Charts were used to facilitate dialogue among stakeholders about the GTIs mapped in Stage 1. Full-text copies, summaries and/or PowerPoint presentations of the identified GTIs were made available to stakeholders. The Commentary Charts comprised three analytical categories—”positive aspects of the GTI,” “negative aspects of the GTI” and “questions to be checked out.” Stakeholders recorded their key deliberations, perspectives and insights about each GTI on the Commentary Chart (see Figure 1). For example, in Ireland, stakeholders met in one large group for each of five PLA sessions. They had identified a “local set” of five GTIs and co-generated five separate Commentary Charts. Where several stakeholder groups met separately (eg Austria) or were geographically dispersed (eg The Netherlands), charts were computerized and circulated around stakeholder groups by email, iteratively accruing additional data. On occasion, researchers took physical charts from one stakeholder group to the next, and data were added incrementally. As Commentary Charts “travelled” around stakeholder groups, the knowledge-exchange and knowledge-enhancing process continued.

The intended practical outcome expected of Commentary Charts in RESTORE was that they would present a visual, tangible data display of stakeholders’ knowledge, expertise and perceptions about the sets of GTIs. Stakeholders could then review the data display and continue their co-analysis activity as they began to use Direct Ranking to select a single GTI for implementation at local level.

Direct Ranking is an interactive technique for identifying priorities or preferences in a democratic manner. It yields a visual result in chart form. Box 1 provides a summary of the generic steps for this technique. In RESTORE, the specific application of Direct Ranking was to produce a clear, documented democratic result—a single GTI for implementation that stakeholders are willing to “sign up to” for Stage 3. The images selected to represent each GTI were photographs of the front covers of GTIs. The agreed ranking criterion was “Prioritize the GTIs in terms of the most-to-least suitable for implementation in our general practice setting and context.” Stakeholders had equal voting power as they had 20 paper clips each (see Figure 2). The intended practical outcome expected of Direct Ranking Charts in RESTORE was that they would present a visual, tangible data display of stakeholders’ decision about which GTI was considered most suitable for implementation in their setting.

Across research sites, the majority of PLA sessions involving Commentary Charts and Direct Ranking were each 2-3 hours in duration. The completed paper-based charts were computerized to preserve them and to make them readily available for further analysis.

2.4 Evaluation of the use of PLA techniques in RESTORE

Stakeholders at all sites participated in qualitative “speed” evaluations (SEs) to document their experiences of involvement in their PLA sessions. A “speed evaluation” is a brief verbal (digitally recorded) or written evaluation, which provides an opportunity for stakeholders to describe experiences in their own words and suggest areas for improvement. It allows researchers to “take the temperature” of the group, to build on positives, and, where possible, to plan suggested improvements for forthcoming PLA sessions. Speed evaluations usually occur at the close of a PLA process or session. Participants respond to an open-ended question in a rapid, interactive and spontaneous way.

For example, on completion of Commentary Charts, the question was as follows: “During this session, we worked together in a participatory way: listening, learning, creating our Commentary Charts. In a word or phrase, please comment on your experience of this.”

Stakeholders at the Irish site also completed a participatory evaluation (PE) because the facilitators there had extensive experience of this technique. This form of more in-depth collaborative evaluation is judgement-based and may be formative or summative. Stakeholders responded to questions focused on Direct Ranking and their experiences of engagement in the PLA process. An “open category” question also invited stakeholders to comment on any aspect of their experience, especially if there were any improvements and/or changes that could be made.

Speed and participatory evaluations were either noted or audio-taped and transcribed at each site, and collated as reports by the researchers. We had ethical approval to include data from all of these reports in the analysis for this paper, apart from the data from the English site.

All academic researchers involved in PLA fieldwork participated in in-depth reflection interviews (R Int), conducted by the PLA trainers, at the close of Stage 2 fieldwork (n=16). Interviews were conducted face-to-face or by Skype; interview length ranged from 50 minutes to 1 hour 45 minutes, depending mainly on the number of researchers involved (eg Austria=1 interviewee, The Netherlands=4). The interview schedule was circulated to all teams in advance. They were asked to reflect on the use of Commentary Charts and Direct Ranking with reference to their experiences of the techniques and their observations of stakeholders’ experiences of them (see File S2 for this topic guide). Interviews were audio-taped and professionally transcribed.

All evaluation data were collated and analysed following the principles of thematic analysis in qualitative research.48, 49 The PLA researchers at the Irish site, who have more than 25 years’ experience of qualitative interviewing, generated a “start list” of codes50, 51 derived from participatory research literature describing meaningful engagement (eg active inclusion, collaboration/collegiality, power-sharing) and its opposite (eg exclusion/passivity, researcher-controlled, powerlessness).6-8 This, augmented by repeated readings of researchers’ and stakeholders’ data, generated a final set of 33 codes. Each code was understood to incorporate its mirror or binary opposite. Data were collated under these codes to identify emerging themes.

The researchers also conducted a basic content analysis to establish the relative weighting of “positive” to “negative” evaluation comments in the final set of themes.52, 53 In keeping with our comparative case study design, the analysis of all evaluation data explored shared and differential findings across the five contexts.

Using different enquiry techniques (speed and participatory evaluations, focus group discussions and team “reflection” interviews) to explore researchers’ and stakeholders’ perspectives about their experiences of the same events (PLA sessions), we achieved a measure of triangulation, or cross-validation. As per Lincoln and Guba,49 prolonged involvement of researchers in fieldwork for “trust-building” and “knowing the culture,” coupled with persistent observation of stakeholders’ reactions to PLA methods and peer debriefing by research teams when producing regular field reports, contributed to study depth. This, in conjunction with the triangulation or cross-validation mentioned above, contributed to and enhanced the trustworthiness, credibility and dependability of the study.49, 53

3.1 Study sample

There was appropriate representation of stakeholders at each site by gender, age group, country of origin and type of stakeholder group (see Table 1), thus validating the purposefulness of the sample.

We monitored attendance at each session and over time. There was good, sustained involvement of stakeholders across settings for the duration of Stage 2, which involved six PLA sessions. Specifically, there were only minor variations in the sample, for example if a stakeholder could not attend due to work or personal commitments. Only two stakeholders dropped out (both from the English setting) during the fieldwork period (September 2012 to May 2013).

There was strong representation of the academic research team, as 16 of 18 individuals participated in PLA fieldwork and its evaluation (see Table 2).

3.2 Emergent themes

We identified five interrelated themes of stakeholders’ experiences that elucidate their positive perspective on the PLA techniques (see Table 3). Findings were relevant across countries and participant groups unless otherwise specified.

Stakeholders described their overall involvement in the PLA process of co-generation and co-analysis in ways that speak powerfully of (1) meaningful engagement in a “safe space”: active inclusion, collegiality, collaboration. They reported that group dynamics were positive. The working environment was considered safe, allowing stakeholders to readily and safely express diverse views in a relaxed and enjoyable manner. This was also noted and commented on by researchers.

Stakeholders described how there was (2a) enhanced learning throughout the process of using Commentary Charts and Direct Ranking. The Commentary Charts facilitated exposure to each other's perspectives about the GTIs and contributed to positive experiences of enhanced learning. Researchers considered that this placed stakeholders in a more informed position from which to prioritize GTIs during Direct Ranking. For example, in Ireland, details about time demands in general practice surgeries were described by the general practice stakeholders. This enhanced other stakeholders’ understanding of the clinical setting and impacted on their assessment and ranking of the GTIs that they were examining (see File S3 for a detailed example from the Irish setting).

It was clear that (2b) enhanced learning led to important new understandings and, on occasion, resulted in shifts away from long-held positions and towards new possibilities. This suggests that the sharing of diverse perspectives during co-analysis enabled stakeholders to broaden their horizons. Researchers’ comments reflected stakeholders’ sense of the “flow” between the two techniques. Researchers noted the interactive energy of stakeholders’ involvement in a PLA process that generated a “win-win” outcome.

Furthermore, the visual nature of Direct Ranking allowed stakeholders to engage in a nuanced decision-making process which was demonstrably democratic. This indicated (3) democracy-in-action. Stakeholders enjoyed positioning GTIs and Post-It notes (stickies) and distributing their 20 “paper clip” votes on the chart. Asymmetrical power relations were balanced in part by the fact that 20 votes per stakeholder meant equal voting power and equal opportunity to influence the outcome. Finally, by arranging the GTI images according to the vote count, the democratic outcome was immediately accessible to all and the chart showed the important result stakeholders had achieved.

Researchers noted the equalizing power of PLA, “levelling the playing field,” and that stakeholders with a lower level of language skills were not disenfranchised—they could “see” the result; the presence (density) compared to absence (paucity) of paper clips visually expressed what a numerical count confirmed.

Stakeholders recognized that their choice represented a fundamental input into the progress and trajectory of the research. Exercising power in this way contributed to a genuine sense of “ownership” of the research project: (4) power, ownership. Researchers commented on the genuineness/authenticity of stakeholders’ interactions—the willingness of some stakeholders to put others before themselves. This was even the case where the vested interests of typically powerful stakeholders might be expected to win the day, but where choices inclusive of all stakeholders actually won out.

An important outcome was (5) sustained engagement. Direct Ranking provided opportunities for stakeholders to experience “positive power,” and researchers noted that they often linked this to the “light” side of PLA: energy, enjoyment, fun and achievement. This “lightness” seemed to offset research fatigue and contributed to sustained meaningful engagement of stakeholders in Stage 2 research.

Negative comments were also identified. Speed evaluation data from The Netherlands comprised a total of 21 stakeholder comments. Of these, there were five negative responses. Speed and participatory evaluation data from Ireland, where evaluations were more extensive, generated 80 stakeholder comments. There were two negative responses. No negative comments were recorded in data from Greece or Austria. However, our content analysis showed that five of 15 stakeholders in Austria chose not to offer evaluation comments of any nature.

4.1 Summary of findings

Our findings show that Commentary Charts and Direct Ranking techniques, with their visual nature and inherent analytical capabilities, were experienced by stakeholders from both community and health-care settings as powerful tools for collaborative decision making. There was consensus among stakeholders and researchers that there were few negative experiences, and numerous multifaceted positive experiences of meaningful engagement: PLA created a “safe space” and a trusting environment in which they learned from each other's perspectives, gained enhanced knowledge via the co-generation of Commentary Charts and used these data to inform their co-analysis during Direct Ranking. Using these two PLA techniques involved stakeholders in an experience of “democracy-in-action” which was empowering and energizing, promoting a sense of ownership and sustained engagement in the research project.

4.2 Contribution to existing literature

There is limited knowledge about suitable methods for involving stakeholders in a meaningful (rather than tokenistic) way in health research partnerships.3, 11 There is evidence that participatory learning and action research approaches and methods seem promising.5, 32-36 In line with this, our findings show that the application of specific PLA techniques in RESTORE proved fit for purpose. Taken together, the five themes show that academic researchers and stakeholders from community and health sector backgrounds reported that it facilitated collaborative decision making and meaningful engagement, automatically incorporating co-generation and co-analysis of data by diverse stakeholders operating in diverse primary care settings.

Furthermore, we provide new evidence about how PLA techniques are capable of delivering this—their combined visual, verbal and tangible nature and inherently analytical capabilities involved stakeholders in structured, integrated co-generation and co-analysis of research data, which delivered practical democratic results at all five sites. This empirical evidence about the capacity of PLA to “deliver” meaningful engagement in data generation and co-analysis is a significant addition to a literature that calls for methodological innovation in this sphere. These findings provide important empirical data about experiences of participation that are lacking in the literature.3 They support Kothari's assertion9 that the types of techniques used by researchers can constrain or enable stakeholder involvement (eg findings from Theme 1 that the use of PLA created a safe and meaningful space for engagement and from Theme 5 that it led to sustained engagement for the duration of the fieldwork). They are also in line with Brett et al.,1 elucidating that involving stakeholders in data analysis ensures a broader interpretation of data (eg findings from Theme 2 about enhanced learning).

In keeping with Tierney et al.,5 we actively explored both positive and negative experiences. In line with Domecq et al.'s review,11 stakeholders were predominantly positive about their experiences. In particular, themes 3 and 4 (Democracy-in-action; Power, ownership) reveal the ways in which power imbalances were reduced and that experiential learning, rather than reiteration of professional concepts, became the common ground upon which democratic decision making took place. This is an important function of PLA: to “level the playing field” where asymmetric power relations between stakeholders/stakeholder groups may exist.7

Our results show that this levelling also occurred between stakeholders and researchers: throughout the Commentary Charts and Direct Ranking, it was stakeholders who exercised power and took on the key responsibility of selecting a GTI for implementation, thereby “setting the agenda” for the final stage of RESTORE fieldwork. Researchers, in their capacity as catalysts, facilitated but did not control this process. This shared ownership and agenda-setting takes us firmly beyond tokenism and towards a “shared power” approach, enhancing the research partnership.

The rare, negative experiences reported by stakeholders in this study warrant attention: the time and pace of research, the need for comprehensive information to facilitate equitable participation, and the possibility that bringing different types of stakeholders together may, on occasion, actually reduce a sense of active inclusion for a minority. While concerns about time demands and tokenistic participation have been reported previously,11 the empirical findings about disadvantages of interactions in mixed stakeholder groups are new. Therefore, this analysis of two PLA techniques to support such dialogues is an important contribution to the literature.

4.3 Methodological critique and suggestions for future research

We were unable, for site-specific ethical reasons, to include the use of stakeholder evaluation comments from the English site in our thematic and content analyses. However, we were able to ameliorate this by including comments from researchers’ reflection interviews.

We cannot claim representativeness of findings for the qualitative study data presented here. However, we emphasize that in this comparative case study, spanning five European countries with very different primary care systems, the same PLA techniques were used and successfully involved diverse stakeholders in data generation and co-analysis.

Regarding the PLA techniques employed, there was variation in our use of PLA Commentary Charts: stakeholders reviewed the Commentary Charts together at the same time in some settings, while some were physically removed from one another and reviewing the Commentary Charts after others had done so. This may have created variation across the sites and this possibility would be worth exploring in future work.

Regarding evaluation methods employed: speed evaluations are efficient and the key messages recorded were similar to those in the additional participatory evaluation conducted at the Irish site. However, the brevity of speed evaluations does not encourage rich, lengthy, in-depth stakeholder responses. The additional participatory evaluation conducted did provide richer data, while researchers’ reflections also added to the quality and depth of evaluation data. Therefore, we suggest using an array of PLA evaluation techniques in future projects to explore all stakeholders’ experiences of involvement in greater depth.

RESTORE ended before our thematic analysis of stakeholders’ evaluation data took place, and we did not, therefore, benefit from their contribution to the development of codes and categories for thematic analysis, nor from their potential insights about the relevance and veracity of evaluation results. While we are confident that data saturation was achieved as the analysis reached a point where the codes and themes were comprehensive, we acknowledge the lack of member checking. In future projects, it would be apposite to invite stakeholders to co-generate evaluation criteria and to co-analyse the results of evaluation data, thus closing the circle of “involvement.”

To add to the evidence base, we need further research and evaluation to explore whether and how PLA techniques might work when applied in projects with very different research foci and stakeholder groups to those in RESTORE. The specifics of involving community and health sector partners in analysis of other qualitative methods, such as interviews and focus groups, would also be valuable.