Measurement and evaluation of community engagement in complex, chronic medical conditions: HIV and obesity as exemplar conditions

2.1 Search strategy

Searches were conducted in Embase, Medline, Informit Health Collection, Global Health, Cochrane Library, SocINDEX, and Psycinfo. The search strategy was formulated by ADB, SA, and CB with input from a research librarian. Additionally, search terms from other reviews of community engagement informed the search terms.^{4, 8, 25, 26} Key concepts and search terms are documented in Table S1. The original search was run on June 27, 2021. A hand search of reference lists from included studies was also undertaken.

2.2 Study selection

The review targeted quantitative studies of initiatives designed to prevent obesity or HIV that explicitly use community engagement, based on the definition provided by the WHO²³ as a component of the initiative. To be included, studies had to be peer-reviewed original research published after 1981 and written in English. Studies were included if they presented quantitative measures of community engagement in prevention. Studies were not included if the only quantitative measure of community engagement was a numeric rating assigned to qualitative data if the study provided quantitative data related to the health outcome of interest but not about community engagement specifically, if the study referred to treatment, or if it only provided quantitative data on an individual's engagement in their own health, as opposed to promoting the health of others.

Duplicates were removed using a combination of the Systematic Review Accelerator,²⁷ Endnote, a method designed by Bramer, Giustini, de Jonge, Holland and Bekhuis,²⁸ Covidence,²⁹ and manual screening. Covidence was used to facilitate study screening. After the initial screening of abstracts, all articles at the full-text review stage were independently reviewed by two reviewers (ADB, JH, TF, PF). One author (ADB) screened all articles at the full-text phase, and three others served as second reviewers (JH, TF, PF). Reviewers worked independently and discussed discrepancies, closely considering the original criteria to reach a final decision, with input from a third reviewer in the case of unresolved discrepancies.

2.3 Data extraction

ADB, SA, CB, JH, and TF collaboratively developed a data extraction schema, which included authors, year of publication, study title, country, project name, project objective, whether it was about HIV or obesity prevention, term or phrase used for community engagement, description of community members carrying out community engagement, study design, type of tool used to collect community engagement data, number of participants, community engagement measurement construct, measured enablers or barriers to community engagement, whether an increase in community engagement over time was reported, and whether there was an association between community engagement and the health behavior or outcome of interest. The term used for community engagement was identified by searching each article for words that (1) aligned with the WHO definition and (2) were connected by the authors of the paper to the quantitative construct measuring community engagement. ADB extracted data from all studies, with TF or JH cross-checking 20% of the studies for accuracy.

2.4 Quality assessment

The Joanna Briggs Institute's critical appraisal tools were used to assess study quality, based on the study type.³⁰ The checklist gives each study a rating of yes, no, unclear, or not applicable for between 8 and 12 items (depending on the study design). Each study was given a numeric rating based on the number of ‘yes’ ratings given. Quality assessment was not used to eliminate studies from the review. ADB assessed all studies, with TF or JH cross-checking 20%.

2.5 Data synthesis

To synthesize the results, data were coded based on the attributes listed above to identify patterns in how community engagement in HIV and obesity prevention was measured. The papers were then grouped by the tool they used for measurement. The tools were coded to the 11 community engagement principles developed by Goodman, Sanders Thompson, Johnson, Gennarelli, Drake, Bajwa, Witherspoon and Bowen²¹ to identify similarities and differences between measurement tools and patterns between measures chosen and effectiveness. Tools to measure community engagement were additionally classified into two categories: internal and external, reflecting two major trends in the measures that emerged from the analysis. Internal measures primarily gauged the intrinsic attributes and perspectives of individuals, such as their knowledge, attitudes, or self-perceived skills related to community engagement. In contrast, external measures evaluated tangible actions or changes in the broader community environment, capturing the observable outcomes that arise due to the collective efforts of community members in health initiatives.

3.1 Description of included studies

The 55 selected studies demonstrated various characteristics of community engagement measurements in HIV and obesity prevention. Table S2 displays the basic characteristics of these studies, which are organized by outcome (HIV or obesity), then by study design, and finally by year. Most HIV studies were conducted in the US (n = 15),^31-45 several countries across Sub-Saharan Africa (n = 11),^46-56 or India (n = 10).^57-66 The remaining studies (n = 6) were spread across the Americas, Asia, and Europe.^67-72 The US similarly had the most studies measuring community engagement in obesity prevention (n = 7),^73-79 but in contrast with HIV, the remaining studies were conducted in Canada (n = 3),^80-82 Australia (n = 1),⁸³ Italy (n = 1),⁸⁴ and one paper comparing two studies in the US and Australia.⁸⁵ Examples of community engagement measures in HIV existed across all continents except Oceania and represented a wide range of types of countries, whereas in obesity prevention, these measures were only documented in high-income countries.

Publications in the HIV prevention literature started earlier (in 1994)³⁵ than in obesity prevention (in 2010).⁷⁷ Recently, the number of studies in both areas appear to have increased at a similar rate, though rapid growth in HIV studies started earlier (2010)^{39, 49, 50} than in obesity (2014)⁷⁸ (Figure 2). In HIV studies, 9 of the 10 studies conducted in India were part of Avahan, an initiative launched by the Bill and Melinda Gates foundation in 2003 to combat HIV in India, distinguished by a focus on community engagement, being data-driven, and operating at scale.^57-65 In obesity prevention, there were more examples of projects with multiple publications: Sustainable Childhood Obesity Prevention through Community Engagement (SCOPE) (n = 3),^80-82 Youth Engagement and Action for Health (YEAH!) (n = 3),^{74, 78, 79} Romp and Chomp (n = 2),^{83, 85} and Shape Up Somerville (n = 2).^{75, 85} Uniquely, one of the publications used the same measurement questionnaire across two different countries, Romp and Chomp and Shape Up Somerville, to further refine and develop the use of the questionnaire.⁸⁵

3.2 Definitions of target populations and community engagement

With respect to target populations for community engagement, both the HIV and obesity prevention described community members who were carrying out engagement as (1) professionals who work directly with grassroots community (n = 8; n = 5, respectively),^{31, 39-41, 49, 69, 71-73, 76, 80, 83, 85}; (2) residents of a geographic area (n = 3; n = 4),^{33, 48, 54, 75, 77, 81, 82} or (3) young people (n = 7; n = 4).^{34, 36, 44, 46, 49, 51, 52, 74, 78, 79, 84} HIV had several additional descriptions related to populations considered at risk, specifically sex workers (n = 11)^{47, 57-65, 67} and men who have sex with men (n = 6).^{44, 45, 47, 65, 66, 70} In other studies in HIV prevention, community was defined based on social identities, particularly race or gender (n = 15).^{31, 34, 35, 37, 38, 40, 42, 44, 45, 50, 51, 55, 56, 68, 86} The terms used to describe community engagement in both areas varied considerably, with engagement being the only overlapping term.

3.3 Study design and measurement characteristics

The majority of HIV and obesity studies relied on self-report surveys or interviews to measure community engagement (e.g. participants' level of agreement that their community would come together in the face of a problem affecting their community) (n = 30 and n = 7, respectively).^{32-40, 46, 48, 50-52, 55-60, 62-65, 67-70, 72-76, 79, 84-86} However, there were examples of other ways to measure community engagement, such as observation (n = 4 and n = 5),^{47, 49, 54, 61, 78, 80-83} and among the HIV studies, network data (n = 2)^{44, 66} and group consensus (n = 1).⁷¹ A small number of studies combined multiple methods to measure community engagement (n = 5 and n = 1).^{31, 41, 42, 45, 53, 77} About half of the studies were cross-sectional, and that approximate ratio has persisted since community engagement research in HIV prevention started (Table S1). Since 2010, there has been a steady increase in the number of randomized control trials (RCTs) incorporating measures of community engagement (Table S1). The HIV literature showcased a range of participant sample sizes when measuring community engagement from 1 to 100 participants (n = 13)^{31, 33, 35, 39, 40, 49, 54, 66, 68, 69, 71, 72, 86} to over 2000 participants (n = 5).^{34, 51, 57, 60, 65} Most of the obesity studies were smaller (1–100 participants, n = 6),^{73, 75, 77, 79, 83, 85} though there were examples of 101–500 (n = 3)^{74, 78, 84} and 501–1000 (n = 1).⁷⁶

The results across the studies support the positive association between community engagement in prevention and health outcomes (Table 1). Of the 22 studies in HIV prevention that statistically tested an association between community engagement and improvement in HIV outcomes or behaviors, 19 studies found a positive association^{34, 38, 42, 44, 47, 48, 51, 55, 57, 58, 60-62, 64, 65, 67, 69, 70, 86} and all four of the obesity prevention studies that carried out a similar analysis found a positive association (Table 2).^{74, 79, 83, 84} Thirteen of the fifteen studies that measured community engagement in HIV prevention at multiple time points found an increase in community engagement over time^{33-35, 38, 44, 45, 47, 49, 52, 61, 68, 71, 86} and six of the seven obesity prevention studies also found an increase (Table 3).^{74, 79-83}

Tables 2 and 3 also display the results of the synthesis of the measurement tools. Both HIV and obesity had a mix of tools that used internal and external methods, along with examples that used multiple measures that made use of both. Coding the tools to the community engagement principles yielded insights into where community engagement in HIV and obesity prevention is focused, with similar patterns emerging across both fields. ‘Foster co-learning, capacity building, and co-benefit for all partners’ and ‘Build on strengths and resources with the community’ were the most common principles measured, and significant gaps emerged in measuring ‘Disseminate findings and knowledge gained to all partners’, ‘Involve a cyclical and iterative process in pursuit of objectives’, and ‘Plan for a long-term process and commitment.’ Notably, HIV had at least one tool for each of the principles, providing potential for translation into obesity prevention.

4.1 Statement of principal findings

This systematic review summarized the published literature of quantitative measures for community engagement in the contexts of obesity prevention and HIV prevention. However, coding the measures into community engagement principles²¹ revealed patterns in what is measured in both fields and where there may be gaps. For HIV and obesity prevention research, most measures were self-reported on a single occasion, but there were some examples of observation methods used over multiple time points, particularly in HIV prevention where RCTs were increasingly being used to evaluate effectiveness. When statistical methods were used to measure changes in engagement over time or associations between community engagement and health behaviors or outcomes, community engagement was generally found to increase and be associated with health outcomes or behaviors of interest.

Given that only a small number of studies reported no significant association with health outcomes or no increases in community engagement, there is insufficient data to determine whether certain community engagement principles are more effective than others. This may be partly due to positive-outcome bias in peer review,⁸⁷ where studies with positive findings are more likely to be published than those with null or negative results. Many of the community engagement studies we reviewed did not report on health outcomes or failed to analyze the relationship between community engagement and specific outcomes. As in other fields, it is crucial that future research publishes studies where community engagement had no effect or even adverse effects, to improve our understanding of effective approaches. One way to address this is by responding to the call for the use of multiple measures of community engagement. Although many studies employed measures that reflected multiple community engagement principles, they did not conduct analyses that differentiated between them. If future studies use multiple measures and test each against relevant health outcomes, more nuanced insights into which components of community engagement are effective—and which are not—could emerge.

Previous reviews in obesity prevention have identified community engagement as critical to obesity prevention initiatives but have not reported on actual measurement of community engagement.^6-11 This has created tension where there is growing acknowledgment of the importance of community engagement but in many cases, insufficient data to study a link between community engagement and health outcomes such as obesity prevalence.⁸⁸ Our review affirms the importance of community engagement and highlights the gap in measurement, with only 13 studies across eight projects quantitatively measuring community engagement in obesity prevention since 2014. It also identifies candidate measures of community engagement from recent studies that have used rigorous designs and quantitative measures of community engagement,^80-82 complex modeling,⁸³ and internal and external measures of community engagement.⁷⁴ These measures have been classified based on an existing framework to reveal areas where there are existing measures to build on and gaps that need to be addressed.

A similar tension has been observed in the broader field of public health.^{1-5, 12-14} A large review carried out by Popay, Attree, Hornby, Milton, Whitehead, French, Kowarzik, Simpson and Povall¹ in 2007 noted that study designs (e.g. cross-sectional) at that time did not allow for attributing causality to community engagement in health outcomes. Further, while RCTs are a stronger design for studying causality, Rifkin¹² highlights the drawbacks of RCTs in understanding community engagement because of the wide definitions of “community” and “participation,” creating context-specific results that are difficult to generalize, calling instead for community engagement to be viewed as a process, involving multiple measures. O'Mara-Eves, Brunton, Oliver, Kavanagh, Jamal and Thomas⁵ echo this call for studies that use multiple measures that capture the various elements of community engagement over time to evaluate community engagement due to its complexity. Considering measures in relation to the community engagement principles contributes to developing approaches of measuring community engagement with multiple interrelated measures.

Our review has offered some candidate measures and methods in HIV that could be applied to obesity. Since 2011, there have been six randomized control studies in HIV prevention that used community engagement measures at multiple time points.^{12, 44, 45, 48, 52, 53} The 42 studies within HIV prevention in this review represent a range of constructs to measure community engagement, and 20 of those studies measured community engagement over time. Community engagement principles covered by HIV but not obesity offer strong initial candidates for trial translation. Both HIV and obesity prevention may be able to use these diverse examples from HIV to continue to develop measurement of community engagement in a way that is responsive to ongoing challenges.

4.2 Strengths of the study

Previous systematic reviews of the community engagement literature have either taken a broad quantitative approach seeking an association between community engagement and health outcomes or a qualitative approach, summarizing barriers and enablers to engagement. A strength of this review is that it strikes a middle ground, pulling together details about specific quantitative measures used to evaluate community engagement and pointing to future research to develop community engagement measures. Another strength of this review is the juxtaposition of two distinct health conditions, offering a fresh perspective for future obesity prevention research, especially given the increasing calls to realize the potential of community engagement. The review also adhered to PRISMA and PROSPERO guidelines to ensure rigor.

4.3 Weaknesses and limitations of the study

One limitation is that this review was limited to the published academic literature. Most community engagement work happens at organizational and grassroots community levels and is never intended for, nor makes it to published academic literature. While this means multiple examples are likely to have been overlooked, the intention here is to draw together what can be learned from studies that have proceeded through peer review and identify replicable methods and detailed critique of the pros and cons of the tools applied.

We limited the data corpus to studies published in English and the studies returned were disproportionately from high-income countries, despite the high burden of both HIV and obesity in other parts of the world. Additionally, different countries may have different relative amounts of quantitative and qualitative research focuses, meaning the focus on quantitative measures focuses on research in countries that may favor quantitative research. The findings of this study therefore should not be interpreted to be about community engagement in HIV or obesity prevention in general, but rather specifically focused on quantitative, published measures that could be built on in future work.

Assessing the quality of the studies in this review was challenging. Many quality assessment tools focused on community-based or action research are focused on qualitative research and are challenging to apply to quantitative research. On the other hand, quantitative evaluation tools embrace traditional linear paradigms and focus on health outcomes, as opposed to processes like community engagement, making them not always fit for purpose.

We used a framework based on community-engaged research to synthesize the measures we found rather than one developed for prevention. Using this framework demonstrated how community engagement in other health areas outside of prevention may be relevant to prevention. The focus areas and gaps in the obesity and HIV prevention literature, and the concept of internal and external measures, provide starting points to develop and empirically test a framework that is tailored to prevention.

4.4 Implications and future research

Measures of community engagement in HIV prevention and obesity prevention demonstrated similarities that indicate at least some of the measures in HIV prevention could be trialed in obesity prevention. Both domains have implemented measures in high-income countries, utilized internal and external engagement metrics, and featured similar definitions of community members. Where there are differences, there may be opportunities for the two fields to learn from one another. For example, while obesity emerged as a health issue in high-income countries, it has now spread globally,⁸⁹ meaning examples from the HIV literature measuring community engagement outside of high-income countries could be relevant in obesity prevention. As the complexity and global scale of obesity are recognized,⁸⁹ there is a need to consider deepening engagement with a broader range of community members, and the studies in HIV with large sample sizes may offer opportunities to do so in obesity prevention. On the side of HIV research, the importance of structural actions is recognized,^{90, 91} and obesity prevention research offers quantitative measures of community engagement defined in ways directly relevant to structural actions such as advocacy, partnership, and co-production. The obesity prevention field consistently calls for evaluating complex community-led initiatives.⁷ HIV literature, with its emphasis on constructs like mobilization or empowerment, could offer valuable insights for these evaluations. This review only considered two health conditions, and future work could apply a similar approach to other health conditions that utilize community engagement to continue to develop a fuller picture of the literature. This review illuminates numerous opportunities to iterate upon existing community engagement measures across health domains, particularly in cases where there already have been multiple uses of the same measure.

Across both health issues, a wide range of constructs were measured to reflect community engagement. While the diversity of terms allows for more specificity in how to define activities that fall under the broad umbrella of community engagement, the definitions of these terms overlap and were not used consistently, creating fragmentation and limiting the ability to build on previous work. In spite of these challenges, HIV and obesity prevention both showcase examples of the multiple studies over time building on and refining measures of community engagement, and the translation of measures across different studies. Both HIV prevention and obesity prevention showcase a diverse range of approaches to capturing internal and external measures of community engagement to capture the range of sites where community engagement may be measured.