Levelling the playing field in police recruitment: Evidence from a field experiment on test performance

1 INTRODUCTION

In the United States, three out of every four police officers are white, making police forces almost 30 percentage points more white than the communities they serve (Ashkenas and Park 2014). After incidents in Ferguson and other cities, both the popular press and policy-makers have highlighted the urgency of improving the diversity of police forces across the country (United States Department of Justice 2015). Similarly in the United Kingdom, all major political parties listed improving the diversity of the police as part of their 2015 election campaign platforms. At its core, this goal is grounded in theories of representative bureaucracy; the argument that a more diverse civil service will allow government to more accurately reflect the preferences of—or better serve the needs of—a diverse public (see Kennedy 2014 for an overview). Specifically, a more representative bureaucracy may impact citizens' perception of public sector legitimacy (Theobald and Haider-Markel 2009) but may also directly influence the ability of the public sector to serve the public effectively (Weitzer and Tuch 2006; Bradbury and Kellough 2011). In policing, recent evidence suggests that these theories hold true. Not only are more diverse police forces perceived as more trustworthy and fair (Riccucci et al. 2014), they also correlate with positive outcomes ranging from less crime (Hong 2016) to better reporting of sexual assault (Meier and Nicholson-Crotty 2006). While we may need more causal evidence linking changes in diversity to better policing outcomes, the public management question remains: how can public sector organizations recruit a more diverse civil service?

The literature has often focused on assessor bias to explain racial disparity in recruitment. Multiple audit studies have shown, for example, that people with names that sound traditionally black (Bertrand and Mullainathan 2004) or more foreign (Oreopoulos 2011) receive fewer call-backs even if they have identical CVs. Other studies show biases in interviews and other stages in the assessment process (see Linos and Reinhard 2015 for a summary). However, more and more components of recruitment are now automated or scored without using human assessors. Yet large racial or gender gaps in performance persist. This is either due to the test being biased (see Jencks and Phillips 1998 for an overview) or to the testing context. Importantly, potential solutions to reducing assessor bias have shown mixed results. Large field experiments—all conducted outside a police recruitment context—have shown that blinding CVs may not lead to recruiting more individuals from underrepresented groups (Behaghel et al. 2011); and implicit bias training does not seem to improve behaviour and may even backfire (Duguid and Thomas-Hunt 2015). Public managers may therefore have to turn to the candidate experience to try to improve recruitment outcomes for people of colour or other underrepresented groups.

A large parallel literature has documented racial performance gaps in educational settings and test-taking. The seminal work on stereotype threat (Steele and Aronson 1995) has been shown to affect student performance on tests, and has been replicated in the lab and in the field multiple times. For example, telling a woman before she takes a test that female test-takers do worse has been shown to significantly decrease their performance (Spencer et al. 1999). The mechanism underlying these effects is probably linked to increased anxiety (Osbourne 2001), emotional regulation (Johns et al. 2008), and mental workload (see Schmader et al. 2008): individuals worry about confirming a stereotype about themselves or their group, which in turn leads to underperformance (see Blascovich et al. 2001).

We expect that stereotype threat applies beyond educational settings, but there may be additional mechanisms at play in recruitment-specific environments. For example, in a recruitment process, both the applicant and the assessor may be using the assessment tasks as a way to ascertain ‘person–organization’ (P–O) fit, especially if these tasks are presented as realistic work scenarios. Drawing on Cable and Edwards' (2004) framework for understanding P–O fit, candidates may be using the selection process to assess values congruence or supplementary fit. But they may also be trying to ascertain complementary fit: whether the organization will satisfy an individual's psychological needs. While psychological needs can be broadly defined, according to self-determination theory they include autonomy, competence, and relatedness (Deci and Ryan 2014). Here, the notion of belonging uncertainty (Walton and Cohen 2007) may be most relevant. If individuals are uncertain of their social bonds—or levels of relatedness–and are trying to gauge how they will fit within an organization, cues that suggest social isolation may adversely impact performance. At this stage in the recruitment process, the assessment tasks themselves may provide cues of belonging or non-belonging that are especially salient for those individuals who are uncertain of their likely social bonds in a given organization. More specifically, we expect that individuals who are underrepresented—in this case black and minority ethnic applicants—would be more likely to be uncertain of their social bonds on the job.

While the phenomenon is clear, whether and how it might be reversed is less so (Ambady et al. 2004). By definition, stereotype threat operates in environments where a stereotype is presumed to be relevant. Thus, it is plausible that priming someone to view the presumed stereotype as irrelevant may improve performance. We know, for example, from early work in this field that African American students performed better on an IQ test if it was presented as a hand–eye coordination test, and therefore any negative stereotype on IQ was not relevant (Katz et al. 1964). More recent work suggests that when intelligence is presented as malleable, rather than fixed, African American GPAs increase, compared to a control group (Good et al. 2003). Moreover, if the underlying mechanism is increased anxiety or mental workload, as has been shown in other cases (Osborne 2001), then language that reduces anxiety should diminish the racial gap in achievement.

Walton and Cohen (2007) describe belonging uncertainty as a hypothesis—rather than a belief—that one may not belong socially, and therefore something that is malleable. For example, in one study, reframing adversity on campus as common and not as an indictment of whether a person of colour ‘belongs’ seems to significantly impact GPA for African American students (Walton and Cohen 2011). Thus, belonging uncertainty should respond (in the Bayesian updating sense) to counter-evidence that implies social belonging. We hypothesized that since applicants from underrepresented groups should be more likely to feel uncertain about their levels of belonging, they should also be more responsive to cues of belonging than would be their white counterparts. The slope—as it were—of a cue-performance line is steeper for non-whites, although the line itself may lie below that of whites. This hypothesis is shown graphically in figure 1.

In addition, a growing body of literature suggests that values affirmation exercises also positively affect performance. Self-affirmation theories suggest that when an individual's sense of self-integrity is threatened—their ‘moral and adaptive adequacy’ (Steele 1988)—they seek ways to restore their self-worth. Priming individuals to think about what they value may reduce anxiety and help reduce perceived threats; specifically, by reflecting on self-defined core values, individuals can re-establish personal worth, ultimately protecting themselves from psychological threat that impacts performance (Miyake et al. 2010; Harackiewicz et al. 2014). Importantly, although the initial theory focused on individual threats and individual responses, the same mechanism can apply to social group identities and social group threats (Sherman and Cohen 2006). Values affirmation exercises operate by reducing the threat of a negative characterization of a valued social identity. As a result, asking candidates to reflect on their values while providing a clear pathway for their social identity to be specifically valued in a police setting may increase their sense of self-worth in a similar fashion. Indeed, because values affirmation exercises operate by buffering against anxiety and mental workload (Creswell et al. 2005), they should improve performance for those candidates who face additional anxiety due to stereotype threat.

With this literature and hypotheses in mind, it seemed plausible to design an intervention intended to reduce the racial performance gap on assessment tasks for police recruitment. Just as in educational settings, we hypothesized that—holding ability and skills constant—non-white applicants would perform worse on tests. We also hypothesized that non-white candidates may be uncertain about their social value within the police force and as representatives of the police among civilians. Thus, we hypothesized that changing the language that applicants saw before taking the test to (a) make it more welcoming, (b) remove anxiety-inducing sentences and (c) ask candidates to reflect on their social value in the police would improve performance of non-whites and not affect white applicants. The design and results of the field experiment are presented below.

2 METHOD

3 RESULTS

Table 3 displays the main results of the trial. Column 1 presents a simple regression model on the percentile score for applicants by treatment group. There is no statistically significant effect on the population as a whole. Column 2 presents the treatment interacted with race. Here effects are highly significant. Non-white applicants in the treatment group gained 12 percentage points in their percentile ranking. White applicants show a two percentage point increase in their percentile ranking. Controlling for age category and gender in column 3 does not change the results significantly. However, it is interesting, given the make-up of the police, that women seem to do significantly better (4.5 percentage points better) on this online test.

Figure 3 and table 4 present the results in terms of the probability of passing the test. Each column presents the marginal effects of a logistic regression model, where the outcome variable is 1 if the applicant passed to the next stage in the process and 0 otherwise. The results match table 3, but also show the practical impact of the treatment. Non-white applicants in the treatment group are 21 percentage points more likely to pass the test. Given that the control group pass rate for non-whites is 40.6 per cent, this is essentially a 50 per cent increase in the probability of passing the test. Most importantly, while the intervention closed the racial gap in percentile score by about 80 per cent, it fully closes the racial gap in the probability of passing the test because of the way scores are distributed. More precisely, as figure 2 suggests, the peak of the score distribution for non-white applicants in the control group is right below the cut-off for passing (which is set at 110), so improving scores has a disproportionately larger effect on pass rates. As is also shown, white applicants in the treatment group are three percentage points less likely to pass, but this effect disappears once we control for age and gender.

While this trial, by design, does not determine the relative contributions of each change in the email, we can use additional analyses to rule out some mechanisms and provide suggestive evidence of others. Table 5 does just that. Given that the email itself was sent before people took the test, and that all recruitment processes have some attrition at each stage, it is possible that the impact of the intervention lies in getting different people to take the test. This does not seem to be the case. Column 1 shows whether being in the treatment group predicts the probability of taking the test. For a non-white applicant, being in the treatment group does not predict whether an applicant will take the test. It is also possible that the intervention motivates applicants to concentrate more during the test or take the test more seriously. While it is hard to observe concentration and overall effort, we can measure the time applicants took to answer questions on the test as a proxy. Column 2 presents a regression on the impact of treatment on the time taken to complete the test. This, too, does not seem to cause the effect; being a non-white applicant in the treatment group does not predict how much time the applicant spends on the test. Indeed, in the control condition, white applicants seem to be spending significantly less time on the test than their non-white counterparts.

The SJT itself splits questions into four categories based on the competencies meant to be tested. These are: Communication and Empathy, Customer Focused Decision Making, Openness to Change and Adaptability, and Relationship Building and Community. Table 6 identifies the results by category. Of these four areas, Communication and Empathy seems to drive the effects of the intervention, which given its nature, may seem appropriate. We primed individuals to feel more comfortable and to recall their values and their own role in their community. If this is true, we should not expect the intervention to affect an applicant's level of adaptability, for example, and indeed we do not see an effect in this category. As table 6 shows, with respect to relationship-building questions, the direction and size of the effect is in line with our hypothesis, but the coefficient is slightly below the statistical significance level of 10 per cent (p-value of .105). Thus, it is difficult to interpret this result further.

4 DISCUSSION

While public administration scholars have highlighted the challenge of government recruitment (Light 2000) and the promise of a representative bureaucracy (Selden 1997) for more than two decades, how to improve recruitment and selection in government is less clear. This study responds to Pitts and Wise's (2010) call for more ‘useable’ research on workforce diversity in the public sector by rigorously testing the impact of a low-cost nudge in police recruitment. Given the public policy importance of increasing diversity in the police, it is worthwhile to investigate every element of the recruitment and selection process from a behavioural perspective. This analysis looked at just one.

In this field experiment we used the principles around stereotype threat, belonging uncertainty and values affirmation to test the impact of making small changes to the email inviting candidates to take part in a Situational Judgment Test. This simple intervention, imposing neither selection risks nor significant costs, managed to increase the probability that a non-white applicant passed the SJT by 50 per cent, closing the gap in pass rates between non-white and white applicants who reached this stage in the process. The results suggest that reduced anxiety and feeling more comfortable with one's role in the community may be driving the results. The results also suggest that some other plausible mechanisms—such as motivation to take the test and time taken to complete the test—do not seem to be driving the effect. Given previous research suggesting that stereotype threat can be induced even without long-term historic stigma (Aronson et al. 1999), we also tested (ex-post) the impact of the intervention on gender. We did not find a statistically significant effect although, directionally, women in the treatment group fare better than women in the control group. That is, the probability of passing the test for women in the treatment group is 62 per cent, versus 59 per cent in the control group (p-value of .38). In contrast, the probability of passing the test for men in the treatment vs. control group is 57 per cent vs. 55 per cent, respectively (p-value of .67). These results match our expectations because this particular police force had many women in important and visible leadership positions, and because women—on average—performed better on these tests than men.1

To the best of our knowledge, this is the first randomized controlled trial measuring how small nuances in language can reduce stereotype threat during testing for public sector recruitment. Importantly, the intervention did not require extensive or frequent writing as do some other successful interventions (Harackiewicz et al. 2014). As a result, one contribution of our research is to demonstrate that even lighter-touch interventions can be effective if introduced at the appropriate time. While there are clear implications for other police forces aiming to increase the diversity of their force, this type of research should also have implications for other policy areas where the diversity of the workforce seems to pose a significant challenge not only for public managers but for the perceived legitimacy of the services delivered. Indeed, the results of this study should inform current policy debates around recruiting more diverse teachers (e.g., Boser 2011), city managers (e.g., Alozie and Moore 2007) or other frontline workers. Given that some oft-discussed policy alternatives range from lowering standards to implementing affirmative action initiatives—all of which have faced significant push back—this type of study also suggests that there are low-hanging fruit in improving current recruitment processes, which can operate alongside broader discussions on large policy shifts.

From a public management perspective, recommending that organizations employ a combination of different behavioural concepts to positively impact diversity in recruitment is appropriate. The social impact might be great, and the use of multiple behavioural concepts does not add operational costs to the entity's bottom line. However, given the surprisingly strong results of this study, future research should help determine the relative contribution of each change, and should iterate on the most effective components of the intervention. Of course, the ultimate question is whether getting more diverse candidates through this stage in the process will have knock-on effects in getting them both to be hired and to remain in the police force. This sample is too small to make that determination, but there does not seem to be a statistically significant effect on who passes through to the end of a lengthy assessment process.2 A related follow-up question is whether having a more diverse successful applicant pool affects who applies to join the police in the future. Ultimately, follow-up studies conducted in cooperation with larger police forces, where racial tensions may be more apparent, will help determine whether the results of our study apply more broadly. While the search to find large, system-wide improvements to selection and recruitment continues, this study suggests that there may be many simpler tools available for organizations to slightly alter testing and significantly support their applicants.