The National Survey of Religious Leaders: Background, Methods, and Lessons Learned in the Research Process

Generating the NSRL Sample

Developing a nationally representative sample of clergy from across the religious spectrum requires overcoming the same obstacle faced in developing a nationally representative sample of congregations: the absence of a comprehensive and unbiased sampling frame. Overcoming this obstacle requires building on the same insight that inspired the congregations sampling strategy first implemented by the 1998 National Congregations Study (Chaves et al. 1999). That is, just as the congregations attended by a nationally representative sample of individuals constitute a nationally representative sample of congregations, the leaders of congregations attended by a nationally representative sample of individuals constitute a nationally representative sample of congregational leaders. One can, therefore, bypass the need for a comprehensive list of clergy from which to sample by starting with a nationally representative sample of individuals, generating a nationally representative sample of congregations by asking those individuals who attend religious services to say where they attend, and, finally, identifying the leaders of those congregations. This was the clergy sampling strategy pursued by Jackson Carroll et al.’s 2001 clergy survey and by Cynthia Woolever's 2008−09 survey, which is why those efforts stand out among earlier clergy surveys. It also is the strategy pursued by the NSRL.

Collecting NSRL Data

In collaboration with NORC at the University of Chicago, the NSRL gathered data from February 2019 to June 2020 primarily via an online self-administered questionnaire.4 We did not have the resources to do extensive follow-ups with all initial nonrespondents, so we prioritized maximizing the response rate from primary leaders. Secondary leaders who did not initially respond received only emailed reminders, and they received those emailed reminders only if we had names and direct email addresses for them. Primary leaders who did not initially respond were more intensively recruited. They were mailed a paper questionnaire, called on the telephone, and offered enhanced incentives to participate. These efforts produced a much higher cooperation rate among primary leaders than among secondary leaders.

The cooperation rates were 70% among primary leaders and 23% among secondary leaders, for an overall cooperation rate of 37%. Taking into account the NCS-IV's own 69% response rate, the NSRL's response rate is approximately 50% for primary leaders and 17% for secondary leaders.5 Eighty-two percent of respondents completed the survey online in a self-administered way, 7% completed the online survey via an interview, and 11% completed and returned a paper questionnaire. The median completion time for the online instrument was 34 min. The final NSRL dataset has 1,600 cases, 890 of whom are primary leaders of their congregations. There is at least one leader in the final NSRL sample from 968 of the 1,234 NCS-IV congregations with religious leaders (78%).

Nonresponse Bias and Sample Weights

Response rates are only a proxy for what we really care about: nonresponse bias. Looking at the NSRL sample as a whole, the very different response rates for primary and secondary leaders mean that secondary leaders are substantially under-represented relative to primary leaders. Conceptualizing the full NSRL sample as two separate samples––one of the primary leaders and one of the secondary leaders––we assessed nonresponse bias within each of those samples by comparing the characteristics of congregations from whom we had an NSRL respondent to all relevant NCS-IV congregations. We discerned essentially no nonresponse bias in either the primary- or secondary-leader samples with respect to size, geography, or political ideology.

There was some nonresponse bias with respect to religious tradition in both the primary- and secondary-leader samples. This bias was very slight in the primary-leader sample and somewhat larger in the secondary-leader sample, where secondary leaders from predominantly white mainline Protestant congregations were somewhat over-represented, and secondary leaders from both predominantly white conservative/evangelical congregations and Black Protestant congregations were somewhat under-represented.6

In light of all this, we developed weights that allow analysts to conceptualize and use the NSRL as three distinct samples. It can be conceptualized as (1) a sample only of primary congregational leaders; (2) a sample only of secondary congregational leaders, or (3) a sample of all congregational religious leaders (including both primary and secondary leaders). Analysts should use the NSRL weight appropriate for each conceptualization. Moreover, the NSRL, like the NCS, is a probability-proportional-to-size sample. Leaders of larger congregations are more likely than leaders of smaller congregations to be in the sample. Consequently, in addition to weighting the data to focus on primary leaders, secondary leaders, or all leaders, analysts can weight the data so that results represent leaders without respect to the size of their congregations, or they can weight the data so that results represent leaders in proportion to the number of people in their congregations.7

Although the NSRL can be used to examine congregations’ primary leaders, secondary leaders, or all leaders, we believe that most analysts, for most purposes, will want to focus on primary leaders. The population of primary leaders is more clearly defined, a better response rate and minimal nonresponse bias in the primary-leader sample make it higher quality than either the sample of secondary leaders or the combined sample of all leaders, and many substantive questions will pertain most directly to the population of congregations’ primary leaders. That means that, although the NSRL offers analysts several options, most will want to focus on primary leaders, weighting the data so that results represent leaders without respect to their congregation's size.8

We describe additional complexities related to deciding which subset of religious leaders is analytically relevant in the “Lessons Learned” section, where we also offer another way to conceptualize a meaningful religious leader population that cuts across the primary- and secondary-leader distinction.

Lesson 1: Who is a Religious Leader?

As observers of congregations are well aware, there are many forms of congregational leadership, so defining the population from which the NSRL would sample was conceptually and empirically challenging, especially since we did not want to limit the sample only to a congregation's primary leader. At the broadest level, we wanted a sample of religious leaders, meaning those doing a congregation's core religious work of preaching, teaching, leading collective worship services and other rituals, and engaging in pastoral care. Leaders of NCS-IV congregations constitute a nationally representative sample of congregational leaders, but deciding exactly which leaders should be included in the NSRL was far from trivial. Indeed, operationalizing “religious leader” was the single biggest challenge we faced.

Several overlapping issues produced this complexity. Should only paid leaders be included? Only leaders who are ordained clergy? How should coleaders be handled? Is a “campus pastor” within a multisite congregation the primary leader of that campus? Who is the primary leader of a small Catholic parish in which a full-time deacon is responsible for the day-to-day religious and administrative work, and which is visited only on Sundays by a priest who says mass? In congregations with multiple staff members, who among the paid staff and volunteer leaders should be included?

We realized early on that the conceptual and practical issues would be different for primary and secondary leaders. For primary leaders, each congregation's primary leader was included in the NSRL sample whether or not they were paid and whether or not they were ordained clergy. Requiring that primary leaders be paid or ordained would have excluded too many congregational leaders and, of course, in a structured rather than random way. Identifying the primary leader usually was straightforward since the vast majority of congregations (94%) have a single person who clearly is the primary religious leader. Most often, this person had a title, such as Pastor, Senior Pastor, Senior Rabbi, Imam, Bishop, or the like.

Still, three kinds of primary-leader situations vexed us. First, we had to decide how to handle coleaders, when no single person is clearly in charge. Did we really want some congregations to be represented by more than one primary leader? If not, how do we choose which of the coleaders to include? In the end, since the NSRL was intended as a sample of leaders rather than congregations, all coleaders were included as primary leaders, a decision that also complicated the development of our sample weights.9 Second, in multisite congregations, we had to decide whether to designate as the primary leader a campus pastor or the senior pastor of the multisite congregation as a whole. Our decisions here were made on a case-by-case basis, hinging on whether a particular campus or the congregation as a whole participated in the NCS, and on how congregational websites and key informants described the leadership structure. Third, in Catholic parishes in which a priest might be present only on the weekend to say mass, while a nonpriest carried the bulk of the pastoral duties, we chose between the priest and the nonpriest as the primary leader depending on how parish websites and key informants described the leadership situation and, in some cases, on how respondents described themselves and their work on the NSRL questionnaire itself.10 None of these situations occurred frequently, but the time we spent grappling with them was far out of proportion to the frequency of their occurrence.

Determining which nonprimary leaders were in-scope for the NSRL was far more challenging. We know, of course, that volunteers do much valuable work in congregations and often are religious leaders in their own right, but including all volunteer leaders was not feasible, and carving out a subset of volunteers who should be included was intractable. This led us to limit the NSRL secondary leader sample to people who were paid for their work in the congregation.

Among paid secondary leaders, identifying those doing religious work was difficult in practice. Paid staff who only did administrative or custodial work were excluded, as were people whose only responsibilities were performing music or managing technology. Sometimes job titles clearly signaled someone who should be included, such as Associate Pastor, Parochial Vicar, Associate Rabbi, or Director of Faith Formation. But not always. Should Executive Pastors be included? What about Worship Leaders or Pastoral Assistants? Or “coordinators” of things like volunteers or family ministries? In these cases, we used congregational websites to try to assess the extent to which the person's responsibilities went beyond administration or music performance to include substantive religious work like preaching, leading worship services beyond musical segments, religious education, counseling, or visiting the sick. Executive Pastors were mainly excluded, as were Worship Leaders in large evangelical congregations. Coordinators, Directors, and Assistants within specific ministry areas were assessed on a case-by-case basis, with an eye to whether their responsibilities went beyond administration and logistics to include substantive religious work. Deacons were included in Catholic parishes (if they were paid), but not in Protestant churches. People with Administrator titles mainly were excluded, except in Catholic parishes where it appeared that an Administrator or Parish Administrator, often a priest, was the congregation's primary leader and engaged in activities typical of a parish priest.

In whatever manner we operationalized “religious leaders,” we had to face the practical issue of identifying such people in NCS-IV congregations. How would we know who was doing what work in each congregation? We employed a three-stage strategy. The first stage occurred after a GSS respondent named a congregation, making it eligible to participate in the NCS, but before a key informant from that congregation was interviewed for the NCS. In this stage, we examined the congregation's website and social media pages, and GSS respondents’ reports of their congregation's leaders’ names, to identify a congregation's leaders. The second stage occurred as part of the NCS interview. NCS key informants were asked to confirm, correct, and supplement the leader information we found before the NCS interview. When we were not able to find any leader information before the NCS interview, NCS key informants were asked to provide that information. The goal of these first two stages was to generate a roster of religious leaders of NCS-IV congregations who would be invited to participate in the NSRL.

If, after these two stages, we still were not sure whether an individual who worked at a congregation met our definition of being a religious leader, we erred on the side of inviting them to participate in the NSRL. Consequently, the third stage in the process of identifying congregations’ religious leaders used information gathered in the NSRL questionnaire itself to refine the lists of leaders generated in the first two stages. That is, we used responses to questions about the respondent's title, responsibilities at the congregation, whether or not they considered themselves to be a congregation's primary leader, and whether or not they were paid, to confirm or adjust respondents’ classifications as primary or nonprimary leaders, or to remove them from the sample if they appeared not to be religious leaders as defined by the NSRL. Specifically, we removed from the final sample respondents who (1) were not primary leaders and who reported being unpaid; (2) reported doing only work pertaining to administration, communications, technology, or facilities upkeep; or (3) reported they never preached, taught, counseled, or visited people in a pastoral capacity.

We did our best to work out our definitions and operationalizations during pretesting, but new job titles and situations constantly came up throughout the entire field period, each of which required our attention. Moreover, sometimes NSRL respondents’ self-reports about their job titles and responsibilities contradicted what we had gleaned from websites and/or NCS key informants. In such cases, we used a holistic approach rather than a firm rule to decide whether the respondent was legitimately in-scope for the NSRL and, if in-scope, whether the person should be considered a primary or secondary leader. At the end of the day, the vast majority of people we identified as the congregation's head clergyperson using online information and NCS interviews also indicated on their NSRL questionnaire that they were, in fact, the primary leader of their congregation. And only about 100 of the approximately 1,700 initial NSRL respondents were excluded from the final sample because of their self-reported job characteristics. Both of these facts increased our confidence that our initial assessments of who was doing religious work were mainly on target, but they did not remove the need for ongoing fine-tuning of our sample.

Another complexity, this one regarding secondary leaders at very large congregations, led to our most important mid-course correction. Websites of large congregations often do not list all staff by name, and it was not practical to ask NCS key informants from such congregations to name every one of what might be 25 or more ministerial staff members. Although we initially invited all of these people to participate in the NSRL by sending a package of questionnaires to the NCS key informant and asking that person to distribute them as appropriate, we soon realized that this impersonal approach, coupled with our inability to send reminders directly to people whose names we did not know and whose direct contact information we did not have, would produce a very low cooperation rate. Rather than accept a very low cooperation rate among secondary ministerial staff at congregations with very large staff, we redefined the NSRL secondary leader population to exclude secondary leaders at congregations with more than 25 secondary ministerial staff members. We noted above that this exclusion does not significantly limit the NSRL's generalizability, but needing to adjust the definition of our religious leader population in the middle of data collection was an important lesson.

Stepping back from the nitty-gritty details of the NSRL, we can offer several observations related to who counts as a religious leader that may be relevant for other researchers. First, the very concept of a religious leader in a congregation is much fuzzier than we anticipated. Moreover, the nature of religious leadership varies importantly across religious groups, congregation size, and other contexts. There were ways in which our research design clashed with how religious leadership is socially organized. We all should reflect constantly on how our definitions of “religious leader” might be warping our view and limiting our understanding of congregational life and leadership. To give just one example, our decision to limit the secondary leader sample to paid staff means that secondary leaders in congregations relying more on unpaid leadership beyond the primary leader––a situation more common in predominantly Black than in predominantly white congregations––are not part of the sample.

Second, unlike textbook presentations of the research process that represent an orderly sampling process that starts with defining the population, proceeds to sampling from that population in a systematic way, and only then gathers data from sampled units, we had neither a final definition of our population nor a final sample until after data collection was complete. As we noted, the practical difficulties of reaching secondary religious staff in congregations with very large staffs led us to redefine our population to exclude those secondary leaders, and we also used information gathered on the NSRL questionnaire itself to make final decisions about who was included or out-of-scope. Additionally, our research team spent many hours reading congregational websites and denominational resources and debating the merits of including or excluding various types of leaders. These investigations and conversations began with NCS-IV pretesting in 2017 and continued well into the cleaning of the completed NSRL dataset in 2021. We suspect that this kind of iterative process in which population definitions and criteria for sample inclusion shift over the course of a project is not uncommon in research of various sorts.

Third, perhaps the biggest practical lesson we learned from this experience was that we encountered orders of magnitude more gray areas, fuzziness, and practical difficulties regarding secondary leaders than primary leaders. All things considered, we are not sure that the substantive value of including secondary leaders in the study was worth the hundreds of hours we spent grappling with edge cases. In analyzing the data to address substantive research questions about subjects, such as clergy political involvement, clergy health and well-being, clergy religious beliefs and practices, and clergy attitudes and practices concerning science, the causes of mental illness, and other things, we find ourselves focusing almost entirely on just the primary leaders. This is mainly because primary leaders are a more clearly defined type of religious leader, and there is so much heterogeneity among secondary leaders that analyzing them together with primary leaders often seems to muddy the water. There may well be substantive subjects for which analyzing just the secondary leaders, or the full sample of primary and secondary leaders, would be appropriate. Time will tell. Until the value of that part of the sample is proven by researchers using the secondary leader sample to advance knowledge in important ways, we would recommend that future national surveys of clergy use available resources to maximize the quality of a primary leader sample rather than expand the sample to include secondary leaders.

Reflecting further on the heterogeneity of the secondary leader sample led us to construct a variable that we did not anticipate constructing. Throughout the course of the project, we were focused on the distinction between two types of religious leaders––primary and secondary. While cleaning the data, however, it occurred to us that some secondary leaders are a lot like primary leaders in that they mainly have the same education and training, engage in the same range of religious work (preaching, teaching, and pastoral care), and often will be primary leaders of a congregation later in their careers. This recognition led us to construct an indicator that included all primary leaders (with one type of exception)11 plus this specific type of secondary leader.

This indicator, then, identifies a subset of religious leaders that includes primary leaders and a generalist subset of secondary leaders who are much like primary leaders and likely will be primary leaders one day. This seems a meaningful category of religious leaders that, for some substantive purposes, might be an appropriate analytical sample. Our early data analyses suggest that substantive results on various subjects are qualitatively the same whether this analytical sample or just the primary-leader sample is used, but the sample size increases from 890 to 1,102 leaders, increasing statistical power. We mention this indicator here because constructing it was a direct result of encountering and grappling with the challenges of defining religious leadership raised in this project, and reflecting on what those challenges might imply for improving the state of the art of research on this subject.12

Lesson 2: Deductive Identification and Linking to the NCS-IV Data

Because all NSRL respondents served congregations who participated in the NCS-IV, and, therefore, NCS-IV data can be linked to NSRL data, we did not have to use valuable NSRL survey time to ask questions about a respondent's congregation's characteristics. Earlier clergy surveys, including Carroll et al. (2001) and Woolever et al. (2009a), did not include a separate extensive effort to gather data about congregations as organizational entities, so the information they gathered about congregations was limited to what was asked as part of the clergy surveys themselves. This meant that information about leaders’ congregations was mainly limited to religious tradition, size, and a few other things. Being able to link to the NCS-IV data thus was a distinctive strength of the NSRL. It meant we could ask leaders more about themselves, while still being able to link them to the wealth of data about their congregations available in the NCS.

When this project began, we assumed that we would publicly release the linked NSRL-NCS-IV dataset. However, in part because of prodding from Duke University's Institutional Review Board (IRB), we had to confront the possibility that this linked dataset would greatly enhance the risk of deductive identification of NSRL respondents. We assessed this by asking research team members to assume the role of a nefarious and skilled member of the public to see if, armed with a linked NSRL-NCS dataset, such a person would be able to identify any NSRL respondents through the creative use of the dataset combined with internet search engines. The answer was yes, leading us to conclude that publicly releasing a linked dataset would not be ethically responsible.

Moreover, since the NCS-IV dataset already was publicly available, and since it contains some information about the congregation's head clergyperson, we had to consider the possibility that this same nefarious and skilled person might be able to figure out how to link NSRL cases to NCS-IV cases even if we released a stand-alone NSRL dataset with only a few congregational variables from the NCS. Our IRB suggested that this possibility meant that including even a few NCS variables in a stand-alone dataset presented a risk of deductive identification.

So, beyond not releasing a dataset that linked the NSRL to the full NCS-IV, we had to figure out how to construct an NSRL dataset that would contain at least basic congregational variables (such as religious tradition and size) that any serious analysis would require while being unlinkable to the already publicly available NCS-IV dataset. To accomplish this, we created NSRL datasets that included various combinations of key NCS variables, with both NSRL and NCS variables coarsened in a number of ways. For each version of the dataset, we tried to link NSRL cases with publicly available NCS cases.

After testing many permutations, we found that no NSRL respondent could be identified from a dataset that included coarsened versions of four NCS variables––religious tradition, region, size, and an indicator of the congregation's predominant racial or ethnic group––along with coarsened versions of the NSRL variables measuring respondent age, year entered ministry, and country of birth. Interestingly, a dataset containing the original detailed versions of these variables would make it possible to link about half the NSRL cases to their congregations in the NCS-IV dataset. After coarsening these variables, an evil, skilled analyst still could produce a few dozen matches between NSRL respondents and NCS congregations, but each of those matches is either incorrect (i.e., it looks like a unique match but the NSRL respondent is in fact matched to the incorrect congregation) or indeterminate (i.e., the correct NCS match can be narrowed only to one of several congregations). Moreover, even in the small number of instances where an analyst could narrow the correct match to a small number of NCS cases, in none of those cases is it possible to combine our data with internet searches to identify the NSRL respondent. This is the NSRL dataset that will be publicly available. The fully linked NSRL-NCS dataset will be available via a sensitive data sharing agreement.

The larger lesson we learned here is that linking datasets at different levels of analysis created risks of deductive identifiability that were more complex than we anticipated. Constructing an ethically responsible public NSRL dataset was made even more difficult by the fact that the NCS-IV dataset already was publicly available. Now that so much personal and organizational information is available online, deductive identification is a greater risk than it was not that long ago, especially for datasets that include both respondent information and information about an organization with which the respondent is associated. We were committed to the public release of an NSRL dataset that included essential congregation-level variables so that the NSRL would be available and useful to everyone, and we eventually found a path to that goal. But researchers aiming to produce publicly available datasets would do well to confront issues like this early in the research process.