3.1 Procedure

Consistent with Hay's (2019) call, we employ best practice techniques to conduct the meta-analysis. The procedures for the literature search and collating, coding and analysing the data follow the guidelines developed by the MAR-Net group (Stanley et al., 2013). Explicit details of the techniques for reporting and summarizing the data and developing the MRA were provided by Stanley and Doucouliagos (2012). We employ these methods and some more recent developments in developing meta-regression models (e.g. Stanley & Doucouliagos, 2014, 2017).

A search of the literature in October 2018 was conducted by a university librarian on subscribed journals and databases available within the university's electronic library resources, covering the databases of Taylor & Francis, ProQuest Business, Wiley Online Library, Scopus, Emerald Insight and Springerlink. The search criteria included material published between 1996 and 2018, and the keywords skepticism/scepticism, professional skepticism/scepticism and audit were used. The researchers also conducted their own literature search, using the same criteria via Google Scholar. The literature synthesis conducted by Hurtt et al. (2013) also served as a useful guide and reference in the search, in addition to the concise summary of studies collated by Robinson et al. (2018) that measured skepticism using the Hurtt (2010) scale.

The first search, in October 2018, focused on articles published only in journals. This was followed by a second search a month later, in November 2018, which focused on working papers and other publications (e.g. peer-reviewed book chapters) found in the Social Science Research Network (SSRN). In shortlisting the articles, care was taken to ensure only experimental studies that measured professional skepticism were included in the meta-analysis. This implies that all editorials, professional commentaries and any publication that was qualitative were excluded. A final replication search was completed by a different university librarian in March 2020. This search followed the same parameters as prior searches and extended it to additional databases, namely, Business Source Complete, ABI, Web of Science and Research Gate.

The search, which included working papers, identified 95 studies that differed in terms of the research methods employed. Figure 1 shows a flow chart of how the studies are classified and selected. For the purpose of the meta-regression, we focus on quantitative research, for which 66 studies are identified. Of these studies, 32 measured and tested the empirical relationship between trait skepticism and state skepticism.

The meta-analysis of measures of trait skepticism includes 35 studies, and the MRA uses the Pearson correlations between trait and state skepticism reported in 14 studies. The authors completed the coding on their own. First, the authors each coded a few papers and then compared notes. Following this, one author completed the remainder of the coding, with the other author conducting regular checks. In the initial coding, certain variables were identified from the outset; however, subsequent observations resulted in additional variables being coded, as shown in Table 1.

3.2 Trait skepticism measures

Table 2 lists all the scales that have been used to date to measure trait skepticism. Trait skepticism has predominantly been measured using the Hurtt (2010) scale and the inverse of Rotter's (1967) Interpersonal Trust scale. The Hurtt scale has represented the neutrality perspective of professional skepticism, whereas the inverse of the Rotter scale has represented the presumptive doubt perspective of professional skepticism. These have been the two predominant perspectives on professional skepticism in the audit literature (Cohen et al., 2017; Quadackers et al., 2014). Some studies have also used self-created measures, but their application and generalizability have not been as widespread as the aforementioned measures captured by Hurtt and Rotter.

Our descriptive summary primarily focuses on the Hurtt (2010) scale to measure professional skepticism as a trait, because this has been the predominant manner in which professional skepticism has been operationalized in the literature. It is also the scale that has guided the discipline in terms of how professional skepticism is measured and operationalized and then subsequently communicated to both researchers and professionals (Cohen et al., 2017). The following section discusses how the Hurtt scale has been employed in professional skepticism research.

3.3 Hurtt trait skepticism measure

As collated in Table 3, the literature search reveals that 35 studies (including seven working papers) have utilized the Hurtt (2010) scale. In the wider sample of studies collected, 12 studies also used alternative measures for professional skepticism (with or without the Hurtt measure), including the inverse of the Rotter (1967) Interpersonal Trust Scale and other self-created measures.

The 30-item Hurtt scale elicits responses on a 6-point agree/disagree scale, with each item added up (after reverse-coding eight items) to calculate a score from 30 to 180. The mean skepticism score across all studies (based on 28 reported means) is 135.2.² This figure is consistent with the means reported by Hurtt (2010) during the development and validation of the scale itself, where the mean skepticism scores for the student and professional samples were 132.7 and 138.6, respectively (Hurtt, 2010, p. 161). On average, the self-reported means are sufficiently high across both student and practitioner samples, and the scale has proven to be reliable, as shown in Table 2, where the highest reported Cronbach α is 0.97, which indicates greater reliability than the other competing measures used in the skepticism literature. It should also be noted that the lowest reported Cronbach α for the Hurtt skepticism measure is 0.68, with an average of approximately 0.85. Overall, since its publication in 2010, the Hurtt measure for trait skepticism has proven to be consistently reliable in measuring trait skepticism, as well as reliably distinctive between individuals with higher/lower levels of trait skepticism. The sample, on average, reports the levels of trait skepticism consistent with expectations in the profession when compared with the initial benchmark scores reported by Hurtt (2010) during the scale's development.

The basis for the discipline favouring the Hurtt measure over other measures of trait skepticism goes beyond its statistical reliability and generalizability. The Hurtt measure appears to reflect the neutrality assumption of professional skepticism (Cohen et al., 2017; Quadackers et al., 2014), an assumption that is consistent with the definition of professional skepticism in the audit profession, which has guided research in operationalizing professional skepticism. Other competing measures have generally been consistent with the presumptive doubt assumption, which is generally incompatible with how the audit profession understands and communicates professional skepticism (International Auditing and Assurance Standards Board [IAASB], 2017; Lee et al., 2013).

The intent of Hurtt's measure of trait skepticism is not to discriminate between individuals with innately higher/lower levels of professional skepticism, because this has limited practical implications, but rather to better understand how their innate skepticism is then subsequently applied to audit judgment and/or audit action as state skepticism. Research has indicated that, regardless of an individual's innate level of skepticism, it is not always applied fully as state skepticism, and in some instances, the contrary has occurred. This means even individuals with higher levels of trait skepticism are not exhibiting high levels of professional skepticism in their audit actions and/or audit judgments. Therefore, it has always been the case that both research and practice are interested in achieving a stronger trait–state skepticism relationship, rather than simply distinguishing between individuals, or even discriminating individuals, on the basis of their trait skepticism (Hammersley, 2011; Hurtt et al., 2013).

With an understanding of how trait skepticism has been operationalized and measured in the literature, we now investigate the trait–state skepticism relationship to determine key drivers and determinants that can provide insight into how trait skepticism is applied as state skepticism. In the following section, an MRA is conducted on correlations reported between trait skepticism and state skepticism.

4.1 Approach and descriptive results

The body of literature investigating the relationship between trait and state skepticism employs a series of methods to analyse relationships, including multiple regression, Pearson correlations and ANOVA-related methods. The number of estimates from multiple regression and ANOVA-related techniques is too small to permit any meaningful quantitative meta-analysis. In contrast, 73 correlation estimates were identified, potentially providing enough information to analyse the relationship between trait and state skepticism across studies. In most cases, the studies employed controlled experimental designs, and simple correlations could thus adequately capture the relationship between trait and state skepticism without contamination by extraneous factors. In cases for which the design was such that other variables impacted the relationship, those impacts are accounted for through the MRA analysing the variation of correlations across studies (Stanley & Doucouliagos, 2012). MRA permits us to explain why the correlations differ among the studies and how this could partly be due to the variables captured by various experimental designs employed.

A total of 15 separate studies reported Pearson correlations between trait and state skepticism. Table 4 presents a descriptive summary of the papers used in the MRA. An initial inspection of the regression results and plots identified an outlier; its removal resulted in 72 separate correlation estimates from 14 studies for subsequent analysis.³ For the data analysis, given the important variability in research design among the studies, we do not employ an average or ‘best correlation estimate’ from each study. Rather, we use all the estimates and explicitly account for multiple estimates from individual studies by using research design variables and cluster error techniques.

To account for the regression error variability among studies and the use of multiple estimates from single studies in the MRA, robust and cluster-robust standard errors are employed (Angrist & Pischke, 2008). Robust standard errors capture variability through the specification of a general form of heteroscedasticity but ignore the correlation among estimates from the same study. In contrast, cluster-robust standard errors recognize both a general form of heteroscedasticity and the correlation from using estimates from the same study. Our sample contains 14 clusters, which could imply significant bias in cluster-robust standard error estimation (Angrist & Pischke, 2008, pp. 233–234). Consequently, we also report on wild bootstrap cluster-robust p-values, which have been shown to perform well with small cluster sizes (Cameron et al., 2008).

The use of multiple estimates from single studies and the recognition of subsequent error correlation among errors from the same study through cluster robust estimation techniques is now common practice in MRA studies. For example, Oczkowski and Doucouliagos (2015) examine the relation between wine prices and quality ratings using 184 estimates from 36 studies. Chaikumbung et al. (2016) use 1432 estimates from 379 studies to evaluate the economic value of wetlands, whereas Hirsch (2018) examines the persistence of firm profits using 413 estimates from 36 studies.

Figure 2 presents a funnel plot of the correlations, illustrating their relation to precision. Precision is measured as the inverse of the associated standard error (Stanley & Doucouliagos, 2012, p. 40). A well-developed funnel plot typically has an inverted V shape, where the estimates of the effect converge to a value with a similarly high degree of precision (Stanley & Doucouliagos, 2012, pp. 53–60). This is not obvious in our case, in which high levels of precision are associated with a variety of correlation estimates. However, consistent with the inverted V shape, the variation in correlation estimates is greater for low levels of precision. Approximately 24% (17) of the correlations are negative and 76% (55) positive. At the 5% level of significance, 6% (one of 17) of the negative correlations and 44% (24 of 55) of the positive correlations are statistically significant. The unweighted average correlation is 0.139, whereas the precision-weighted average correlation is 0.106, which is regarded as a small correlation, according to Cohen's (1988) guidelines. We can tentatively conclude, on average, that there is only a weak empirical association between trait and state skepticism.

It is potentially important to recognize that measurement scales are employed to measure the key constructs of interest. The measurement error associated with the use of multi-item scales implies that the correlations suffer from attenuation bias (Hunter & Schmidt, 2004). Sufficient information has been provided in studies to correct for this bias, using Cronbach's α coefficients for trait skepticism only.⁴ The average of the corrected correlation estimates in unweighted form is 0.155, and the precision-weighted average is 0.118. The differences between the reported and attenuated bias-corrected averages are less than 0.02, which appears to be practically unimportant.

An important requirement of meta-analysis estimates is they be comparable. We assess data comparability by determining whether the quality of the article unduly influences the results. We test whether the correlation estimates and their precision differ according to the quality of the article. Three measures of article quality are considered: (1) the Australian Business Deans Council (ABDC, at abdc.edu.au) journal quality ratings, (2) the number of Google Scholar citations and (3) the number of Google Scholar citations per year (measured from the publication year). Given that working papers are included in the data set, the measures based on journal quality ratings will not adequately reflect the quality of all articles. Additionally, any journal quality measure does not necessarily reflect the quality of any particular article in the journal. In contrast, the use of individual article citations could be considered a ‘revealed quality preference’ approach to measuring study comparability (Stanley & Doucouliagos, 2012, p. 34). The use of the total number of citations compared with the number of citations per year favours older papers, which might not necessarily be superior in quality to more recent papers. Table 5 presents estimates for the correlations and their precision and their relation to article quality measures, using various estimators that recognize error variability and correlation.

Table 5 suggests that the various measures of article quality do not significantly impact correlations, but they do impact precision. For precision, all three measures suggest the better the article quality, the weaker the precision of the estimates. This result is contrary to expectations, where better quality papers will tend to report more precise estimates. The reasons for this finding are not obvious but could partly be due to the absence of other important variables in these regressions. Of the three quality measures, the number of citations per year has the best explanatory power for precision, as indicated by higher R² values and a more statistically significant estimate.

The results in Table 5 suggest that estimates in the meta-analysis dataset might not be comparable and may depend on article quality. One approach to mitigating any consequences of the non-comparability of MRA results is to include a measure of article quality as a control variable in the meta-regressions (Stanley & Doucouliagos, 2012, p. 35). Given our results, the number of citations per year will be included in subsequent analyses for the MRA.

4.2 Publication selection bias

It is important to examine whether publication selection bias influences the study results. Generally, publication selection bias is the notion that only ‘significant results’ are published. Because of the vagaries of publishing research, statistically non-significant results could be underreported in a body of literature (Roberts & Stanley, 2005). Stanley (2005, 2008) advocates a funnel asymmetry precision effect size test (FAT–PET) for publication bias. The FAT–PET involves regressing the correlations against a constant and the correlation's standard error. If there is no publication selection bias, then estimated correlations will not be correlated with their standard errors (Egger et al., 1997). In other words, publication bias exists if there is a significant correlation between the correlations and their standard errors, due to the practice of publishing significant estimates only.

Formally the FAT–PET model can be written as follows:

(1)

where r denotes the correlation between trait and state skepticism, se denotes the correlation's standard error, and ԑ is an error term. Equation 1 is estimated by weighted least squares (WLS, with w = 1/se²) to capture error variance differences in the precision of the correlation estimates across studies. This weighting has been shown to result in optimal weights (Hedges & Olkin, 1985). Stanley and Doucouliagos (2017) also illustrate how, for meta-regression, WLS is preferred to other techniques, such as using fixed effects or random effects estimators to estimate models. A test of β₀ = 0 (PET) is a test of the existence of an underlying effect after correcting for publication selection, whereas a test of β₁ = 0 (FAT) is a test of the existence of publication selection bias.

The FAT–PET results are presented in Table 6. Column (1) employs the reported correlations. The results of a series of robustness checks are also presented: column (2) uses Fischer's z-transformed correlations.⁵ Column (3) employs the trait skepticism attenuated bias–corrected correlations. Column (4) reports on the precision effect estimate with standard error (PEESE), where se² is used as a regressor, rather than se (Stanley & Doucouliagos, 2014). The PEESE could more accurately identify selection bias than using the standard error. In all cases, when using estimation procedures that recognize error correlation due to clustering, β₀ = 0 cannot be rejected, but β₁ = 0 can be rejected at the 5% level of significance. This result implies significant publication selection bias and no statistically significant effect size. In other words, the correlation between trait and state skepticism is not significantly different from zero, after the publication selection bias is recognized. The estimates in Table 6 (columns (1) to (3)) suggest a correlation effect size of approximately 0.03, which is very small.

4.3 Meta-analysis regression of trait and state skepticism correlations

We employ MRA to explain the variation in the correlations of trait and state skepticism across studies. Following Stanley and Doucouliagos (2012), we generalize Equation 1 and employ the following regression model:

(2)

where z_k represents regressors that capture the variation in studies due to differences in experimental design, article quality, or other extraneous factors. Effectively, the correlation effect is now estimated byβ₀+Σβ_kz_ki. Equation 2 should be estimated by WLS (using w = 1/se²) and standard error estimators that account for heteroscedasticity and correlation within studies.

The regressors proposed for Equation 2 and their summary statistics are outlined in Table 7. Three variables are employed to capture differences in research design across studies: (1) uncontrolled (vs. controlled) experiment setting, (2) student (vs. practitioner) participants and (3) low-risk (vs. high-risk or neutral-risk) environment.⁶ For our meta-analysis dataset, 47% of the estimates are coded as conducting the experiment in an uncontrolled environment, and 53% as conducting the experiment in a controlled environment. Student participants were employed in 22% of the cases and practitioners of various types in 78% of cases. The studies differed in the type of risk environment employed by the research design, with 17% having a low-risk environment, 44% a high-risk one and 39% a neutral-risk environment. Two variables are employed to capture differences in the measures used by the studies. For trait skepticism, 25% of the estimates used Rotter's scale, 64% employed Hurtt's measure and 11% other measures. For state skepticism, 49% of the estimates used judgment scenarios to capture this variable, whereas 51% of the estimates used action scenarios. The numbers of citations per year are also included in the MRA, with the results in Table 5, as is the possibility of article quality affecting the precision of the correlation estimates.

The MRA estimates for Equation 2 using the regressors in Table 7 are presented in Table 8. We also considered fixed effects due to individual studies. Only the study by Cohen et al. (2017) impacted the results, and interactions between the other variables and studies also proved to be unimportant in terms of their impact on the estimates.⁷ Consequently, Table 8 also includes the impact of Cohen et al. (2017) on the correlation estimates. The standard WLS model results with various standard error estimates are presented in column (1). The results of four robustness checks are provided in columns (2) to (5). Column (2) uses Fischer's z correlation transformation. Column (3) corrects the correlations and their standard errors for trait skepticism measurement error. Column (4) replaces the standard error by its square, according to the PEESE procedure. Column (5) presents robust regression estimates (Rousseeuw & Leroy, 1987) that account for the impact of extreme observations.

For the standard WLS model, the constant estimate in the MRA is 0.025, which indicates a correlation close to zero.⁸ The standard error and standard error squared estimates all indicate that there is no significant publication selection bias, after including the additional research design variables. This finding suggests the significant publication bias identified in Table 6 is not actually publication bias but, rather, could be due to heterogeneity in research design features across studies. Across most model types, the number of citations per year appears to be both statistically and practically unimportant in terms of their impact on the estimates. This implies that the impact of the article quality on the precision of the correlation estimates presented in Table 5 could also be due to the variation in research design and not to article quality per se.

In terms of statistical significance, the importance of variables appears to differ depending on which standard errors are employed. The use of robust standard errors suggests that uncontrolled experiments, student participants, the Rotter trait measure, the judgment state measure and the study by Cohen et al. (2017) have significant effects on the majority of alternative models. When cluster standard errors are used, statistically significant variables do not arise for the majority of models, except for the variables for student participants, the judgment state measure and the study by Cohen et al. (2017). When using the wild bootstrap clustered standard errors, only the low-risk environment and the study by Cohen et al. (2017) are statistically important. In part, the absence of statistical significance for most of the variables and the variability in results across the use of standard errors could be due to the relatively small sample size used for the MRA.

The values of the estimated coefficients are reasonably consistent across all model types and point to potentially important results. The study by Cohen et al. (2017) has the greatest impact reducing correlations by approximately 0.16, on average, across all five specifications. The use of a judgment state measure reduces the correlation between trait and state skepticism by approximately 0.13. The next largest impact occurs for uncontrolled experiments, whose use increases the correlation by approximately 0.12. The other impacts are less than 0.1 in absolute value and hence less important. The use of a low-risk environment and Rotter's trait measure in a research design both reduce the correlation by approximately 0.08, whereas the use of student participants increases it by approximately 0.08.

4.4 Discussion

The MRA of published correlation estimates in the experimental audit literature found, at best, that there was a weak positive correlation between trait and state skepticism. The unweighted average correlation is 0.14 among studies, whereas the precision weighted correlation is 0.11. Importantly. however, this association did not hold once publication bias in studies was recognized, with the correlation falling to a statistically insignificant r = 0.03, indicating no discernable correlation. In other words, there is no statistically significant or practically important relation between trait and state skepticism once the publication bias in studies has been recognized.

The findings of the meta-regression analysis identified some key drivers and determinants of the trait–state skepticism relationship that were otherwise not clearly identified (Hammersley, 2011; Nolder & Kadous, 2018) in the audit literature. The MRA results call into the question of the efficacy of capturing trait skepticism as a proxy for professional skepticism and how it is then applied in audit settings as state skepticism. Therefore, the knowledge of the key drivers and determinants of the trait–state skepticism relationship is necessary, in order for the application of trait skepticism as state skepticism to be effective in meeting audit quality standards and expectations. To address these issues and claims, this study documented the trait–state skepticism relationship in the experiment-based literature to calculate the overall average correlation in the literature to date, as well as identify this effect's key drivers. This process also involved documenting all the scales and measures used in the literature to capture trait skepticism as a construct.

The meta-analysis and meta-regression also provide some reaffirmation that the Hurtt (2010) measure is an appropriate operationalization for professional skepticism as a trait, given that it improved the strength of the trait–state skepticism relationship as opposed to alternative measures such as the inverse of Rotter's (1967) Interpersonal Trust Scale.⁹ However, even with the Hurtt (2010) measure, the weak relationship between trait skepticism and state skepticism is driven by contextual factors, as revealed by the MRA, which systematically impact the trait–state skepticism relationship.

One of the main findings of the MRA indicate how the student sample proves to be more effective than the practitioner sample in demonstrating a positive trait–state skepticism relationship. This finding supports the observation that student/graduate auditors tend to apply professional skepticism in a manner that is unhindered by professional experience or pressures (Farag & Elias, 2012; Farag & Elias, 2016; Peytcheva, 2014; Popova, 2012)¹⁰ This finding is useful, because it is important for the profession to not only screen new graduates with regard to qualities underlying appropriate professional skepticism but also investigate whether the profession has succeeded in providing an environment that can encourage and nurture professional skepticism.

Another finding in the meta-regression suggests audit environments where risk is high tend to induce higher levels of professional skepticism and hence strengthen the positive association in the trait–state skepticism relationship. However, in the absence of such obvious risk factors, auditors do not appear inclined to exercise appropriate professional skepticism, which weakens the positive association in the trait–state skepticism relationship (Glover & Prawitt, 2014; Payne & Ramsay, 2005). This finding is important, because the general expectation of the profession is for auditors to exercise appropriate skepticism at all times, even in the absence of obvious red flags in the audit environment (International Accounting Standards Board [IAASB], 2018). This expectation for professional skepticism also aligns with the neutrality assumption of professional skepticism (Cohen et al., 2017; Hurtt, 2010; Quadackers et al., 2014).

The meta-regression analysis also provides insight on additional variables, namely, the experiment setting (controlled vs. uncontrolled) and the manner in which state skepticism has been operationalized (i.e. as either an audit action or audit judgment), but the implications of these findings require further investigation. Regarding experiment setting, uncontrolled experiment settings appear to be capturing a stronger trait–state skepticism relationship, and the reasons for this are difficult to understand, but it is possible that uncontrolled settings allow participants to respond more freely without pressure of social desirability, given that there is far more anonymity guaranteed than in controlled settings (Dandurand et al., 2008). Regarding the operationalization of state skepticism, the MRA findings indicate that the trait–state skepticism relationship was stronger in actions than in judgments. This may be due to the fact that auditors are more likely to exercise professional skepticism when conducting actions, whereas the pressure of making a judgment often means that there are potentially other influences, such as superior/managerial preferences, which may reduce the extent to which they apply their skepticism to judgments (Carpenter & Reimers, 2013; Nelson, 2009; Noviyanti & Winata, 2015). However, this observation needs to be made carefully, as the literature moves towards better understanding how audit actions and audit judgments interact in complex audit work environments (Kadous & Zhou, 2019; Nolder & Kadous, 2018).