Forming Composites of Cognitive Ability and Alternative Measures to Predict Job Performance and Reduce Adverse Impact: Corrected Estimates and Realistic Expectations
Corresponding Author
Denise Potosky
Pennsylvania State University
*Address for correspondence: Denise Potosky, Great Valley School of Graduate Professional Studies, Pennsylvania State University, 30 E. Swedesford Rd., Malvern, PA 19355.Search for more papers by this authorCorresponding Author
Denise Potosky
Pennsylvania State University
*Address for correspondence: Denise Potosky, Great Valley School of Graduate Professional Studies, Pennsylvania State University, 30 E. Swedesford Rd., Malvern, PA 19355.Search for more papers by this authorAbstract
Although there has been empirical attention paid to the criterion-related validity of predictor composites, there has been much less attention paid to the standardized ethnic group differences associated with these composites. One important area of inquiry in predictor composite research is the influence of adding predictors to a test of general mental ability. The limited empirical literature on this practice is mixed, but the prevailing expectation is that there is likely to be higher validity and less adverse impact. Unfortunately, much of the previous work is limited by the presence of inaccurate validity and standardized ethnic group difference values. In this analysis we formed meta-analytic matrices to more accurately estimate the validity and standardized ethnic group differences of several composites that combine a measure of cognitive ability with measures of conscientiousness, a structured interview, or biodata. While results were somewhat complex, we found that adding alternative predictors does not result in a situation in which validity automatically goes up and adverse impact potential automatically goes down. In fact, the reductions in adverse impact (if any) from adding “non-cognitive” predictors were more modest than much of the literature suggests.
References
- Ackerman, P.L. and Heggestad, E.D. (1997) Intelligence, personality, and interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219–236.
- Barrett, G.V., Phillips, J.S. and Alexander, R.A. (1981) Concurrent and predictive validity designs: A critical reanalysis. Journal of Applied Psychology, 66, 1–6.
- Bernardin, H.J. (1984) An analysis of black-white differences in job performance. Paper presented at the Academy of Management meetings, Boston, MA.
- Bobko, P. and Roth, P. (2004) The four-fifths rule for assessing adverse impact: An arithmetic, intuitive, and logical analysis of the rule and implications for future research. In J. Martocchio (Ed.), Research in personnel and human resources management (Vol. 23, pp. 177–197). Amsterdam: Elsevier Press.
- Bobko, P., Roth, P.L. and Bobko, C. (2001) Correcting the effect size of d for range restriction and unreliability. Organizational Research Methods, 4, 46–61.
- Bobko, P., Roth, P.L. and Potosky, D. (1999) Derivation and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors and job performance. Personnel Psychology, 52, 561–589.
- Bureau of Labor Statistics. (2000) Employment and earnings, 46 (1), 13–14 (Chart A-4).
- Carlson, K.D, Scullen, S.E., Schmidt, F.L., Rothstein, H. and Erwin, F. (1999) Generalizabile biographical data validity can be achieved without multi-organizational development and keying. Personnel Psychology, 52, 731–753.
- Cascio, W.F. and Phillips, N.F. (1979) Performance testing: A rose among thorns. Personnel Psychology, 32, 751–766.
- Cortina, J.M., Goldstein, N.B., Payne, S.C., Davison, H.K. and Gilliland, S.W. (2000) The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology, 53, 325–351.
- Dean, M. (1999) On biodata construct validity, criterion validity and adverse impact. Unpublished doctoral dissertation, Louisiana State University.
- Douglas, E.F., McDaniel, M.A. and Snell, A.F. (1996) The validity of non-cognitive measures decays when applicants fake. Paper presented at the 11th Annual Conference of the Academy of Management, Cincinnati.
- Ellingson, J.E., Sackett, P.R. and Hough, L.M. (1999) Social desirability corrections in personality measurement: Issues of applicant comparison and construct validity. Journal of Applied Psychology, 84, 155–166.
- Ellingson, J.E., Smith, D.B. and Sackett, P.R. (2001) Investigating the influence of social desirability on personality factor structure. Journal of Applied Psychology, 86, 122–133.
- Gandy, J.A., Dye, D.A. and MacLane, C.N. (1994) Federal government selection: The individual achievement record. In G.S. Stokes, M.D. Mumford and W.A. Owens (Eds.), Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction (pp. 275–310). Palo Alto, CA: CPP Books.
- Graham, K.E., McDaniel, M.A., Douglas, E.F. and Snell, A.F. (2002) Biodata validity decay and score inflation with faking: Do item attributes explain variance across items? Journal of Business and Psychology, 16, 573–592.
- Gully, S.M., Payne, S.C., Kiechel, K.L. and Whiteman, J.K. (2000). The impact of error training and individual differences on training outcomes: An attribute treatment interaction perspective. Working manuscript.
- Gully, S.M., Phillips, J.M., Beaubien, J.M. and Payne, S.C. (1998, April) Effects of Individual differences in goal orientation and self-regulatory tendencies on learning. In S.M. Gully and J.E. Mathieu (Chairs), Individual differences, learning, motivation, and training outcomes. Symposium presented at the 13th Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
- Hakel, M.D. (1998) Beyond multiple choice: Evaluating alternatives to traditional testing for selection. Mahwah, NJ: Earlbaum.
- Hough, L. (1998) Personality at work: Issues and evidence. In M. Hakel (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection (pp. 131–159). Mahwah, NJ: Erlbaum.
- Hough, L., Oswald, F. and Ployhart, R. (2001) Adverse impact and group differences in constructs, assessment tools, and personnel selection procedures: Issues and lessons learned. International Journal of Selection and Assessment, 9, 152–194.
- Huffcutt, A.I. and Arthur, W. Jr. (1994) Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79, 184–190.
- Huffcutt, A.I., Conway, J., Roth, P.L. and Stone, N. (2001) Identification and meta-analysis of constructs measured in employment interviews. Journal of Applied Psychology, 86, 897–913.
- Huffcutt, A.I. and Roth, P.L. (1998) Racial group differences in interview evaluations. Journal of Applied Psychology, 83, 179–189.
- Huffcutt, A.I., Roth, P.L. and McDaniel, M.A. (1996) A meta-analytic investigation of cognitive ability in employment interview evaluations: Moderating characteristics and implications for incremental validity. Journal of Applied Psychology, 81, 459–473.
- Hunter, J.E. and Hunter, R.F. (1984) Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72–98.
-
Hunter, J.E. and
Schmidt, F.L. (2004) Methods of meta-analysis: Correcting for error and bias in research findings ( 2nd Edn.). Newbury Park: Sage.
10.4135/9781412985031 Google Scholar
- Hunter, J.E., Schmidt, F.L. and Judiesch, M.K. (1990) Individual differences in output variability as a function of job complexity. Journal of Applied Psychology, 75, 28–42.
- Hurtz, G.M. and Donovan, J.J. (2000) Personality and job performance: The big five revisited. Journal of Applied Psychology, 85, 869–879.
- Kehoe, J.F. (2002) General mental ability and selection in private sector organizations: A commentary. Human Performance, 15, 97–106.
- Kriska, S.D. (2001, August) The validity-adverse impact trade-off: Real data and mathematical model estimates. Paper presented at the Society for Industrial and Organizational Psychology meetings, San Diego, CA.
- McHenry, J., Hough, L., Toquam, J., Hanson, M. and Ashworth, S. (1990) Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, 335–354.
- McDaniel, M.A., Whetzel, D.L., Schmidt, F.L. and Maurer, S.D. (1994) The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599–616.
- Mount, M.K. and Barrick, M. (1995) The Big Five personality dimensions: For research and practice in human resources management. Research in Personnel and Human Resources Management, 13, 823–854.
- Ones, D. and Viswesvaran, C. (1998) Gender, age, and race differences on overt integrity tests: Results across four large scale job applicant data sets. Journal of Applied Psychology, 83, 35–42.
- Ones, D.S., Viswesvaran, C. and Reiss, A.D. (1996) Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660–679.
- Ostroff, C. and Harrison, D.A. (1999) Meta-analysis: Level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior. Journal of Applied Psychology, 84, 260–270.
- Phillips, J.M. and Gully, S.M. (1997) Role of goal orientation, ability, need for achievement, and locus of control in the self-efficacy and goal setting process. Journal of Applied Psychology, 82, 792–802.
- Pulakos, E. and Schmitt, N. (1996) An evaluation of two strategies for reducing adverse impact and their effects on criterion related validity. Human Performance, 9, 241–258.
- Reilly, R.R. and Chao, G. (1982) Validity and fairness of some alternative employee selection procedures. Personnel Psychology, 35, 1–62.
-
Reilly, R.R. and
Warech, M.A. (1994) The validity and fairness of alternatives to cognitive ability tests. In
L. Wing and
B. Gifford
(Eds.), Policy issues in employment testing. Boston: Kluwer.
10.1007/978-94-011-2202-3_4 Google Scholar
- Roth, P.L., BeVier, C.A., Bobko, P., Switzer, F.S. III. and Tyler, P. (2001) Ethnic group differences in cognitive ability in employment and education settings: A meta-analysis. Personnel Psychology, 54, 297–330.
- Roth, P.L. and Bobko, P. (1997) A research agenda for multi-attribute utility analysis in human resource management. Human Resources Management Review, 7, 341–368.
- Roth, P.L., Bobko, P., Switzer, F.S. III. and Dean, M.A. (2001) Prior selection causes biased estimates of standardized ethnic group differences: Simulation and analysis. Personnel Psychology, 54, 591–617.
- Roth, P.L., Van Iddekinge, C.H., Huffcutt, A.I., Eidson, C.E. Jr. and Bobko, P. (2002) Correcting for range restriction in structured interview ethnic group differences: The values may be larger than researchers thought. Journal of Applied Psychology, 87, 369–376.
- Rothstein, H., Schmidt, F.L., Erwin, F., Owens, W. and Sparks, C.P. (1990) Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75, 174–184.
- Ryan, A.M., Ployhart, R.E. and Friedel, L.A. (1998) Using personality testing to reduce adverse impact: A cautionary note. Journal of Applied Psychology, 83, 298–307.
- Sackett, P.R. and Ellingson, J. (1997) The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology, 50, 707–721.
- Sackett, P.R. and Roth, L. (1996) Multi-stage selection strategies: A Monte Carlo investigation of effects on performance and minority hiring. Personnel Psychology, 49, 1–18.
- Sackett, P.R., Schmitt, N., Ellingson, J.E. and Kabin, M.E. (2001) High stakes testing in employment, credentialing, and higher education: Prospects in a post affirmative action world. American Psychologist, 56, 302–318.
-
Salgado, J.F.,
Viswesvaran, C. and
Ones, D.S. (2001) Predictors used for personnel selection: An overview of constructs, methods, and techniques. In
N. Anderson,
D. Ones,
H. Sinangil and
C. Viswesvaran
(Eds.), Handbook of industrial, work, & organizational psychology (pp.
165–199). London: Sage.
10.4135/9781848608320.n10 Google Scholar
- Schmidt, F.L. and Hunter, J.E. (1998) The validity of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.
- Schmitt, N., Clause, C.S. and Pulakos, E.D. (1996) Subgroup differences associated with different measures of some common job relevant constructs. In C.L. Cooper and I.T. Robertson (Eds.) International review of industrial and organizational psychology (Vol. 11, pp. 115–137). Chichester, UK: John Wiley.
- Schmitt, N., Rogers, W., Chan, D., Sheppard, L. and Jennings, D. (1997) Adverse impact and predictive efficiency of various predictor combinations. Journal of Applied Psychology, 82, 719–730.
- Stokes, G.S., Mumford, M.D. and Owens, W.A. (1994) Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction. Palo Alto, CA: Consulting Psychologists Press.
- U.S. Equal Opportunity Employment Commission, U.S. Civil Service Commission, U.S. Department of Labor, U.S. Department of Justice. (1978) Uniform guidelines on employee selection procedures. Federal Register, 43, 38295–38309.
-
Viswevaran, C. and
Ones, D.S. (1995) Theory testing: Combining psychometric meta-analysis and structural equations modeling.
Personnel Psychology, 48, 865–886.
10.1111/j.1744-6570.1995.tb01784.x Google Scholar
- Weisner, W. and Cronshaw, S. (1988) The moderating impact of interview format and degree of structure on the validity of the employment interview. Journal of Occupational and Organizational Psychology, 61, 275–290.
- Wonderlic Inc. (2000) Manual for the personal characteristics inventory. Libertyville, IL: Wonderlic.