ORIGINAL ARTICLE
PISA 2022 Creative Thinking Assessment: Opportunities, Challenges, and Cautions
Baptiste Barbot,
James C. Kaufman,
Corresponding Author
Baptiste Barbot
Psychological Sciences Research Institute, UCLouvain (University of Louvain), Ottignies-Louvain-la-Neuve, Belgium
Child Study Center, Yale University, New Haven, USA
Correspondence:
Baptiste Barbot ([email protected])
Search for more papers by this authorJames C. Kaufman
Neag School of Education, University of Connecticut, Storrs, USA
Search for more papers by this authorBaptiste Barbot,
James C. Kaufman,
Corresponding Author
Baptiste Barbot
Psychological Sciences Research Institute, UCLouvain (University of Louvain), Ottignies-Louvain-la-Neuve, Belgium
Child Study Center, Yale University, New Haven, USA
Correspondence:
Baptiste Barbot ([email protected])
Search for more papers by this authorJames C. Kaufman
Neag School of Education, University of Connecticut, Storrs, USA
Search for more papers by this authorBoth authors were members of the creative thinking expert group (CTEG) that contributed to shaping the competency model, developing items, and guiding the PISA 2022 CT Assessment progress from 2018 to 2023. We thank Natalie Foster and Mario Piacentini for their suggestions on a prior version of this manuscript. Baptiste Barbot is supported by the 2025–2027 Francqui Foundation Research Professor Mandate.
ABSTRACT
The OECD's PISA program assesses 15-year-old students globally in key competencies every 3 years, providing influential data on education quality and spurring policy debates. In the latest cycle, the innovation domain focused on creative thinking, assessing over 140,000 students across 60+ countries, in the largest study of adolescent creativity to date. This innovation domain included a cognitive test covering multiple creative thinking processes (generating creative ideas, generating diverse ideas, and evaluating/improving ideas) and creativity domains (written expression, visual expression, social problem-solving, and scientific problem-solving), as well as an extensive survey on factors influencing creativity (such as openness, creative self-efficacy, or growth mindset). While this dataset offers unprecedented research opportunities due to its scale and international scope, challenges arise from its aggregated scoring and complex sampling design. Missteps in using this data in secondary analyses could lead to fragmented and inconsistent findings. This paper provides an overview of the PISA 2022 creative thinking assessment's framework, methods, and findings, highlighting both the potential and the caution needed for impactful creativity research.
Conflicts of Interest
The authors declare no conflicts of interest.
Open Research
Data Availability Statement
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
- Abdulla Alabbasi, A. M., T. L. Thompson, M. A. Runco, L. A. Alansari, and A. E. A. Ayoub. 2022. “Gender Differences in Creative Potential: A Meta-Analysis of Mean Differences and Variability.” Psychology of Aesthetics, Creativity, and the Arts 19, no. 1: 87–100. https://psycnet.apa.org/record/2022-77688-001.
10.1037/aca0000506 Google Scholar
- Amabile, T. M., and M. G. Pratt. 2016. “The Dynamic Componential Model of Creativity and Innovation in Organizations: Making Progress, Making Meaning.” Research in Organizational Behavior 36: 157–183. https://doi.org/10.1016/j.riob.2016.10.001.
- Asquith, S. L., X. Wang, D. S. Quintana, and A. Abraham. 2024. “Predictors of Change in Creative Thinking Abilities in Young People: A Longitudinal Study.” Journal of Creative Behavior 58: 262–278. https://doi.org/10.1002/jocb.647.
- Baer, J., and J. C. Kaufman. 2017. “ The Amusement Park Theoretical Model of Creativity: An Attempt to Bridge the Domain Specificity/Generality Gap.” In Cambridge Handbook of Creativity Across Domains, edited by J. C. Kaufman, V. P. Glaveanu, and J. Baer, 8–17. Cambridge University Press.
10.1017/9781316274385.002 Google Scholar
- Baird, J., T. Isaacs, S. Johnson, et al. 2011. “Policy Effects of PISA.” https://ora.ox.ac.uk/objects/uuid:26c9fccd-ae47-424e-ba40-0c84ebedfc3e/files/m3781c5020f1ce13427a6bed35d8b7c6d.
- Barbot, B. 2018. “The Dynamics of Creative Ideation: Introducing a New Assessment Paradigm.” Frontiers in Psychology 9: 2529.
- Barbot, B., M. Besançon, and T. Lubart. 2016. “The Generality-Specificity of Creativity: Exploring the Structure of Creative Potential With EPoC.” Learning and Individual Differences 52: 178–187.
- Barbot, B., and W. Rogh. 2020. “ Developmental Trends in Creative Abilities and Potentials.” In Encyclopedia of Creativity, edited by S. Pritzker and M. Runco, 3rd ed., 323–326. Academic Press. https://doi.org/10.1016/B978-0-12-809324-5.06167-8.
10.1016/B978-0-12-809324-5.06167-8 Google Scholar
- Barbot, B., and P. P. Tinio. 2015. “Where Is the “g” in Creativity? A Specialization–Differentiation Hypothesis.” Frontiers in Human Neuroscience 8: 1041. https://doi.org/10.3389/fnhum.2014.01041.
- Beghetto, R. A., and J. C. Kaufman. 2014. “Classroom Contexts for Creativity.” High Ability Studies 25, no. 1: 53–69. https://doi.org/10.1080/13598139.2014.905247.
- Beghetto, R. A., J. C. Kaufman, and J. Baer. 2014. Teaching for Creativity in the Common Core Classroom. Teachers College Press.
- Boldt, G. T., and J. C. Kaufman. in press. “Creative Subprocess Frequencies and Their Relation to Personal Characteristics and Product Creativity: Insights From a Drawing Task Think Aloud Study.” Journal of Creative Behavior. https://doi.org/10.1002/jocb.629.
- Breakspear, S. 2012. “The Policy Impact of PISA.” https://www.oecd-ilibrary.org/content/paper/5k9fdfqffr28-en.
- Carson, S. H., J. B. Peterson, and D. M. Higgins. 2005. “Reliability, Validity, and Factor Structure of the Creative Achievement Questionnaire.” Creativity Research Journal 17, no. 1: 37–50. https://doi.org/10.1207/s15326934crj1701_4.
- Chen, Q., R. E. Beaty, and J. Qiu. 2020. “Mapping the Artistic Brain: Common and Distinct Neural Activations Associated With Musical, Drawing, and Literary Creativity.” Human Brain Mapping 41, no. 12: 3403–3419.
- Diedrich, J., E. Jauk, P. J. Silvia, J. M. Gredlein, A. C. Neubauer, and M. Benedek. 2018. “Assessment of Real-Life Creativity: The Inventory of Creative Activities and Achievements (ICAA).” Psychology of Aesthetics, Creativity, and the Arts 12, no. 3: 304–316. https://doi.org/10.1037/aca0000137.
- Eschleman, K. J., J. Madsen, G. Alarcon, and A. Barelka. 2014. “Benefiting From Creative Activity: The Positive Relationships Between Creative Activity, Recovery Experiences, and Performance-Related Outcomes.” Journal of Occupational and Organizational Psychology 87, no. 3: 579–598. https://doi.org/10.1111/joop.12064.
- Gajda, A., M. Karwowski, and R. A. Beghetto. 2017. “Creativity and Academic Achievement: A Meta-Analysis.” Journal of Educational Psychology 109, no. 2: 269–299.
- Glăveanu, V. P., and J. C. Kaufman. 2019. “ Creativity: A Historical Perspective.” In Cambridge Handbook of Creativity, edited by J. C. Kaufman and R. J. Sternberg, 2nd ed., 11–26. Cambridge University Press.
10.1017/9781316979839.003 Google Scholar
- Goldstein, H. 2004. “International Comparisons of Student Attainment: Some Issues Arising From the PISA Study.” Assessment in Education: Principles, Policy & Practice 11, no. 3: 319–330. https://doi.org/10.1080/0969594042000304618.
10.1080/0969594042000304618 Google Scholar
- Han, Z., Q. He, and M. von Davier. 2019. “Predictive Feature Generation and Selection Using Process Data From PISA Interactive Problem-Solving Items: An Application of Random Forests.” Frontiers in Psychology 10: 02461. https://doi.org/10.3389/fpsyg.2019.02461.
- Hopmann, S. T. 2008. “No Child, no School, no State Left Behind: Schooling in the Age of Accountability.” Journal of Curriculum Studies 40, no. 4: 417–456. https://doi.org/10.1080/00220270801989818.
- Kaufman, J. C. 2012. “Counting the Muses: Development of the Kaufman Domains of Creativity Scale (K-DOCS).” Psychology of Aesthetics, Creativity, and the Arts 6, no. 4: 298–308. https://doi.org/10.1037/a0029751.
- Kaufman, J. C., and V. P. Glăveanu. 2019. “ A Review of Creativity Theories: What Questions Are We Trying to Answer?” In Cambridge Handbook of Creativity, edited by J. C. Kaufman and R. J. Sternberg, 2nd ed., 27–43. Cambridge University Press.
10.1017/9781316979839.004 Google Scholar
- Kerr, B. A., M. Birdnow, J. D. Wright, and S. Fiene. 2021. “They Saw It Coming: Rising Trends in Depression, Anxiety, and Suicidality in Creative Students and Potential Impact of the COVID-19 Crisis.” Frontiers in Psychology 12: 611838.
- Komatsu, H., and J. Rappleye. 2017. “A PISA Paradox? An Alternative Theory of Learning as a Possible Solution for Variations in PISA Scores.” Comparative Education Review 61, no. 2: 269–297. https://doi.org/10.1086/690809.
- Liu, H., Y. Liu, and M. Li. 2018. “Analysis of Process Data of PISA 2012 Computer-Based Problem Solving: Application of the Modified Multilevel Mixture IRT Model.” Frontiers in Psychology 9: 01372. https://doi.org/10.3389/fpsyg.2018.01372.
- Lopez-Persem, A., T. Bieth, S. Guiet, M. Ovando-Tellez, and E. Volle. 2022. “Through Thick and Thin: Changes in Creativity During the First Lockdown of the Covid-19 Pandemic.” Frontiers in Psychology 13: 821550.
- Lubart, T., M. Besançon, and B. Barbot. 2011. Evaluation du Potentiel Créatif (EPoC). Editions Hogrefe.
- Lucas, B., G. Claxton, and E. Spencer. 2013. Progression in Student Creativity in School: First Steps Towards New Forms of Formative Assessments (86; OECD Education Working Papers). OECD Publishing. https://doi.org/10.1787/5k4dp59msdwk-en.
- Mercier, M., F. Vinchon, N. Pichot, et al. 2021. “COVID-19: A Boon or a Bane for Creativity?” Frontiers in Psychology 11: 601150.
- Meyer, H.-D., and A. Benavot. 2013. PISA, Power, and Policy: The Emergence of Global Educational Governance. Symposium Books Ltd. https://books.google.com/books?hl=en&lr=&id=fqFwCQAAQBAJ&.
10.15730/books.85 Google Scholar
- OECD. 2009. “PISA Data Analysis Manual: SPSS, Second Edition.” https://www.oecd-ilibrary.org/content/publication/9789264056275-en.
- OECD. 2019. Creative Thinking Framework. Third Draft. OECD. https://www.oecd.org/pisa/publications/PISA-2021-creative-thinking-framework.pdf.
- OECD. 2024a. PISA 2022 Results (Volume III): Creative Minds, Creative Schools. PISA, OECD Publishing. https://doi.org/10.1787/765ee8c2-en.
10.1787/765ee8c2-en Google Scholar
- OECD. 2024b. “PISA 2022 Technical Report.” https://www.oecd-ilibrary.org/content/publication/01820d6d-en.
- Patston, T. J., J. Kennedy, W. Jaeschke, et al. 2021. “Secondary Education in COVID Lockdown: More Anxious and Less Creative—Maybe Not?” Frontiers in Psychology 12: 613055.
- Qian, M., J. A. Plucker, and X. Yang. 2019. “Is Creativity Domain Specific or Domain General? Evidence From Multilevel Explanatory Item Response Theory Models.” Thinking Skills and Creativity 33: 100571. https://doi.org/10.1016/j.tsc.2019.100571.
- Rubenstein, L. D., L. M. Ridgley, G. L. Callan, S. Karami, and J. Ehlinger. 2018. “How Teachers Perceive Factors That Influence Creativity Development: Applying a Social Cognitive Theory Perspective.” Teaching and Teacher Education 70: 100–110. https://doi.org/10.1016/j.tate.2017.11.012.
- Rutkowski, L., E. Gonzalez, M. Joncas, and M. von Davier. 2010. “International Large-Scale Assessment Data: Issues in Secondary Analysis and Reporting.” Educational Researcher 39, no. 2: 142–151. https://doi.org/10.3102/0013189X10363170.
- Said-Metwaly, S., B. Fernández-Castilla, E. Kyndt, W. van den Noortgate, and B. Barbot. 2020. “Does the Fourth-Grade Slump in Creativity Actually Exist? A Meta-Analysis of the Development of Divergent Thinking in School-Age Children and Adolescents.” Educational Psychology Review 33, no. 1: 275–298. https://doi.org/10.1007/s10648-020-09547-9.
10.1007/s10648-020-09547-9 Google Scholar
- Sellar, S., G. Thompson, and D. Rutkowski. 2017. The Global Education Race: Taking the Measure of PISA and International Testing, 777–779. Brush Education.
- Sjøberg, S. 2015. “PISA and Global Educational Governance–A Critique of the Project, Its Uses and Implications.” Eurasia Journal of Mathematics, Science and Technology Education 11, no. 1: 111–127.
- Sjøberg, S., and E. Jenkins. 2022. “PISA: A Political Project and a Research Agenda.” Studies in Science Education 58, no. 1: 1–14. https://doi.org/10.1080/03057267.2020.1824473.
- Tan, C. Y., and K. F. Hew. 2019. “The Impact of Digital Divides on Student Mathematics Achievement in Confucian Heritage Cultures: A Critical Examination Using PISA 2012 Data.” International Journal of Science and Mathematics Education 17, no. 6: 1213–1232. https://doi.org/10.1007/s10763-018-9917-8.
- Taylor, C. L., and B. Barbot. 2021. “Gender Differences in Creativity: Examining the Greater Male Variability Hypothesis in Different Domains and Tasks.” Personality and Individual Differences 174: 110661. https://doi.org/10.1016/j.paid.2021.110661.
- Taylor, C. L., S. Said-Metwaly, A. Camarda, and B. Barbot. 2024. “Gender Differences and Variability in Creative Ability: A Systematic Review and Meta-Analysis of the Greater Male Variability Hypothesis in Creativity.” Journal of Personality and Social Psychology 126, no. 6: 1161–1179. https://doi.org/10.1037/pspp0000484.
- Wigert, B. G., V. R. Murugavel, and R. Reiter-Palmon. 2024. “The Utility of Divergent and Convergent Thinking in the Problem Construction Processes During Creative Problem-Solving.” Psychology of Aesthetics, Creativity, and the Arts 18, no. 5: 858–868. https://doi.org/10.1037/aca0000513.
- Yuan, F., and J. Zhou. 2008. “Differential Effects of Expected External Evaluation on Different Parts of the Creative Idea Production Process and on Final Product Creativity.” Creativity Research Journal 20, no. 4: 391–403. https://doi.org/10.1080/10400410802391827.
- Zhao, Y. 2016. “ Who's Afraid of PISA: The Fallacy of International Assessments of System Performance.” In Leading Futures, edited by A. Harris and M. S. Jones, 7–21. Sage.
10.4135/9789354799280.n1 Google Scholar
