ORIGINAL ARTICLE
Meta-Analysis of PISA Creative Thinking Assessment Data: A Guide for Creativity Researchers
Sameh Said-Metwaly,
Belén Fernández-Castilla,
Wim Van den Noortgate,
Corresponding Author
Sameh Said-Metwaly
Faculty of Psychology and Educational Sciences, Leuven, Belgium
Imec Research Group Itec, Leuven, Belgium
Faculty of Education, Damanhour University, Damanhour, Egypt
Correspondence:
Sameh Said-Metwaly ([email protected])
Search for more papers by this authorBelén Fernández-Castilla
Faculty of Psychology, Universidad Nacional de Educación a Distancia, Madrid, Spain
Search for more papers by this authorWim Van den Noortgate
Faculty of Psychology and Educational Sciences, Leuven, Belgium
Imec Research Group Itec, Leuven, Belgium
Search for more papers by this authorSameh Said-Metwaly,
Belén Fernández-Castilla,
Wim Van den Noortgate,
Corresponding Author
Sameh Said-Metwaly
Faculty of Psychology and Educational Sciences, Leuven, Belgium
Imec Research Group Itec, Leuven, Belgium
Faculty of Education, Damanhour University, Damanhour, Egypt
Correspondence:
Sameh Said-Metwaly ([email protected])
Search for more papers by this authorBelén Fernández-Castilla
Faculty of Psychology, Universidad Nacional de Educación a Distancia, Madrid, Spain
Search for more papers by this authorWim Van den Noortgate
Faculty of Psychology and Educational Sciences, Leuven, Belgium
Imec Research Group Itec, Leuven, Belgium
Search for more papers by this authorFirst published: 20 June 2025
Funding: The authors received no specific funding for this work.
Sameh Said-Metwaly and Belén Fernández-Castilla contributed equally to this work.
ABSTRACT
Interest in understanding creativity through Programme for International Student Assessment (PISA) data is on the rise, yet researchers face methodological challenges in synthesizing findings across various constructs, measures, and datasets. Meta-analysis—a valuable methodology for synthesizing quantitative data—remains underutilized in creativity research involving large-scale assessments like PISA. This paper provides guidelines for applying meta-analytic techniques to PISA creative thinking assessment data to help researchers address these challenges. It introduces meta-analysis by outlining its definition and advantages, followed by key steps and methodological considerations for synthesizing bivariate and multivariate relationships within PISA. Finally, the paper discusses techniques for managing the computational complexity of meta-analyzing PISA data. Ultimately, these guidelines aim to support researchers in effectively synthesizing PISA data to advance the study of creativity.
Conflicts of Interest
The authors declare no conflicts of interest.
Open Research
Data Availability Statement
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
- Abo-Zaid, G., B. Guo, J. J. Deeks, et al. 2013. “Individual Participant Data Meta-Analyses Should Not Ignore Clustering.” Journal of Clinical Epidemiology 66: 865–873.
- Acar, S., L. E. Lee, and R. Scherer. 2024. “A Reliability Generalization of the Torrance Tests of Creative Thinking-Figural.” European Journal of Psychological Assessment 40: 396–411.
- Ahn, E., and H. Kang. 2018. “Introduction to Systematic Review and Meta-Analysis.” Korean Journal of Anesthesiology 71: 103–112.
- Aloe, A. M., and B. J. Becker. 2012. “An Effect Size for Regression Predictors in Meta-Analysis.” Journal of Educational and Behavioral Statistics 37: 278–297.
- Barbot, B., and S. Said-Metwaly. 2020. “Is There Really a Creativity Crisis? A Critical Review and Meta-Analytic Re-Appraisal.” Journal of Creative Behavior 55: 696–709.
10.1002/jocb.483 Google Scholar
- Becker, B. J. 2000. “ Multivariate Meta-Analysis.” In Handbook of Applied Multivariate Statistics and Mathematical Modeling (499525), edited by H. E. A. Tinsley and E. D. Brown. Academic Press.
10.1016/B978-012691360-6/50018-5 Google Scholar
- Becker, B. J., and M. J. Wu. 2007. “The Synthesis of Regression Slopes in Meta-Analysis.” Statistical Science 22: 414–429.
- Booth, A. 2006. “Clear and Present Questions: Formulating Questions for Evidence-Based Practice.” Library Hi Tech 24: 355–368.
- Burke, D. L., J. Ensor, and R. D. Riley. 2017. “Meta-Analysis Using Individual Participant Data: One-Stage and Two-Stage Approaches, and Why They May Differ.” Statistics in Medicine 36: 855–875.
- Byron, K., S. Khazanchi, and D. Nazarian. 2010. “The Relationship Between Stressors and Creativity: A Meta-Analysis Examining Competing Theoretical Models.” Journal of Applied Psychology 95: 201–212.
- Chen, X., J. Q. Cheng, and M. G. Xie. 2021. Divide-and-Conquer Methods for Big Data Analysis arXiv preprint arXiv:2102.10771.
- Cheung, M. W.-L. 2015. Meta-Analysis: A Structural Equation Modeling Approach. John Wiley & Sons.
10.1002/9781118957813 Google Scholar
- Cheung, M. W., and W. Chan. 2005. “Meta-Analytic Structural Equation Modeling: A Two-Stage Approach.” Psychological Methods 10: 40–64.
- Cooke, A., D. Smith, and A. Booth. 2012. “Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis.” Qualitative Health Research 22: 1435–1443.
- Cooper, H., and E. A. Patall. 2009. “The Relative Benefits of Meta-Analysis Conducted With Individual Participant Data Versus Aggregated Data.” Psychological Methods 14: 165–176.
- Egger, M., and G. D. Smith. 1997. “Meta-Analysis: Potentials and Promise.” British Medical Journal 315: 1371–1374.
- Ellington, E. H., G. Bastille-Rousseau, C. Austin, et al. 2015. “Using Multiple Imputation to Estimate Missing Data in Meta-Regression.” Methods in Ecology and Evolution 6: 153–163.
- Else-Quest, N. M., J. S. Hyde, and M. C. Linn. 2010. “Cross-National Patterns of Gender Differences in Mathematics: A Meta-Analysis.” Psychological Bulletin 136: 103–127.
- Fan, J., F. Han, and H. Liu. 2014. “Challenges of Big Data Analysis.” National Science Review 1: 293–314.
- Feist, G. J. 1998. “A Meta-Analysis of Personality in Scientific and Artistic Creativity.” Personality and Social Psychology Review 2: 290–309.
- Fernández-Castilla, B., L. Jamshidi, L. Declercq, S. N. Beretvas, P. Onghena, and W. Van den Noortgate. 2020. “The Application of Meta-Analytic (Multi-Level) Models With Multiple Random Effects: A Systematic Review.” Behavior Research Methods 52: 2031–2052.
- Fernández-Castilla, B., S. Said-Metwaly, R. S. Kreitchmann, and W. Van den Noortgate. 2024. “What Do Meta-Analysts Need in Primary Studies? Guidelines and the SEMI Checklist for Facilitating Cumulative Knowledge.” Behavior Research Methods 56: 3315–3329.
- Funder, D. C., and D. J. Ozer. 2019. “Evaluating Effect Size in Psychological Research: Sense and Nonsense.” Advances in Methods and Practices in Psychological Science 2: 156–168.
- Glass, G. V. 1976. “Primary, Secondary, and Meta-Analysis of Research.” Educational Researcher 5: 3–8.
10.3102/0013189X005010003 Google Scholar
- Gleser, L. J., and I. Olkin. 1994. “ Stochastically Dependent Effect Sizes.” In The Handbook of Research Synthesis, edited by H. Cooper and L. V. Hedges, 339–355. Russel Sage Foundation.
- Goecke, B., M. Benedek, J. Diedrich-Rust, B. Forthmann, S. Patzl, and S. Weiss. 2024. Being Female and Being Well-Situated Implies Higher Performance on Creative Thinking Tests: Evidence Across 62 Countries From PISA 2022.
- Grissom, R. J., and J. J. Kim. 2005. Effect Sizes for Research: A Broad Practical Approach. Lawrence Erlbaum Associates Publishers.
- Groot, L. J., K. J. Kan, and S. Jak. 2024. “Checking the Inventory: Illustrating Different Methods for Individual Participant Data Meta-Analytic Structural Equation Modeling.” Research Synthesis Methods 15: 872–895.
- Haase, J., P. H. Hanel, and N. Gronau. 2023. “Creativity Enhancement Methods for Adults: A Meta-Analysis.” Psychology of Aesthetics, Creativity, and the Arts.
- Haidich, A. B. 2010. “Meta-Analysis in Medical Research.” Hippokratia 14: 29–37.
- Hansen, C., H. Steinmetz, and J. Block. 2022. “How to Conduct a Meta-Analysis in Eight Steps: A Practical Guide.” Management Review Quarterly 72: 1–19.
- Hunter, J. E., and F. L. Schmidt. 2004. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. 2nd ed. Sage Publications.
10.4135/9781412985031 Google Scholar
- Jackson, D., R. Riley, and I. R. White. 2011. “Multivariate Meta-Analysis: Potential and Promise.” Statistics in Medicine 30: 2481–2498.
- Jak, S. 2015. Meta-Analytic Structural Equation Modelling, 1–88. Springer International Publishing.
10.1007/978-3-319-27174-3 Google Scholar
- Jak, S., and M. W.-L. Cheung. 2020. “Meta-Analytic Structural Equation Modeling With Moderating Effects on SEM Parameters.” Psychological Methods 25: 430–455.
- Jak, S., H. Li, L. Kolbe, H. de Jonge, and M. W.-L. Cheung. 2021. “Meta-Analytic Structural Equation Modeling Made Easy: A Tutorial and Web Application for One-Stage MASEM.” Research Synthesis Methods 12: 590–606.
- Kambach, S., H. Bruelheide, K. Gerstner, J. Gurevitch, M. Beckmann, and R. Seppelt. 2020. “Consequences of Multiple Imputation of Missing Standard Deviations and Sample Sizes in Meta-Analysis.” Ecology and Evolution 10: 11699–11712.
- Koh, D., K. Lee, and K. Joshi. 2019. “Transformational Leadership and Creativity: A Meta-Analytic Review and Identification of an Integrated Model.” Journal of Organizational Behavior 40: 625–650.
- Koopman, L., G. J. van der Heijden, A. W. Hoes, D. E. Grobbee, and M. M. Rovers. 2008. “Empirical Comparison of Subgroup Effects in Conventional and Individual Patient Data Meta-Analyses.” International Journal of Technology Assessment in Health Care 24: 358–361.
- H. Liu, and H. Motoda, eds. 2007. Computational Methods of Feature Selection. CRC press.
10.1201/9781584888796 Google Scholar
- López-López, J. A., M. J. Page, M. W. Lipsey, and J. P. Higgins. 2018. “Dealing With Effect Size Multiplicity in Systematic Reviews and Meta-Analyses.” Research Synthesis Methods 9: 336–351.
- McGowan, J., M. Sampson, D. M. Salzwedel, E. Cogo, V. Foerster, and C. Lefebvre. 2016. “PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement.” Journal of Clinical Epidemiology 75: 40–46.
- OECD. 2009. PISA Data Analysis Manual: SPSS. 2nd ed. OECD Publishing.
10.1787/9789264056275-en Google Scholar
- Pigott, T., R. Williams, and J. Polanin. 2012. “Combining Individual Participant and Aggregated Data in a Meta-Analysis With Correlational Studies.” Research Synthesis Methods 3: 257–268.
- Pustejovsky, J. E., and E. Tipton. 2022. “Meta-Analysis With Robust Variance Estimation: Expanding the Range of Working Models.” Prevention Science 23: 425–438.
- Resche-Rigon, M., and I. R. White. 2018. “Multiple Imputation by Chained Equations for Systematically and Sporadically Missing Multilevel Data.” Statistical Methods in Medical Research 27: 1634–1649.
- Riley, R. D., P. C. Lambert, and G. Abo-Zaid. 2010. “Meta-Analysis of Individual Participant Data: Rationale, Conduct, and Reporting.” BMJ 340: c221.
- Riley, R. D., P. C. Lambert, J. A. Staessen, et al. 2008. “Meta-Analysis of Continuous Outcomes Combining Individual Patient Data and Aggregate Data.” Statistics in Medicine 27: 1870–1893.
- Rong, M., D. Gong, and X. Gao. 2019. “Feature Selection and Its Use in Big Data: Challenges, Methods, and Trends.” IEEE Access 7: 19709–19725.
- Rosenthal, R., and M. R. DiMatteo. 2001. “Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews.” Annual Review of Psychology 52: 59–82.
- Rubin, D. B. 2004. Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons.
- Said-Metwaly, S., B. Fernández-Castilla, E. Kyndt, and W. Van den Noortgate. 2018. “The Factor Structure of the Figural Torrance Tests of Creative Thinking: A Meta-Confirmatory Factor Analysis.” Creativity Research Journal 30: 352–360.
- Said-Metwaly, S., B. Fernández-Castilla, E. Kyndt, W. Van den Noortgate, and B. Barbot. 2021. “Does the Fourth-Grade Slump in Creativity Actually Exist? A Meta-Analysis of the Development of Divergent Thinking in School-Age Children and Adolescents.” Educational Psychology Review 33: 275–298.
- Said-Metwaly, S., C. L. Taylor, A. Camarda, and B. Barbot. 2022. “Divergent Thinking and Creative Achievement—How Strong Is the Link? An Updated Meta-Analysis.” Psychology of Aesthetics, Creativity, and the Arts 18: 869–881.
10.1037/aca0000507 Google Scholar
- Simmonds, M. C., J. P. Higginsa, L. A. Stewartb, J. F. Tierneyb, M. J. Clarke, and S. G. Thompson. 2005. “Meta-Analysis of Individual Patient Data From Randomized Trials: A Review of Methods Used in Practice.” Clinical Trials 2: 209–217.
- Smith, C. T., and P. R. Williamson. 2007. “A Comparison of Methods for Fixed Effects Meta-Analysis of Individual Patient Data With Time to Event Outcomes.” Clinical Trials 4: 621–630.
- Stewart, G. B., D. G. Altman, L. M. Askie, L. Duley, M. C. Simmonds, and L. A. Stewart. 2012. “Statistical Analysis of Individual Participant Data Meta-Analyses: A Comparison of Methods and Recommendations for Practice.” PLoS One 7: e46042.
- Stewart, L. A., and M. K. Parmar. 1993. “Meta-Analysis of the Literature or of Individual Patient Data: Is There a Difference?” Lancet 341, no. 8842: 418–422.
- Tang, L., L. Zhou, and P. X.-K. Song. 2016. Method of Divide-and-Combine in Regularised Generalised Linear Models for Big Data. arXiv preprint arXiv:1611.06208.
- Taylor, C. L., S. Said-Metwaly, A. Camarda, and B. Barbot. 2024. “Gender Differences and Variability in Creative Ability: A Systematic Review and Meta-Analysis of the Greater Male Variability Hypothesis in Creativity.” Journal of Personality and Social Psychology 126: 1161–1179.
- Tsai, C. F., W. C. Lin, and S. W. Ke. 2016. “Big Data Mining With Parallel Computing: A Comparison of Distributed and MapReduce Methodologies.” Journal of Systems and Software 122: 83–92.
- Van den Noortgate, W., J. A. López-López, F. Marín-Martínez, and J. Sánchez-Meca. 2013. “Three-Level Meta-Analysis of Dependent Effect Sizes.” Behavior Research Methods 45: 576–594.
- Wang, C., M. H. Chen, E. Schifano, J. Wu, and J. Yan. 2016. “Statistical Methods and Computing for Big Data.” Statistics and Its Interface 9: 399–414.
- Wang, X., and Y. He. 2016. “Learning From Uncertainty for Big Data: Future Analytical Challenges and Strategies.” IEEE Systems, Man, and Cybernetics Magazine 2: 26–31.
10.1109/MSMC.2016.2557479 Google Scholar
- White, I. R., P. Royston, and A. M. Wood. 2011. “Multiple Imputation Using Chained Equations: Issues and Guidance for Practice.” Statistics in Medicine 30: 377–399.
- Xie, J., Z. Song, Y. Li, et al. 2018. “A Survey on Machine Learning-Based Mobile Big Data Analysis: Challenges and Applications.” Wireless Communications and Mobile Computing 2018: 8738613.
10.1155/2018/8738613 Google Scholar
- Zhang, Y., T. Huang, and E. F. Bompard. 2018. “Big Data Analytics in Smart Grids: A Review.” Energy Informatics 1: 1–24.
10.1186/s42162-018-0007-5 Google Scholar
