1.1 The Creative Ability Measure May Have Been Overly Focused on Utility of Ideas

The way creativity is defined influences how it is measured, and therefore, the outcomes obtained (Reiter-Palmon and Schoenbeck 2020). Within the creativity research field, the most widely endorsed definition is the ability to generate ideas that are both original and useful within a specific context (Diedrich et al. 2015; Runco and Jaeger 2012; Torrance 1963). Aligning with this tradition, PISA defined creativity as “the competence to engage productively in the generation, evaluation, and improvement of ideas that can result in original and effective solutions, advances in knowledge, and impactful expressions of imagination” (OECD 2024a, p. 47).

Despite their definitional emphasis on originality, PISA's operationalization of creative thinking, however, appears to have placed a stronger emphasis on utility over originality. For instance, in assessing the three ideation processes targeted in the PISA creative thinking assessment—generation of diverse ideas; evaluation and improvement of ideas; and generation of original ideas—more items focused on the first two processes (12 and 9 items, respectively) than on generating original ideas (11 items). Perhaps most importantly, the coding process for the items prioritized utility as the primary criterion, with responses initially assessed for their practical usefulness before being evaluated for originality (OECD 2024a). For instance, the item instructions focused on assessing the originality of ideas, but the corresponding coding guidelines specified that responses had to be useful first before they could be original. Even if a response lacked originality, it could still receive partial credit, but a response that was original but not useful would have received no credit. Full credit, however, was awarded only if the response incorporated an innovative approach, implementation, or idea that was simultaneously original and useful (OECD 2024a, Chapter 1, pp. 52–55). Therefore, it is evident that the assessment of creativity was influenced significantly by a utility-driven framework, potentially overshadowing the evaluative focus on originality.

The high correlation observed between the creativity measure and academic achievement in mathematics (r = 0.67) and reading (r = 0.66) further highlights this issue, suggesting that the assessment may favor convergent skills associated with traditional academic success (OECD 2024a, p.83). Consequently, the highly convergent nature of the creative performance assessment may fail to capture the divergent and exploratory skills that creative activities, both in and out of school, might have been intended to cultivate, thereby reducing or even reversing the correlation with creative activities.

1.2 The Creative Ability Measure Aggregated All Domains Into One Score

The PISA creative thinking assessment may have faced a significant limitation due to the aggregation of scores from multiple subscales into a single general creative proficiency score (OECD 2024b). While this approach provided a quantitative overview of overall creative performance, it may have lacked the precision required to capture the three dimensions of creativity targeted by PISA (i.e., originality, diversity, and evaluation of ideas) across domains of learning. Furthermore, by combining scores across varied domains—written expression, social problem-solving, scientific problem-solving, and visual expression—the PISA creative thinking assessment may have obscured domain-specific strengths and intraindividual variations in performance. In the same way that it would be ineffective to aggregate scores into a single proficiency level in mathematics, targeting domain-specific knowledge—such as algebra, geometry, or numerical reasoning—requires distinct assessments that capture each of these bodies of knowledge separately (e.g., Yao 2010).

Reiter-Palmon et al. (2012) provided empirical support for the domain-specificity of creativity, whereas earlier scholars such as Guilford (1967), Torrance (1998), and Plucker (1999) advocated for a domain-general perspective. Despite this, the majority of contemporary researchers align with the domain-specific framework while acknowledging certain domain-general characteristics like personality traits, dispositions, and behaviors (Baer 2012; Baer and Kaufman 2005; Barbot et al. 2016; Feist 1998; Kaufman and Baer 2004). For example, today it is not highly controversial to argue that creative accomplishment and impact require domain-specific knowledge (Dumas et al. 2024), and that domain knowledge also affects evaluations of creative ideas, both from self and from others (Dumas and Kaufman 2024).

As Silvia et al. (2009) noted, “the domain-specific camp can't win if it makes domain-general statistical assumptions” (p. 141). Although the PISA creative thinking measure claimed domain generalizability, the social and scientific problem-solving tasks included within it appear to be highly dependent on domain-specific knowledge. For example, in the Save a River task (OECD 2024a, pp. 65–67), item 1 asked students to generate diverse, scientifically valid ideas to explain why there are fewer frogs where the river flows out of the city. Item 2 required students to improve an experiment testing whether pollution from factories is causing the decline in frog populations in an original way. Both items seem to rely on preexisting knowledge of pollution and scientific experimentation methodology that might be taught within the academic domains of biology or ecology within a high school science class.

Aggregating scores into a domain-general framework and applying domain-general assumptions risk overlooking the nuanced, domain-specific knowledge both targeted in the assessment and potentially cultivated through domain-specific creative activities. This misalignment may have contributed to the observed negative correlation between creative thinking and participation in creative activities.

1.3 The Creative Activities Measure Appeared Vague and Possibly Inexact

The Likert response scale employed in the PISA creative activities measure may have lacked the granularity necessary to effectively capture the nuanced nature of creative engagement. To illustrate this, consider the item: “Indicate how often you participate in each activity listed below in your school. Indicate if it is not available at your school.” The listed creative activities were Art classes/activities (e.g., painting, drawing); Creative writing classes/activities; Music classes/activities (e.g., chorus, band); Debate club; Dramatics/theater classes/activities; Publications (e.g., newspaper, yearbooks, and literary magazine); Science club; Computer programming classes/activities. Students selected from the following response categories: Never or almost never, About once or twice a year, About once or twice a month, About once or twice a week, Every day or almost every day, and Not available (OECD 2024b). One key limitation is its focus on the frequency of activities without addressing their quality. For example, a child participating in an art project once or twice a month may invest substantial time and effort in planning and execution, resulting in a deeply meaningful and impactful creative experience. Conversely, daily creative activities might become habitual or routine, offering less opportunity for substantial creative engagement and creative thinking development.

Another limitation is the ambiguity surrounding the interpretation of the frequency categories. For instance, the phrase “about once or twice a week” provided no clear indication of the actual time spent on the activity—whether it referred to a few minutes or several hours. This lack of specificity likely undermined the ability to accurately assess the depth, intensity, and personal significance of students' participation in creative activities. Without a clearer framework for measuring both time investment and qualitative aspects of engagement, the scale may risk oversimplifying and misrepresenting the complexities of creative participation.

As Scott et al. (2004) observed, the effectiveness of creative training programs is highly dependent on factors such as duration, the inclusion of challenging and real-world problems, domain-specific tasks, and the application of cognitive strategies. Accordingly, many well-established measures of creative activities employ more sophisticated frameworks that account for both the length and quality of these activities (Paek and Runco 2017). Descriptively, the PISA creative activities assessment, with only eight items, is notably smaller in scope compared to other widely used creative activity measures. For instance, the Kaufman Domains of Creativity Scale (K-DOCS) comprises 50 items (Kaufman 2012), the Biographical Inventory of Creative Behaviors (BICB) contains 34 items (Batey 2007), the Creative Achievement Questionnaire (CAQ) includes 65 items (Carson et al. 2005), and the Inventory of Creative Activities and Achievements (ICAA) consists of 48 items (Diedrich et al. 2018). The Creative Behavior Scale (CBI) features an extensive 90 items (Hocevar 1979). The limited number of items in the PISA creative activity measure could have presented a challenge in fully capturing the multifaceted nature of creative activity participation. This brevity of the measure and its focus on quantity instead of quality may have contributed to the negative correlation between creative activities and creative thinking.

2.1 Creative Activities Were Simply Time to Goof Off, Not Learn to be Creative

Adolescents may participate in creative activities for reasons unrelated to enhancing creativity, such as social interaction, peer bonding, parental expectations, or fulfilling extracurricular obligations (Fredricks 2011; Mahoney et al. 2006). These motivations might divert their focus from developing creative skills to meeting social goals. Moreover, unstructured creative activities may encroach on time that could otherwise be devoted to academic learning (Marsh 1992). This reduction in instructional time could adversely affect performance on assessments requiring task-specific and convergent creative thinking, such as PISA's focus on scientific and social problem-solving (Moga et al. 2000), potentially leading to a negative correlation among the activities that reduced the instructional time and the (highly convergent) creative thinking outcome measure.

Unstructured activities might lack the necessary scaffolding, clear goals, challenges, or cognitive strategies required for fostering significant creative development (Dumas et al. 2024; Scott et al. 2004). Without these elements, students may perceive such activities as opportunities for leisure rather than skill building. When creative activities lack alignment with clear learning objectives, students are less likely to develop the declarative or procedural knowledge needed for high performance on creativity assessments (Runco 2014). While adolescents often value autonomy, excessive freedom in informal learning contexts might not promote effective skill development. Additionally, activities that are overly simplistic or disconnected from adolescents' personal goals and identities can result in disengagement (Kolb and Kolb 2005; Plucker et al. 2004). This suggests that without structured and goal-oriented approaches, creative activities may fall short of their potential to enhance students' creativity meaningfully and may even result in paradoxically lower creative thinking performance (as suggested by the negative correlations observed here).

In general, while formal education plays a central role in talent development, informal learning through extracurricular activities has been shown to contribute only modestly to differences in individual achievement within the same curriculum or program (Dumais 2006; Gagné 2000). Furthermore, such activities often contribute minimally to the development of students' knowledge and skills (Shulruf et al. 2007) and may negatively correlate with performance measures due to the reduction in more effective instructional time. This may be particularly relevant given the convergent nature of the PISA creative thinking assessment and its focus on scientific and social creative problem-solving tasks, which might be better taught to students within a formal instructional setting rather than an informal extracurricular setting.

2.2 Gains in Creativity During the Activities Did Not Transfer to the Assessment

The PISA creative thinking assessment focused on two key cognitive processes: divergent thinking, which involves generating a broad range of ideas, and convergent thinking, the ability to evaluate and select optimal solutions for specific tasks (Guilford 1967; OECD 2024a). While these cognitive processes are conceptualized as domain-general, half of PISA's creative thinking tasks were embedded within specific domains such as scientific and social problem-solving. Research suggests that creativity rarely transfers across domains (Cheng 2016; Dumas and Doherty 2024; Haskel, 2000). Domain-specific interventions, such as art programs, may enhance creativity within the specific domain they target but fail to influence other domains requiring distinct knowledge, such as music or creative writing (Moga et al. 2000). This underscores a potential misalignment between the focus of the creative activities that students participated in and the knowledge assessed by the PISA creative thinking assessment.

Moreover, creative thinking skills developed in one context may only transfer to another context effectively through explicit instruction, the use of strategic learning approaches, and a supportive creative learning environment (Lubart et al. 2011; Scott et al. 2004). This suggests that creative activities may not have supported students in developing the specific skill set needed for creativity assessment, or that skill transfer did not occur due to a lack of intentional scaffolding, targeted strategies, or the appropriate environment for skill application (Hennessey and Amabile 2010). Although the lack of transfer is more likely to produce null rather than negative correlations among creative activity participation and creative thinking, it could also result in a negative correlation when combined with other measurement artifacts we discussed previously, such as the aggregation of scores into a single domain-general score for measuring creative thinking and an emphasis on quantity rather than quality in measuring creative activities.

Key processes such as self-explanation, metacognition, exposure to varied contexts, social learning opportunities, constructive feedback, self-reflection, and self-monitoring are essential for enabling the transfer of knowledge across varied tasks (Billing 2007; Hattie and Timperley 2007; Perkins and Salomon 1992). These strategies foster adaptability, enhancing cognitive flexibility and the application of skills in novel situations.

Furthermore, assessments operate within a unique and formalized context, wherein students might prioritize rational, well-developed, or previously tested strategies over exploratory and innovative approaches. This tendency to favor safe strategies may arise from a desire to minimize risk, reducing students' willingness to engage in novel or experimental thinking during evaluations. Such cautious behavior could limit the expression of creative potential, negatively impacting performance on standardized creativity assessments, even in the case that students had developed creative thinking abilities in their creative activities inside or outside of school.

2.3 Highly Creative Teens Might Not Choose to Participate in Adult-Led Creative Activities

Adolescents frequently prioritize autonomy and independence, favoring time spent with peers over engagement in adult-led activities. This preference for independence may result in disengagement from extracurricular programs or creative assessments. One potential reason for this disengagement is that adult-led activities might be overly structured, limiting opportunities for self-directed exploration and creative expression, which are particularly valued by teenagers. Overly structured creative activities can inhibit the development of independent thinking, one of the key processes necessary for excelling in creative assessments (Hennessey and Amabile 2010). Research indicates that low levels of autonomy can impede creative performance, not only in organizational contexts (Amabile 1996) but also within educational environments (Núñez and León 2015; Reeve et al. 2004). Adolescents are believed to benefit from creative programs when they are given autonomy, relevance to their personal interests, and opportunities to solve real-world problems (Ryan and Deci 2000). Intrinsic motivation is particularly important in creative activities, as it encourages exploration, risk-taking, and persistence—all crucial elements for fostering creativity (Amabile 1985; Deci and Ryan 1985). In school environments, teacher-led activities that are excessively controlled or prescriptive could stifle students' creative engagement, especially for highly creative students who are likely to value self-direction.

In addition, a lack of recognition of creativity by teachers may exacerbate this issue. Teachers may misinterpret creative behaviors as disruptive, particularly when these behaviors challenge conventional norms or classroom management strategies (Aljughaiman and Mowrer-Reynolds 2005; Kettler et al. 2018; Scherbakova et al. 2024; Kagan et al. 2024; Westby and Dawson 1995). In such cases, creative expression may be met with punitive responses, discouraging students from engaging in creative pursuits. As a result, teachers may design overly structured activities aimed at minimizing disruptions that inadvertently suppress opportunities for creative expression. This restrictive approach may leave some highly creative students (who may perform well on creative thinking assessments) left out of creative activities for the paradoxical reason that their creative behaviors are not welcome within that context. If this pattern occurred, it may have produced the observed negative correlation between the creative activities and ability measures because of a reversed selection effect wherein the highly creative students (who scored highly on the ability assessment) selected out of creative activities, rather than in.