Developing an Automatic Pronunciation Scorer: Aligning Speech Evaluation Models and Applied Linguistics Constructs

Pronunciation as a Component of the Construct of Second Language Speech

Until relatively recently, pronunciation had largely been absent from the L2 speaking proficiency research and assessment landscape (Suzukida & Saito, 2022). Since the introduction of automatic speech scorers, however, pronunciation has gained ground as a component of L2 speaking proficiency, especially given the capabilities of such systems to measure acoustic as well as temporal features of L2 speech (Hsieh et al., 2019; Zechner et al., 2009). Additionally, compared to earlier approaches to L2 pronunciation, the growing status of English as an international language has motivated a shift from the native-speaker standard in pronunciation learning and assessment to a new standard focusing on intelligible speech (producing individual words or utterances that are understood by a listener; Jenkins, 2006; Levis, 2020). This intelligibility-based pronunciation assessment centers around successful communication rather than comparing a test taker's pronunciation with an inner-circle English accent (e.g., American English or British English; Brown, 2020; Kachru, 1997).

With this shift, research has attempted to identify the segmental (individual phonemes), suprasegmental (prosodic), and temporal features that are most important for being intelligible (e.g., Kang et al., 2022) and easily understood (i.e., comprehensible; e.g., Kang, 2010) and that can be used to characterize L2 pronunciation proficiency. The field of L2 assessment, however, is yet to agree on features that define the construct of L2 pronunciation and the related subconstructs of intelligibility and comprehensibility that are usually the focus of L2 pronunciation learning (Huensch & Nagle, 2023; Saito & Plonsky, 2019). Nevertheless, attempts have been made to define and empirically identify these features, linking them to L2 pronunciation and spoken proficiency.

With regard to segmentals, several earlier frameworks identified contrastive features that may affect intelligibility and comprehensibility and, by extension, affect L2 oral proficiency evaluations. These frameworks include Catford's (1987) functional load principle that proposes a ranked list of English phonemes where phonemic contrasts that are most crucial for distinguishing meaning between words carry the heaviest “functional load” (see Brown, 1991). Munro and Derwing (2006) provided empirical evidence of functional load, finding that speakers who substitute certain high functional load consonants with contrasting phonemes are perceived as less comprehensible than those who make low functional load substitutions. These findings were supported in Derwing and Munro's (2015) more recent conceptualization of segmental features that affect intelligibility. In this work, issues that may lead to less intelligible speech include segmental deviations (e.g., insertion, deletion, substitution, and distortion) and prosodic deviations (e.g., in sentence stress, rhythm, and intonation). In the assessment context, Kang and Moran (2014) trialed the functional load for evaluating oral proficiency. Although these frameworks were not incorporated into existing automatic pronunciation scorers, they served as a foundation for later empirical efforts to validate pronunciation features.

Research has also suggested that certain suprasegmental and temporal features are important predictors of judgments about L2 speakers’ comprehensibility and oral proficiency (Kang, 2010). Some of these features include prominence (e.g., Field, 2005), pitch range (e.g., Kang, 2010), intonation composition of utterances (e.g., Kang & Johnson, 2018), as well as speech rate, length of silent pauses, and, to some extent, rhythm (e.g., Kang et al., 2022). Lexical stress, defined as prominence on one syllable, which makes the syllable sound longer, higher (in frequency), and louder, has also been associated with intelligibility and comprehensibility of L2 speech (Hahn, 2004; Zielinski, 2008). Other fluency-related features associated with intelligible and comprehensible speech include breakdown fluency (Hsieh et al., 2019), repair fluency (Iwashita et al., 2008), mean length of run and expected pause ratio (Kang & Johnson, 2018), and other fine-grained microfluency features (Yan et al., 2021).

At the linguistic competence level, pronunciation has not been explicitly discussed in well-known communicative models of language assessment (e.g., Bachman & Palmer, 1996); however, it has been somewhat incorporated into the Common European Framework of Reference (CEFR; Council of Europe [COE], 2001, 2020). The CEFR describes pronunciation competence across A1–C2 proficiency levels in terms of intelligibility, individual sounds, prosodic features, and fluency. Taken together with features born out of previous empirical work, these competence descriptions serve as a useful source for rubric development for human scores that can be used for training automatic pronunciation scorers. Note that in developing the automatic pronunciation scorer in this paper and validating it via human expert ratings, we took a similar approach. That is, we operationalized pronunciation as a combination of features related to segmental and suprasegmental speech features, fluency features, and perceived intelligibility.

Pronunciation in Automatic Speech Scorers

Automatic pronunciation scorers almost always rely on some form of automatic speech recognition (ASR). ASR used in speech autoscorers has been criticized for having a higher error rate in recognizing spontaneous L2 speech compared to scripted speech (Zechner & Evanini, 2019). These difficulties have generally stemmed from the lack of L2 speech datasets to train systems on, as well as considerable variability in L2 speech production (Qiao et al., 2021). Finally, for pronunciation specifically, L2 spontaneous speech typically displays features hindering ASR performance, including mispronounced words and disfluencies like false starts and filled pauses (Kim et al., 2024; Qiao et al., 2021). However, in recent years, ASR and automatic pronunciation scorers have improved their capabilities to accurately process L2 speech and its pronunciation features (Tejedor-García et al., 2021).

Existing automatic scorers largely reflect a growing recognition of the importance of pronunciation for communicative success, as well as the role of intelligibility and comprehensibility in pronunciation. Well-known automatic speech scorers of high-stakes English language proficiency tests (such as the SpeechRater, CASE v1.9, the Ordinate Scoring System, and the Duo Speaking Grader) each incorporate pronunciation as one of the measured subconstructs; however, there are differences in how pronunciation is operationalized. Below, we detail how each scorer operationalizes pronunciation and how the features included in the scorers relate to intelligibility- and comprehensibility-focused description of L2 pronunciation proficiency.

First deployed in 2006 for scoring the low-stakes Test of English as a Foreign Language (TOEFL) Practice Online assessments, SpeechRater (Zechner & Evanini, 2019; Zechner et al., 2009) is currently used for rating the internet-based TOEFL (iBT) speaking assessment. Overall, for the TOEFL iBT speaking section scores, SpeechRater to human rater agreement reaches r = .85, slightly lower than human–human interrater agreement of r = .90 (Zechner, 2019). SpeechRater derives its features from the subconstructs of the TOEFL iBT Speaking section rubrics (Chen et al., 2018) and from findings in the fields of second language learning and assessment (Zechner & Evanini, 2019). Pronunciation along with fluency is integrated in one of these subconstructs, namely, delivery (Chen et al., 2018). For fluency, a three-way distinction is made (Hsieh et al., 2019): breakdown fluency (related to pause frequency, length, and placement), speed fluency (e.g., syllables per second or mean length of runs), and repair fluency (related to restarts, repetitions, and self-corrections).

For pronunciation, SpeechRater includes segmental and suprasegmental features. The five segmental measures relate to the likelihood of phones compared to a reference model. Suprasegmental features relate to stress (five features), tone/intonation (two features), and rhythm (12 features). For the stress features, shorter distances and more uniform intervals between stressed syllables receive higher scores. Similarly, tone features relate to distance between tone units but correlate less to proficiency scores. The rhythm features focus on interval-based aspects of vowel sounds, consonant sounds, and syllables in the speech signal. In terms of construct representation and previous research on pronunciation-related features, it appears that SpeechRater, although providing a comprehensive coverage of temporal, fluency, stress, and rhythm features, is somewhat limited in its assessment of segmental features. For example, although phone likelihood comparison is included, it provides only one measure for vowel duration (comparing mean test taker vowel and expert speaker vowel durations). At the suprasegmental level, tone/intonation is limited to only two features.

The ordinate scoring system (Pearson, 2019) is the automatic speech scorer that underlies the PTE Academic. It is trained on a large number of responses from test takers and scores from human raters. Pearson (2019, p. 6) claims to measure “many hundreds of linguistic and acoustic features in each response,” with the following example types mentioned specifically for pronunciation: pace, timing, rhythm, power of voice, emphasis, intonation, and accuracy of pronunciation. These features are grouped into pronunciation and fluency and intonation. Pearson (2019) reports that the correlation between ordinate machine scores and human scores was r = .96. There is limited transparency about how specific ordinate features map to overall pronunciation (including comprehensibility and intelligibility) and little detail on how certain segmental and suprasegmental features are measured.

Finally, CASE is the automatic speech scorer used to score the Linguaskill General Speaking test (Linguaskill) by Cambridge Assessment English. Xu et al. (2021) have described the features of CASE (Version 1.9) as being related to the subconstructs of intelligibility, intonation, fluency, vocabulary, and grammar. The exact features (or number of features) of CASE are not publicly available; instead, validation of the automatic scorer is presented in terms of the reliability of the results when compared to human ratings (56.8% exact agreement and 96.6% adjacent agreement between automatic speech scorer and human grades; Xu et al., 2020).

For each speaking item scored by the Duo Speaking Grader, the automatic speech scorer extracts features and then combines them into a single portfolio that is graded using a machine learning (ML) model. The current features have been grouped into six subconstructs: content, discourse coherence, vocabulary, grammar, fluency, and pronunciation (Cardwell et al., 2024). The current model contains pronunciation and fluency features that relate to the length of the response (e.g., total number of tokens, total duration), the speed of the response (e.g., words per second), and accuracy, measured through the acoustic model's confidence in the transcription. In essence, this score can be seen as a proxy for intelligibility as it is the extent to which the ASR model can actually “understand” test taker speech.

Current automated scoring systems include elements that are typically proxies for intelligibility and comprehensibility, operationalized through finer-grained fluency, segmental, and suprasegmental features. This paper reports on the development of a feature for the Duo Speaking Grader that is a more holistic measure of pronunciation, including aspects of intelligibility and comprehensibility, in order to improve measurement of the speaking construct.

Machine Learning Methods on Pronunciation Assessment

The application of machine learning methods in pronunciation assessment has largely centered on computer-assisted pronunciation training (CAPT), focusing on providing pronunciation feedback for L2 learners (Neri et al., 2008; Rogerson-Revell, 2021). Goodness of pronunciation (GOP) has emerged as a widely used measure for evaluating L2 speech by quantifying pronunciation accuracy on the basis of confidence scores derived from ASR models. Initially, GOP was computed using hidden Markov model (HMM)-based ASR systems, which assessed pronunciation quality by estimating the likelihood that specific phonemes were present in the speech signal at each moment (Kim et al., 1997; Witt & Young, 2000). With advancements in deep neural network (DNN)-based ASR, GOP estimation has improved, enhancing phoneme discrimination and reducing word error rates (Amodei et al., 2016). Beyond phoneme-level assessments, DNN models support multigranularity scoring, aggregating GOP scores at the phoneme, word, and utterance levels to provide a holistic evaluation of pronunciation (Gong et al., 2022; Lin et al., 2020). Further refinements leverage acoustic features from pretrained DNN models to enhance pronunciation assessment beyond traditional GOP scores, for example, by integrating phoneme transition patterns and linguistic-acoustic similarity (Lin & Wang, 2021; Liu et al., 2023).

However, CAPT's construct differs substantively from high-stakes L2 pronunciation assessments. CAPT tools are designed to enhance learners’ pronunciation by providing detailed phonetic feedback on various aspects such as phoneme articulation, stress, and intonation. In contrast, high-stakes assessments require a holistic evaluation that integrates pronunciation into the broader context of language proficiency. Such evaluation involves ensuring that the automatic scorers not only accurately reflect pronunciation quality but also align with theoretical constructs of communicative competence (Choi & Kang, 2023), including how segmental, suprasegmental, and temporal aspects of pronunciation play into that competence.

Bias in Automated Scoring

Another aspect of automatic pronunciation scoring that needs to be accounted for is potential bias toward certain speakers’ attributes, mainly due to the scorer's underlying acoustic models used for ASR and its training data. Early automatic scorers struggled with L2 speech because the ASR systems were designed for first language (L1) English speakers (Litman et al., 2018). Such systems tend to compare speaker pronunciations against a standard “ideal” accent, even though achieving native-like pronunciation is not necessarily realistic, or even desirable, for effective communication (Levis, 2020; Suzukida & Saito, 2022).

Studies have also found that speaker characteristics like L1, dialect, gender, and age can substantially impact how automatic scorers evaluate pronunciation. Just as human assessors’ perceptions can be influenced by a speaker's linguistic background and identity (Carey et al., 2011; Kartushina & Frauenfelder, 2014), automatic scorers show varying sensitivity to speaker characteristics. This bias tends to appear particularly in how they handle accented speech (i.e., the extent to which L2 speech is perceivably different from the L1 norm; Kang, 2010), with many automatic pronunciation scorers struggling to fairly evaluate pronunciations that differ from their training data's accent profile (Eskenazi, 2009). Saenz and Hain (2021) found that different automatic pronunciation scorers showed varying sensitivity to speaker background characteristics like L1 and dialect used in everyday life, residence in a multilingual household, and other characteristics. These findings align with human assessor behavior, in which prior experience with certain accents and perceived speaker identity are known to influence pronunciation evaluation (Winke et al., 2013). Finally, studies have noted that audio quality can introduce bias when L2 speech is automatically measured and scored (Chen et al., 2018), emphasizing the need to investigate the effect that devices, which capture speech, have on scores. Given these possible limitations of automatic speech scorers, there needs to be an estimation of the degree to which they may display bias towards speakers on the basis of construct-irrelevant characteristics, such as their L1, gender, and the technology used to capture their spoken responses. In the context of high-stakes language assessments, investigations of these scorers for potential bias generate core pieces of validity evidence (Williamson et al., 2012) and are a central practice of responsible use of AI in language assessment development (Burstein, 2023).

Automatic Pronunciation Scorer Development

This section describes our proposed automatic pronunciation scorer. We first briefly introduce the hierarchical pronunciation scorer based on the study by Liu et al. (2023), which was originally proposed for read-aloud tasks in which speakers all read the same script. Then, we describe the method used to adapt this pronunciation scorer to the responses for open-ended speaking tasks common in high-stakes assessments. The following section provides an explanation of the developed pronunciation scorer (see Appendix S1 in the online Supporting Information for further technical details).

Hierarchical Pronunciation Scorer

The pronunciation scorer uses DNNs to analyze the audio and progressively aggregates information from four stages: frame-level (a small, fixed-length segment of audio data processed as a unit), phoneme-level (meaningful English speech sounds), word-level (constructed from phonemes), and utterance-level (a unit of speech bounded by silence, often equivalent to written phrases, clauses, or sentences). A pronunciation score for the audio input is generated from the utterance-level features.

Extracting Frame-Level and Phoneme-Level Acoustic Features

We employed a phoneme recognition model to analyze raw audio waveforms and extract frame-level acoustic features. These features encode small segments (around 20 milliseconds) of the waveform into patterns that facilitate phoneme identification. To obtain phoneme-level features, consecutive frames corresponding to the same phoneme were grouped together, allowing the model to capture phoneme-specific characteristics more effectively.

An example of frame-level and phoneme-level acoustic features extracted by a phoneme recognition model from the audio waveform is shown in Figure 1. The bottom of the figure shows a direct representation of the raw audio waveform. The x-axis represents time, and the y-axis represents the amplitude of the sound. The vertical dashed lines mark the boundaries of individual phonemes. The middle of the figure displays the frame-level acoustic features extracted by the phoneme recognition model. These features are depicted as a sequence of colored vertical lines, where each line represents a frame of approximately 20 milliseconds. The frame-level features are averaged over the duration of each phoneme, providing a compact representation of the phonetic content of each phoneme (depicted at the top of the figure). The color intensity indicates the magnitude of the acoustic features at each time step.

Adapting to Open-Response Pronunciation Assessments

The automatic pronunciation scorer described above was primarily designed for read-aloud tasks, which allow direct comparison between an L2 speaker's speech and a canonical text. In contrast, open-response speaking assessments reflect more dynamic and authentic use of language. These responses are typically longer, comprising multiple utterances. Therefore, the original pronunciation scorer, which evaluates single utterances against a fixed canonical written text, was adapted to assess spontaneous speech.

Extracting Reference Transcriptions

Figure 2 illustrates our proposed assessment pipeline for the open-response scenario. Initially, the Whisper ASR system (Radford et al., 2023), a large-scale trained DNN model, was employed to transcribe the spoken response. Table 1 provides examples of how Whisper ASR processes and transcribes speaking responses, illustrating its handling of disfluencies and pronunciation variations. As these examples illustrate, Whisper simplifies and clarifies the spoken responses by removing redundancies and hesitations (in gray) and making corrections (in blue). This cleaning process emphasizes the system's ability to capture the essential content of a response. The Whisper transcription therefore serves as a reference text, enabling the pronunciation scoring system, which was originally designed for read-aloud tasks, to be applied to open-response speech by aligning the spoken input with an automatically generated transcript.

Table 1. Example of large-sized Whisper's automated speech recognition transcription of a B1 response (original transcription → Whisper's transcription)

• Note. [PAUSE] represents noticeable pause. Gray words with strikethrough indicate deletions. Blue words with strikethrough indicate replacements.

Dataset Construction: Human Ratings Pronunciation Dataset

In this section we describe the creation of the training dataset, the rubrics used to assign gold-standard labels, and the rater training procedures and their rates of agreement. To train and evaluate new automatic pronunciation scorers, we created a gold-standard dataset of pronunciation ratings from expert human raters. Human ratings are used extensively as part of the training and evaluation of the DET's writing and speaking graders, as described by Cardwell et al. (2024). However, unlike previous human rating datasets that scored overall writing/speaking proficiency holistically, the ratings in this dataset focused on the subconstruct of pronunciation.

In total, 2,624 individual speaking samples from 1,312 test takers were rated (Table 2). This total is a combination of three subdatasets: (a) a pilot subset, (b) the main subset of L2 English test takers, and (c) the main subset of L1 English test takers. Unlike other pronunciation rating datasets (e.g., Zhang et al., 2021), the samples in the dataset are representative of DET test takers with respect to background and proficiency across all six CEFR levels, including pronunciation ability. The inclusion of L1 English speakers as a baseline comparison group for pronunciation assessment is common practice in L2 speech intelligibility research (e.g., Kang, 2010).

The pilot dataset was created primarily to trial all elements of the procedure described below, including the appropriateness of the rubrics and the length of the speech samples. On the basis of the pilot dataset, the rubrics were amended slightly, but otherwise no other changes were deemed necessary. In addition, the pilot ratings were used as benchmarks as part of the training and standardization materials for the raters. In the main subset, approximately 20% of all ratings were double-rated to ensure rater reliability while still considering practicality. Among these double-rated samples, only five resulted in a rating difference of two (on a scale from 1 to 6), and no differences were greater than two. These five responses were adjudicated by a third rater.

Dataset Compilation

Prior to creating the dataset, we decided that the ideal sample length would be approximately 30 seconds, rather than the entire test taker responses, which have maximum durations of 1 to 3 minutes, depending on the task. We chose this sample length to balance the shorter length preferred for machine learning with the longer length desired for human raters. This decision was also informed by reliability research, which shows that human rater reliability improves up to 30 seconds (compared to 5 and 10 seconds) but does not show further improvement at 60 seconds (Cucchiarini et al., 2002; Moran et al., 2019).

To compile the dataset, we extracted approximately 30-second segments from the original speaking responses using text–audio alignment. The segments were selected at the utterance level, with each segment defined by end sentence punctuation (“.”, “?”, “!”) in the transcription. Consecutive utterance segments were combined to ensure the total duration was approximately 30 seconds. From these segments, one was randomly selected from the available options. This method ensured that there were no mid-utterance or mid-word cuts in the extracted segments. These segments underwent no preprocessing steps, such as voice activity detection or enhancement, which might otherwise have influenced the integrity of the raw data.

In following this stratification scheme, we sought to ensure a diverse sample of test takers, especially in terms of accent/English variety, representative of the global DET test taker population.

Rubrics

The rubrics used to assign the pronunciation ratings consist of a scale from 1 to 6, corresponding to CEFR bands A1 to C2 (Appendix S2 in the online Supporting Information). Each score point is accompanied by the descriptors from three CEFR linguistic competence scales: overall phonological control, sound articulation, and prosodic features. However, these scales were not rated separately as analytic ratings but as one holistic rating. The descriptors were slightly edited after the piloting process. These rubrics can be seen to encompass key aspects of pronunciation including segmental features, prosodic features, and perceived intelligibility/comprehensibility. In addition, raters had the option to mark responses as having poor audio quality or as having insufficient language to accurately assess pronunciation. These samples were then excluded from further analysis.

Raters

Four expert human raters were used in this project: two “in-house” raters for the pilot subset and two “contractor” raters for the main subset. The two in-house raters developed and refined the rubrics and training materials on the basis of their piloting. The two contractor raters then underwent training and certification prior to commencing their rating (delivered by the lead in-house rater using selected samples from the pilot data). All contractor ratings were completed independently, and monitoring and feedback were provided by the lead in-house rater.

Rater Agreement

All reported agreement rates refer to the 421 double-rated items in the main subset after filtering out nonvalid responses. Three common rater agreement measures were used, quadratic weighted kappa (QWK), Spearman's rank correlation coefficient (Spearman's ρ), and two-way random single-score intraclass correlation coefficient (ICC). The agreement values for each of these measures are provided in Table 4. Regardless of the measure used, the agreement rates were very high according to the interpretation guidelines described above. Overall, for valid double-rated responses, the mean contractor ratings were 3.67 (4.0 median) and 3.57 (4.0 median), respectively. On the basis of these results, we could confidently conclude that the two raters have high interrater agreement in terms of assigning CEFR bands, regardless of test taker proficiency level, and as a result, the single-rated ratings can be considered reliable for research and development purposes.

Empirical Evaluation

In this section, we present the results of a comprehensive empirical evaluation of our automatic pronunciation scorer. We compared its performance to several baselines and assessed its fairness. This evaluation provided insights into the scorer's reliability, accuracy, and potential biases, ensuring its robustness and applicability in high-stakes language assessments.

Experimental Setups

Experimental Results

In this section, we present the empirical evaluation of our automatic pronunciation scorer and compare its performance, along with that of several baseline scorers, against construct-aligned human ratings. We evaluated performance using Spearman's ρ and QWK between the machine scores and the human ratings. When calculating QWK, we trained an isotonic regression model to project continuous ratings onto the human rating scale and rounded the results to obtain discrete scores (1–6). Table 5 presents the results of the proposed pronunciation scorer compared to other baseline systems.

Among all baseline systems evaluated, the GOP scores exhibited the weakest performance, Spearman's ρ = .66, QWK = .60. The ASR scores, Spearman's ρ = .72, QWK = .69, derived from the Whisper model, demonstrated improved performance over GOP.

MPA prosody scores, Spearman's ρ = .77, QWK = .71, consistently outperformed all other baseline methods, likely due to the critical role that prosodic features play in the assessment of spontaneous speech. MPA pronunciation scores, Spearman's ρ = .75, QWK = .70, exceed those of the GOP scores and Whisper ASR scores but fall short of the MPA prosody score.

When assessing the effectiveness of the proposed pronunciation scorer, the choice of training dataset plays a critical role in the scorer's performance. Comparative results demonstrate that the scorer trained on the DET's construct-aligned human ratings pronunciation dataset, Spearman's ρ = .82, QWK = .81, substantially outperformed the one trained on the Speechocean762 dataset, Spearman's ρ = .71, QWK = .65.

The proposed pronunciation scorer outperformed all baseline systems, nearly matching the correlation scores of human raters. The Steiger's (1980) test, which tests for significant differences between correlation coefficients, shown in Table 6, further validates the proposed pronunciation scorer. The statistically significant differences in correlation scores indicate that it consistently outperformed baseline systems in alignment with human ratings.

Evaluating Bias With Differential Feature Functioning Analyses

To evaluate the fairness of our automatic pronunciation scorer, we used differential feature functioning (DFF) analysis (Zhang et al., 2017) to determine whether our developed pronunciation scorer or any of the baselines had bias with respect to different subgroups of test takers. In the context of pronunciation scoring, DFF refers to a phenomenon where a subgroup of test takers tends to receive higher or lower feature values than other subgroups of test takers of the same proficiency.

In this model, b₀ is the intercept representing the expected value of F_i when G_i and θ_i are both zero. The error term ϵ captures the variation in F_i that is not explained by the grouping variable G_i or the proficiency variable θ_i. The grouping variable G_i is binary. We tested DFF of 16 distinct groups, including gender, operating system (Windows or Mac—a proxy for socioeconomic status), and 13 language family groups3 as detailed in Table 3. For the gold-standard language proficiency variable θ_i, we used human rating of pronunciation. Both the pronunciation score F_i and the language proficiency θ_i were normalized (zero mean and unit standard deviation) to ensure that the regression coefficients were interpretable across different scales of measurements.

Ideally, the pronunciation feature F_i should only be predicted by the proficiency variable θ_i without interference from the grouping variable G_i. In this case, the linear coefficient b₁ of the grouping variable G should be 0, meaning that whether an individual belongs to the grouping (G = 1) or not (G = 0) has no impact on the prediction of this individual's pronunciation. On the contrary, if the regression coefficient b₁ is statistically significantly different from zero, it suggests the presence of DFF with respect to the grouping variable G. This would mean that the subgroup represented by G = 1 systematically scores higher (b₁ > 0) or lower (b₁ < 0) on the feature F_i, independent of their overall language proficiency, compared to the subgroup for which G = 0.

In simpler terms, DFF occurs when a feature used in scoring behaves differently for certain subgroups, even if the subgroups have the same level of proficiency. For example, if male test takers (G = 0) score consistently lower than female test takers (G = 1) on a pronunciation feature, despite having the same overall language proficiency, this would indicate a bias in that feature. Such a bias would be concerning because it means the score is unfairly influenced by the test taker's group membership rather than their actual ability. Figure 3 shows the regression coefficients b₁ for the grouping variable G across various automatic pronunciation scorers and 16 subgroups, highlighting whether any features exhibit significant bias for the tested subgroups. The error bars represent 95% confidence intervals for each estimate. Regression coefficients b₁ with p-values smaller than .05 (indicated by a red dot) indicate the presence of potential DFF.

Automatic Pronunciation Scorer Development Outcomes

In response to the first research question, we developed a dataset of human ratings of pronunciation aligned with the CEFR standards and recent developments in L2 pronunciation, which centered intelligibility and comprehensibility as key aspects of pronunciation. We then built a pronunciation scorer based on recent advancements in machine learning, which is trained and evaluated on the developed pronunciation dataset.

To ensure the quality and reliability of our developed pronunciation dataset, we first analyzed the interrater agreement among human raters. Human raters demonstrated high interrater agreement, as evidenced by Spearman's ρ, QWK, and ICC. This high agreement rate is evidence of the quality of the rubrics and rater expertise and thus, evidence supporting the validity of the pronunciation construct defined in this study. The interrater reliability reported in the study is comparable with that reported in existing meta-analytic studies of L2 pronunciation constructs (e.g., Plonsky & Derrick, 2016; Saito, 2021). The observed consistency of rater judgements suggests that the construct of L2 pronunciation was well-defined in the rubrics and that the raters were able to effectively apply the definition in assigning the scores.

High interrater reliability provided a foundation for the training and evaluation of the automatic pronunciation scorer. The proposed pronunciation scores demonstrated a strong, positive correlation with human ratings. In fact, the observed correlations were stronger than correlations observed in other similar studies, such as Ramanarayanan et al. (2017), who compared human and automatic scoring of pronunciation features in human–machine spoken interactions (Spearman's ρ between .54 and .61 for interrater agreement and between .47 and .58 for human–machine agreement), and Hong and Nam (2021), who evaluated the reliability of an automatic pronunciation scorer, r = . 43 and r = .50 for human–machine agreement. Importantly, the human–machine correlations in this study nearly matched those demonstrated by human raters amongst themselves. This observation implies that the ability of the developed pronunciation scorer to assess L2 speakers’ English pronunciation was very close to that of humans who used rubrics aligned with the field definition of effective L2 pronunciation. This high level of agreement between human raters and the automatic scorer boosts confidence in the accuracy and reliability of the scorer, suggesting that the scorer can effectively capture the target L2 pronunciation construct.

Developed Automatic Scorer's Performance Compared to Other Automatic Scorers

To address the second research question, we evaluated the efficacy of the developed pronunciation scorer in comparison to several existing baseline automatic scorers, namely GOP, Whisper ASR confidence score, MPA, and a Speechocean762-trained scorer. The developed pronunciation scorer exhibited superior agreement with human ratings, suggesting that it measured the pronunciation construct defined in the study more accurately than its counterparts.

This outcome underscores several features of our pronunciation scorer that separate it from the other automatic scorers and, possibly, play a role in making it more effective. First, in developing the automatic scorer, we clearly defined the construct of L2 pronunciation via rubrics that we aligned with the CEFR as well as the field's view on the construct. Additionally, in refining the automatic scorer, we used an L2 corpus that included speech of English learners from different L1 backgrounds, enhancing the scorer's applicability across various linguistic contexts. Furthermore, the alignment of the scorer with CEFR standards and L2 pronunciation research ensures that the scorer is comprehensive and standardized. The quality of the dataset used to train the scorer, including the diverse L1 backgrounds, length of speech files, and assessment standard-aligned human ratings of pronunciation, also emerged as a possible variable in enhancing the automatic scorer's performance, relative to the baseline scorers it was compared with.

Addressing Differential Feature Functioning

In addressing the third research question, we found no evidence of DFF by gender, consistent negative bias towards Windows operating systems, and negative bias towards two of the 14 language families. We discuss these potential biases in our developed pronunciation scorer, proposing various strategies to enhance fairness and accuracy.