2.1 Search Procedures

This study utilized a systematic review process of academic universal screening research published in eight major school psychology journals between January 2001 and January 2024. The results of the systematic review process are documented in a PRISMA flow diagram in Figure 1. Articles were extracted from the table of contents from the following School Psychology-focused journals: the Canadian Journal of School Psychology (n = 460); Contemporary School Psychology (n = 545), the Journal of Applied School Psychology (n = 418), the Journal of School Psychology (n = 945), Psychology in the Schools (n = 2337), School Psychology Quarterly/School Psychology (n = 833), School Psychology International (n = 870), and School Psychology Review (n = 910). Additional entries were entered as technical errors in 13 articles. A total of 716 articles were excluded at this stage. The initial total was 6615 entries. The table of contents for each journal labeled and organized the articles by type of article. Exclusion criteria included specific types of articles: About the Contributors (n = 1), Acknowledgment of Reviews (n = 3), Annotated Bibliography (n = 2), Announcement (n = 3), Annual Index (n = 3), Antiracism Statement (n = 1), Call for Papers (n = 3), Case Study (n = 2), Commentary (n = 109), Contents (n = 4), Contribution (n = 1), Discussion (n = 19), Editorial (i.e., Editorial, Editorial Board, Editorial Comment, Editorial Introduction, Editor's Note, Editorial Note, Editorial Notice, Guest Editorial, From the Editor, From the Guest Editor, Guest, Letter to the Editor, n = 189), Epilogue (n = 1), Erratum (i.e., Correction and Corrigendum; n = 37), Essay (n = 4), Ethics (n = 3), Introduction (n = 60), Issue Information (n = 89), Journal Update (n = 1), Notice (n = 1), Retraction Notice (n = 1), Review (i.e., Book Review, Test Review, Review, Reviewers List, n = 146), Short Communication (n = 1), and Tools for Practice (n = 32).

Next, a total of 6615 abstracts were screened to exclude articles that did not mention the words (a) universal screening or screening and (b) academic and/or (c) academic skills such as writing, math, reading, early literacy, oral reading fluency (ORF), or spelling. At this stage, 6468 articles were excluded. The intent was to identify universal screening studies specifically and not include studies that solely established predictive or concurrent validity. Next, 147 abstracts were further reviewed to identify articles that were empirically based (i.e., extant datasets, retrospective analyses, nonconceptual studies). At this stage, 19 articles were excluded, including meta-analyses and other systematic reviews.

A full-text review was conducted for 128 articles, first identifying if the articles included diagnostic accuracy analysis (i.e., classification accuracy, receiver operator characteristic [ROC] curve, area under the curve [AUC], sensitivity, specificity, etc.). Articles that only used validity, reliability, or growth modeling analyses were excluded (n = 60). The remaining articles (n = 68) were further reviewed to identify if an academic universal screener and outcome measure were utilized and that the study sample included elementary-aged participants and older. Any assessment that measured an academic construct such as CBM, CATs, and teacher ratings was categorized as an academic universal screener. At this stage, studies that measured academic behavior, such as engagement were excluded as well as studies involving primarily prekindergarten and kindergarten students. The following exclusion criterion were used at this stage: (a) did not include an academic universal screener and an academic outcome measure (n = 20), (b) did not include a sample of first grade and/or older students (n = 5), and (c) did not include either of these criteria (n = 4). Longitudinal and cohort studies where the students started in kindergarten and progressed through multiple grades were included.

The final 39 articles included in the review (a) were empirically based (b) utilized diagnostic accuracy statistics, (c) included an academic universal screener and academic outcome measure, and (d) included elementary-aged students. The articles included in the review are presented in a supplemental file along with the corresponding demographic characteristics in each article.

2.2 Data Extraction

Each article was coded to report information about sample, participant demographic information, measures, and analysis/results. Sample information included sample size, grade(s), setting, and school type. Participant demographic information included race/ethnicity, gender, ELL/LEP status, other language information, if students qualified for Free or Reduced-Price Lunch (FRL), other socioeconomic status information, and special education status. Information about measures included academic universal screener(s) used, outcome measures used, and time between administration of the universal screener and the outcome measure. The analysis and results section included information about whether the articles disaggregated the data and if they did disaggregate, how data were reported.

3.1 Measures

3.2 Sample Information

Before addressing information related to student demographic characteristics, information regarding the participants such as sample sizes, grade(s), geographic information, and type of school studies were conducted in are reported. The median sample size was 543 students (SD = 6972.30; range = 67–34,855). There were three outliers regarding sample size in our sample (M = 19,417; M = 21,237; M = 34,855). Overall, students in first through eighth grade were represented in the sample. The distribution of grades included in studies was: 33.33% of the sample from Grade 1, 33.33% from Grade 2, 58.97% from Grade 3, 46.15% from Grade 4, 33.33% from Grade 5, 23.08% from Grade 6, 17.95% from Grade 7, and 15.38% from Grade 8. (Note: the percentages do not add up to 100 as multiple studies included multiple grades). In total, 10 studies covered one grade and 29 studies covered multiple grades. More specifically, three studies were longitudinal and followed students as they progressed through multiple grades (e.g., kindergarten to Grade 3).

Next, we identified whether the authors stated the educational-level and type of school the participants attended. The participants attended the following educational levels: elementary (n = 20; 51.28%), middle (n = 3; 7.69%), and both elementary and middle (n = 4; 10.26%). A large proportion of studies did not specific the level of the school from which the students were drawn (n = 12; 30.77%). Authors mentioned that the participants attended a public school (n = 7; 17.95%), private (n = 1, 2.56%), both public and private (n = 2; 5.13%), university-affiliated developmental research school (n = 1; 2.56%), a bilingual education program (n = 3; 7.69%), or did not specify (n = 3; 64.10%).

Authors reported the geographic locations of the schools in 92.31% (n = 36) of the studies. The general region of the United States was stated in 15 (41.67%) studies: Midwest or upper Midwest (n = 4; 26.67%), mid-Atlantic (n = 3; 20.00%), south (n = 3; 20.00%), southwest (n = 1; 6.67%), southeast (n = 1; 6.67%), upper plains (n = 1; 6.67%), more than one region (n = 2; 13.33%). Students from individual states were sampled as well in 17 (47.22%) studies: Pennsylvania (n = 4; 23.53%), Texas (n = 5; 29.41%), Wisconsin (n = 2; 11.76%), Florida (n = 2; 11.76%), Massachusetts (n = 1; 5.88%), Utah (n = 1; 5.88%), Delaware (n = 1; 5.88%), and Michigan (n = 1; 5.88%). The samples of two studies represented multiple states; one study encompassed 24 states, including Louisiana, Minnesota, New York, and Illinois, and one study represented Tennessee and Wisconsin. Last, two studies were conducted internationally, one in Oman and another in the Canary Islands.

3.3 Demographic Information

Information regarding the following demographic characteristics of the samples was reported: race/ethnicity, gender, FRL, ELL, and special education status. Race/ethnicity was not included in four (10%) studies, gender was not included in nine (23%) studies, FRL status was not included in 15 (38%) studies, ELL status was not included in 15 (38%) studies, and special education status was not included in eight (21%) studies. See Supporting Information S1: Table A1 in the appendices for gender and other demographic characteristics reported in each study. See Supporting Information S1: Table A2 in the appendices for racial and ethnic demographics of students reported in each study.

For studies that reported information for each demographic characteristic category, results were compared to national demographic trends calculated from publicly available data from the National Center for Education Statistics (e.g., not all studies reported gender demographic details, so those studies were excluded when comparing student gender representation). All data were gathered for students in grades 1–8, except for students with disabilities, where data were only available for students aged 3–21. For the studies that included demographic information, the weighted average of each demographic characteristic category was calculated (Table 2). The weighted averages for each demographic category from available data were compared to the national averages. Overall, the student demographic characteristics were well-represented in this sample of studies. White students were represented at approximately the same percentage for a given sample, compared to the national average (45.12% vs. 49.34%). The same was true for students that qualified for free or reduced priced lunch (48.1% vs. 52.1%) and the percentage of Black (16.43% vs. 13.75%), Hispanic or Latino students (26.29% vs. 25.77%), and Asian/Pacific Islander (7.82% vs. 5.73%) students. In contrast, ELL students (18.09% vs. 11.79%) were slightly overrepresented in the research literature. Students identifying as more than one race (2.71% vs. 4.6%), and students receiving special education services (10.34% vs. 12.76%) were underrepresented in the research literature.

When examining the articles included in the review, three articles contained samples that were considered outliers (i.e., 19,417, 21,237 and 34,855). These samples may have contributed to inaccurate representations of the demographic make-ups of the samples in the articles. We calculated weighted averages of the demographic characteristic categories excluding the three articles we considered outliers. These calculations are presented in Table 2 as “trimmed analysis.” Gender representation did not change, nor did special education status. The percentage of Hispanic/Latino students decreased slightly but remained commensurate with national demographics (25.32% vs. 25.77%). However, changes were seen among other race/ethnicity categories. After the trimmed analysis, White (57.42% trimmed vs. 49.34% nationally) and Native American/Alaskan Native students (2.04% vs. 0.79%) became overrepresented in the sample as opposed to being at approximately the same percentage before outliers were excluded. Black students previously were overrepresented in the samples of studies reviewed, but after the trimmed analysis, Black students were very underrepresented (from 16.43% to 7.47% in the trimmed analysis vs. a national 13.75%). Asian students also went from over- to underrepresented (from 7.82% to 4.71% in the trimmed analysis vs. a national 5.73%). Multiracial students (from 2.71% to 1.59% in the trimmed analysis vs. a national 4.6%) and ELLs (from 18.09% to 21.84% in the trimmed analysis vs. a national 11.79%) started as underrepresented and then became more so. The most dramatic difference was seen among students receiving FRL, dropping from 48.1% to only being representative of 33.65% of the sample compared to 52.1% nationally.

4.1 Screeners and Outcome Measures

CBMs were the most frequently used type of screener, comprising of 58.76% of the measures that were used across studies. This is likely because CBMs are fast, easy to administer, and capture behaviors students are expected to demonstrate in the classroom (Kettler and Albers 2013). CATs were only represented nine times—the oldest was published in 2012 and all other papers were published in 2015 and beyond. Technical documents from vendors were not included in this review due to the focus on independently conducted research. The disparity between the volume of technical documents and published peer reviewed literature suggests the number of independent evaluations of CATs are limited relative to CBM. Nevertheless, CATs can be group-administered and have the potential to provide diagnostic information in greater detail than other types of screeners, more research is likely to be conducted with CATs in the future to coincide with their increase in popularity (Balu et al. 2015). Other categories of screeners were used across the entire range of years targeted in this review.

Reading screeners were the most popular domain (69.07% of included screening instruments); only three screeners assessed writing. Writing can be challenging to assess for several reasons, as students must have knowledge of a topic, adequate planning and revising ability, and an understanding of conventions of writing like spelling, word choice, and grammar (Wilson 2018). Several of the studies focusing on writing used automatic essay graders, which could reduce some of the subjectivity that a human grader might have when judging the overall quality of a student's written work (e.g., Wilson 2018; Wilson and Rodrigues 2020).

State tests were the most popular academic outcome measure, but no category of outcome measure was overwhelmingly represented among the articles reviewed. State tests may be selected for use in screening studies due to their alignment with the core curriculum that students are exposed to throughout the school year (e.g., Hosp et al. 2011; Klingbeil et al. 2019a, 2019b). Also, state tests are a measure that all students in a state are already administered and would not require researchers to administer an additional measure. Finally, state tests can be considered high stakes in their role in accountability decisions for schools and their historical ties to grade retention and graduation, which some argue has perpetuated inequities in educational outcomes for students from disadvantaged backgrounds (Acosta et al. 2020). Comparisons between states could also be taken with caution if content on state tests do not align. Few studies used writing or a combination of reading and math when predicting student performance on a measure.

4.2 A Comparison of Study Student Demographics to National Demographics

The amount and detail of reported demographic information varied across studies. Most studies were not conducted with the intent to evaluate screening practices with diverse student groups, but several studies (e.g., Basaraba et al. 2022; Pearce and Gayle 2009; Stevenson et al. 2016) sampled specifically from students who identified as ELL, students identified as Native American, or students with disabilities. Few studies (e.g., Shapiro et al. 2017; Stevenson et al. 2016) reported very vague or district-, not student-, level demographics across categories. Past systematic reviews and meta-analyses (e.g., January and Klingbeil 2020; Kilgus et al. 2014) have listed the limited level of demographic information reporting as a limitation in their work.

As stated previously, three studies included very large samples of students. Petscher and Kim (2011) sampled 34,855 students from culturally and racially diverse backgrounds, among whom over 75%–80% received FRL. Klingbeil et al. (2021a) analyzed data from 21,237 students across a demographically diverse school district in Texas. Paly et al. (2022) sampled from 19,417 students in a mid-size district in Texas, of whom over 60% were non-White. Utilizing a large sample of students can be very beneficial and in the cases of these studies, allowed for large numbers of students from culturally and linguistically diverse backgrounds to be included in research. Although positive, accessing large datasets may be unrealistic for most authors. When including the three large studies, the mean sample size across the articles reviewed was 3420 students, but the median sample size was 548. It appeared that the majority of researchers across the studies reviewed did not utilize samples larger than 5000, much less 20,000 or 30,000 students.

After excluding the three outlier samples in the trimmed analysis, the representation of demographic information shifted. The mean sample size dropped to 1381 students, though the median remained roughly the same at 541. Though some large samples were still included in this analysis, most researchers utilized a sample size of 1000 or less. This affected the weighted averages of each demographic category. These results from the trimmed analysis align more with the concerns reported by other researchers. When students from diverse demographic backgrounds are not included in research, it is not clear how accurate the measures will be as screening tools.

4.3 Studies That Reported Disaggregated Results

Aside from general concerns regarding the inclusion of students from culturally and linguistically diverse backgrounds in research, another concern is to what extent do these universal measures accurately categorize the risk status of students who are not predominantly White, from the middle- to upper-class, nondisabled, and English-speaking. Many of the authors of studies mentioned above that included samples that were large and diverse did not conduct bias analyses to evaluate the performance of the universal screening instruments specifically among student sub-groups. One method to address this concern is to disaggregate research results and run statistical analyses on specific subgroups within their population. Disaggregating data is important because it allows for a more equitable and non-colorblind approach that can help educators and researchers better tailor intervention programs and their decision-making process to the specific population of students they serve (National Forum on Education Statistics 2016). Unfortunately, disaggregation among research analyses does not always occur. In the sample of studies reviewed, only six studies disaggregated their data: Hall et al. (2022); Hosp et al. (2011); Landry et al. (2022); Pearce and Gayle (2009); Stevenson et al. (2016); and VanDerHeyden and Witt (2005). These studies tended to have a focus on a specific subgroup of students, which likely led to the increased need for data disaggregation. Once disaggregated, the data did show differences in the effectiveness of the various measures used across groups of students in these highlighted studies. Additionally, Baker et al. (2022) and Basaraba et al. (2022) utilized samples of nearly 100% Latine students, so all results essentially reflected results for a non-White population.

The most common method for disaggregation was comparing ELL students to non-ELLs (e.g., Hall et al. 2022; Hosp et al. 2011; Landry et al. 2022). Other studies disaggregated based upon other racial characteristics (e.g., Native American vs. White, Pearce and Gayle 2009; minority vs. White, VanDerHeyden and Witt 2005), student disability status (e.g., Hosp et al. 2011; Stevenson et al. 2016), or FRL status (e.g., Hosp et al. 2011; Stevenson et al. 2016). Results from these studies were generally similar between groups, but important differences were noted. The results of the study conducted by Hosp et al. (2011) indicated that the universal screening measure produced higher accuracy when determining which ELL students were at risk, but more ELL students may be missed in screenings compared to non-ELL students. Pearce and Gayle's (2009) findings were similar to those of Hosp et al. (2011) but pertained to Native American students. VanDerHeyden and Witt (2005) also compared a teacher referral system to the universal screening and determined the teacher-driven referrals tended to overidentify White students compared to an under-identification of minority students. Taken as a whole, it can be noted that certain student demographic groups could identified at different rates through universal screening systems, yet without these highlighted authors disaggregating their data, these important considerations would remain unknown. It is also important to note that out of the six disaggregated studies, only two disaggregated math results, and none disaggregated writing data. Though research should move to increased disaggregation practices regardless of subject, math screening studies should be an additional focus to ensure culturally and linguistically diverse students have access to opportunities to close proficiency gaps in the subject.

4.4 Implications for Research and Practice

4.5 Limitations and Future Directions

There were several potential limitations of this study. First, since school psychologists are often key implementers of academic universal screening, school psychology journals were the focus of this systematic review. Eight of the most popular school psychology journals were selected, and a nearly 23-year period was reviewed. Though a wide selection of screening studies was identified, there are other universal screening studies that fit the inclusion criteria for the current study that were published in journals outside of school psychology, within school psychology but in other smaller journals, or outside of the selected years. Yet, a purpose of this study was to allow school psychology researchers to engage in the degree to which their work is achieving newly adopted expectations of practice related to advocating for social justice (National Association of School Psychologists 2017). Scholars have also made the call for researchers to engage in critical self-reflection in the manner they conceptualize and engage in research (Sullivan et al. 2022). This study provides a snapshot of the current state of affairs of the school psychology literature's contribution to academic universal screening practices. Future studies that expand this study could expand to screening for other area (e.g., behavior; social emotional functioning) and by extension be more inclusive of more journal databases. Even in the current study, changing inclusion criteria that might yield a different set of articles.

This study only included students in grades 1 through 8. Students in prekindergarten or kindergarten were excluded intentionally because screening does not occur as frequently for very young students and criterions used to assess proficiency vary considerably. High school students were unintentionally excluded. There were no studies on academic universal screening that included high school-aged students in the school psychology literature from 2001 onward. Throughout the screening process, some studies were identified that highlighted behavioral or mental health screenings for high school students (e.g., Dowdy et al. 2016; Kim et al. 2017), but academic universal screening did not have the same emphasis for this demographic. Other studies that may have included an academic screening component may not have required students to complete a separate academic universal screening measure but instead collected academic data like grades or teacher reports (e.g., Margherio et al. 2019). The research in academic universal screening for high school students is limited. On the NCII website (2021), there are only four vendors with universal screening measures validated for high school-aged students: Achieve3000's LevelSet (reading); Iowa Assessments (reading and mathematics); Lexia RAPID Assessment (reading); and STAR (reading and mathematics). This can be compared to the 26 measures available for elementary-aged students. Additional research should be conducted by independent researchers targeting academic universal screening for high school students to determine if the available measures have adequate diagnostic accuracy, include disaggregated data, and are practical in the settings for which they were developed.

Conflicts of Interest

The author declares no conflicts of interest.