Making a difference? The effects of Teach For America in high school

Teach For America (TFA) recruits and selects graduates from some of the most selective colleges and universities across the country to teach in the nation's most challenging K−12 schools throughout the nation. TFA has grown significantly since its inception in 1990, when it received 2,500 applicants and selected and placed 500 teachers. In 2005, it received over 17,000 applicants and selected and placed a little over 2,000 new teachers, and the program anticipates expanding to over 4,000 placements in 2010. In total, the program has affected the lives of nearly 3 million students.

The growth of the program suggests that TFA is helping to address the crucial need to staff the nation's schools, a particularly acute need in high-poverty schools, but TFA is not without its critics. The criticisms tend to fall into two categories. The first is that most TFA teachers have not received traditional teacher training and therefore are not as prepared for the demands of the classroom as traditionally trained teachers. TFA corps members participate in an intensive five-week summer national institute and a two-week local orientation and induction program prior to their first teaching assignment.1 2 The second criticism is that TFA requires only a two-year teaching commitment, and the majority of corps members leave at the end of that commitment. The short tenure of TFA teachers is troubling because research shows that new teachers are generally less effective than more experienced teachers (Rivkin, Hanushek, & Kain, 2005; Rockoff, 2004).

The research reported here investigates the relative effectiveness (in terms of student tested achievement) of TFA teachers and examines the validity of the criticisms of TFA. Specifically, we look at TFA teachers in high schools, and especially in math and science, where considerable program growth is planned over the next few years. To the best of our knowledge, this is the first study of TFA at the high school level.

Using individual-level student data linked to teacher data in North Carolina, we estimate the effects of having a TFA teacher compared to a traditional teacher on student performance. The North Carolina data we employ are uniquely suited for this type of analysis because it includes end-of-course (EOC) testing for students across multiple subjects. This allows us to employ statistical methods that attempt to account for the nonrandom nature of student assignments to classes and teachers, which have been shown to lead to biased estimates of the impact of teacher credentials (Clotfelter, Ladd, & Vigdor, 2010; Goldhaber, 2007).

The findings show that TFA teachers are in general more effective, as measured by student exam performance, than traditional teachers. Moreover, they suggest that the TFA effect, at least in the grades and subjects investigated, offsets or exceeds the impact of additional years of experience, implying that TFA teachers are at least as effective as experienced high school teachers in math classes and more effective than experienced teachers in science classes.

PREVIOUS RESEARCH

Research examining the impact of TFA teachers on student performance is relatively limited given its rapid expansion and the attention that the program has received from the education policy community, college students, and school districts serving low-income communities.

We found no research on TFA at the high school level. Most work has focused on elementary school teachers and some on middle school teachers. Appendix Table A1 2 3 summarizes three recent studies that provide direct estimates of the TFA effect on student achievement. The most prominent study is the random assignment study conducted by Mathematica (Decker, Mayer, & Glazerman, 2006). The Mathematica study compares student achievement outcomes in grades 1 through 5 among students taught by TFA teachers and other teachers in the same schools and at the same grade levels. Students were randomly assigned to teachers prior to the beginning of the school year to ensure there were no systematic differences between the student groups at the outset of the study. The control group included all traditional teachers, representing the set of teachers who would likely have taught the students in the absence of TFA. The study found that TFA teachers outperformed traditional teachers in student math achievement by about 0.15 standard deviation on average. When TFA teachers were compared with inexperienced traditional teachers, the impact on math achievement was even larger. No significant difference was found between TFA teachers and traditional teachers in improving students' reading achievement.

Two other studies, both using large-scale data from New York City, also found significant positive TFA effects in math. In addition to using nonexperimental data, these two studies differ from the Mathematica study by focusing on reading and math performance of students in grades 4 though 8; both differentiated non-TFA teachers into multiple categories of teachers (e.g., in terms of certification), and both explicitly took experience into account.

Kane, Rockoff, and Staiger (2006) used six years of data and found a small positive effect for TFA on student math achievement (0.02 standard deviation) relative to certified teachers, controlling for years of teaching experience. The effect was somewhat smaller for elementary school teachers (0.015) and larger for middle school teachers (0.027). They also found that the returns to experience were greater for TFA teachers than for traditionally certified teachers, though not statistically significant. The experience differentials overall were small, such that even a small difference in effectiveness may offset turnover.

Boyd et al. (2006) compared the performance of teachers entering teaching in New York City from different pathways, including TFA. Compared with “college recommended” teachers—those who fulfilled certification requirements at a university-based program registered with the state—TFA teachers were found to have a positive effect of 0.05 standard deviation on students' middle school math performance and no effect on students' elementary school math performance.

In addition to these three studies, two smaller TFA studies were conducted using data from Houston (Raymond, Fletcher, & Luque, 2001; Darling-Hammond et al., 2005). Both found positive effects for TFA in math on the state test, though the second study found negative effects on other subjects and tests. The first study compared TFA teachers to other teachers in the district; the second study compared TFA teachers to other teachers holding standard certification. These two studies, however, are not as rigorous as the New York City studies.3 4

This study focuses on TFA effects in high school, where teacher academic qualifications are particularly important (Goldhaber & Brewer, 2000). Four sections follow. We first describe the data and the variables used in the analysis. The next section discusses the analytic strategy we employ. This is followed by a presentation of results. The final section discusses the implications for policy and practice.

DATA

We focus our analysis on North Carolina because of the rich administrative databases available through the North Carolina Education Research Data Center (NCERDC) at Duke University. Since the late 1990s, the state of North Carolina has required schools to administer subject-specific end-of-course (EOC) exams during the last two weeks of the school year.4 5 We estimate the effect of Teach For America teachers relative to traditional-route teachers on student achievement in high school using EOC exam outcomes.

NCERDC collects data from the North Carolina Department of Public Instruction (NCDPI) at the end of each school year and compiles the data into annual data sets at the student, teacher, classroom, and school levels. Student data contain information on ethnicity, gender, exceptionality status, grade level, district and school code, survey data on parent education and homework habits, and scale score achievement levels for any EOC exams taken by a student in a given year.5 6 Teacher data include salary, experience, licensure, educational attainment, PRAXIS test scores, and National Board Certification. Teach For America staff helped us construct a separate data set showing Teach For America corps members, which NCERDC later linked to their teacher data using Social Security numbers. Finally, the classroom data contain records for each activity that occurred in a North Carolina public school in a year. Records list course title, section number, semester, subject, grade level, student ethnicity and gender counts, and teacher experience, ethnicity and gender.

We limit our data to the 2000–2001 through 2006–2007 school years, the years of data available during which Teach For America corps members were teaching in North Carolina. We further limit our sample to the 23 Local Education Agencies (LEAs) that hired at least one TFA teacher at any point during this time period. Not all schools in these LEAs had TFA teachers during the study period. Since schools that hired TFA teachers may be different from those that did not in terms of school leadership, management, teacher support, and recruitment strategies, our final analytic samples are restricted to teachers and students in schools that hired at least one TFA teacher. We merge each annual student data set into a student longitudinal file. We apply the same method to the teacher and classroom data, so that we have three longitudinal files, one each at the student, teacher, and classroom levels.

To estimate the effect of a teacher on her students' testing outcomes, we must link students to their classroom instructor for the relevant EOC exam. This presents a challenge in North Carolina. The student data identify the proctor of each student's EOC exam, but the proctor is not necessarily the instructor for that student's class. In Goldhaber and Anthony (2004), the authors cite North Carolina state officials who say that at least 90 percent of the time, the students' proctor is the same person as the actual classroom teacher. They verify this information by contacting 20 large school districts and find that the proctor matches the students' classroom teacher 80 percent of the time at the elementary level. At the high school level, in Clotfelter, Ladd, and Vigdor (2010), the authors link classroom data to the student data using the classroom instructor code and the student exam proctor code and verify those matches using a fit statistic based on classroom demographics. They found a match in about 70 to 75 percent of the cases. Given the success of this method, we apply a matching and verification method similar to that used in Clotfelter, Ladd, and Vigdor, as described as follows.

First, individual students on the EOC file were aggregated into test classrooms by district and school code, year, test proctor, subject, and class period. Each resulting record is associated with one proctor and lists classroom-level demographic, exceptionality, and grade-level information. Next, we turn to the actual classroom data. We keep only course descriptors requiring EOC assessments as stipulated by the North Carolina Department of Public Instruction (Appendix Table A2) and collapse records for the same course meeting that differ only on the semester variable into one record per year.6 7 With both the EOC and classroom data aggregated into unique classroom-by-year-by-subject records, they can be matched. To do so, we link all exam classrooms in a school and year with all course activities related to the test subject in that school for that school year. Then we verify the matches using the teacher ID variable and a fit statistic similar to the one used by Clotfelter, Ladd, and Vigdor (2010). This statistic measures the expected squared deviations of total classroom membership count, number of white students, and number of male students between test classrooms and actual instructional classrooms.

We go through a number of steps to verify possible matches. First, we consider those classes matched by a uniquely identifiable teacher ID. If more than one exam classroom match occurred for an actual classroom teacher in the same section, course, school, and year, we keep the match with the lowest fit statistic (thus closer resemblance between the exam and actual classrooms). Among these retained matches, cases where the fit statistic is greater than or equal to 1.5 are deemed unreliable and hence discarded. The remaining cases are considered “good” matches with reasonable confidence. They constitute our first classroom and teacher subsample (sample A).

With those matches set aside, we use the fit statistic to verify classroom matches within school, year, and subject that do not match on teacher ID. The general idea is that, when an EOC exam is administered by a teacher other than the classroom instructor, if the test classroom sufficiently resembles the instruction classroom in terms of student demographic compositions, a classroom instructor can be reliably assigned to that group of students. The success of this strategy relies on the number of test classrooms within a school–year–subject combination and how distinctive they are. In our high school data set, the median number of test classrooms within each school, year, and subject is 6, and they appear to be sufficiently different from each other to be distinguished by demographic distributions. For each unique actual instructional classroom, we keep the test classroom that matched with the lowest fit statistic. Even after identifying the best match, if the fit statistic is equal to or greater than 1.5, we drop that classroom. The remaining matches constitute our second classroom and teacher subsample (sample B). We then combine those classrooms matched by teacher code and verified with those matched using only the fit statistic. In this data set, if a test proctor matches two actual classrooms, we keep the match with the lower fit statistic.

Once classroom instructors are identified, we attach them back to the student-level test data and link teacher characteristics data to the actual classroom instructor. Using this method, we are able to match about 84 percent of students to their teachers. For the purpose of model estimation, we use two alternative analytical samples to ensure estimated TFA effects are robust to matching methods. The first sample includes all teachers who are either matched on their ID and verified or matched by class demographic variables only. As we are less confident with cases where proctors and instructors are matched solely on the basis of class demographics, the second sample includes only those teachers who are matched on ID and verified by class demographics.

ANALYTIC STRATEGY

A key challenge to the estimation of TFA teacher effects is possible nonrandom sorting of teachers and students both across and within schools. Evidence has shown a matching between observed teacher qualifications (such as years of experience) and student achievement, possibly as a result of teacher preference and parent pressure (Clotfelter, Ladd, & Vigdor, 2010). When both teacher quality and student performance are systematically related to student ability and motivation, the relationship between teacher and student performance cannot be reliably estimated. In this particular study, if TFA teachers are assigned to students with greater needs, estimated TFA effects are likely to be downwardly biased; on the other hand, if TFA teachers are systematically assigned to less challenging classes, OLS estimates of TFA effects are likely to be biased upward.

To mitigate such potential biases resulting from nonrandom matching of teachers to students, student fixed effects models are typically used when longitudinal data are available. These models take advantage of repeated student performance measures over time, and identify teacher effects using within-student variation of teacher inputs:

(1)

where y_it represents student i's test score in year t, and X is a vector of time-varying individual characteristics and other control variables. In this model, the residual term includes two components: a time-constant component c_i and a “usual” residual component u_it that is homoskedastic, uncorrelated with any independent variables or c_i , and not autocorrelated. c_i captures any student characteristics that are fixed over time, both observed (such as gender and race and ethnicity) and unobserved characteristics (such as ability and academic orientation) that may be related to teacher sorting. Since these characteristics are constant for each student over time, they drop out of the equation by demeaning Equation (1). In this way, the confounding factors of nonrandom teacher – student sorting are removed, and teacher effects can be consistently estimated.

In our high school analysis, however, we do not have repeated measures of student performance in a particular subject over time. Most often, students take a subject, such as Algebra I, once. As a result, this study adopts an alternative fixed effects model used by Clotfelter, Ladd, and Vigdor (2010) given the rich nature of the North Carolina data. Instead of using within-student variation over time, the model takes advantage of within-student variation across subjects that are evaluated by EOC exams in North Carolina.7 8

(2)

The subscript j denotes EOC subjects. T_ij represents student i's teacher in subject j. The key variable of interest in this vector is a TFA indicator variable that is equal to 1 if the teacher is affiliated with the TFA program and 0 otherwise. T_ij can also include other teacher qualification variables such as teacher experience as well as classroom variables. Analogous to a standard student fixed effects model, c_i captures student characteristics such as unobserved ability that are constant across test subjects. Although it may be reasonable to assume general student ability to be relatively stable over time, whether or not student ability captured by this error term is constant across subjects needs to be verified. If c_ij = c_i for all j, the fixed effects transformed equation of this cross-subject model is:

(3)

where variables with a superscript bar denote student-specific means across subjects and c_i is removed from the equation.

As should be clear, whether or not the student-specific error term varies by subject is key to the validity of cross-subject fixed effects models. If the assignment of TFA teachers is based on subject-specific student ability that is multidimensional, the nonrandom matching of teachers to students remains unaccounted for in these fixed effects models.

Using the same North Carolina high school data, Coltfelter, Ladd, and Vigdor (2010) investigate this crucial question in great detail. They find that students with strong math ability and students with strong reading ability in the eighth grade have similar high school course track assignments (either in an advanced algebra course or in an advanced English course). Among students with different abilities in math and reading, those with relative strengths in one subject are no more likely than those relatively weak in the subject to be assigned to an advanced course. In addition, they find no relationship between the relative credentials of a student's algebra and English teachers and the student's ability in math relative to reading. Their investigation concludes that in North Carolina high schools, student ability varies little by subject; when schools assign students to classrooms, they appear to consider student ability to be “single dimensional.”

For further assurance, we conducted a direct examination of the eight core EOC subjects using an exploratory factor analysis. The results show that all eight tests are loaded predominantly on one single underlying factor, with factor loadings (representing the correlations between test scores in each subject and the factor) ranging from 0.74 to 0.87 (Figure 1). The analysis shows that a single underlying factor accounts for 67 percent of the variance in test scores across all eight subjects. None of the remaining factors accounts for more than 8 percent of the total variance. When we apply the same analysis separately for math subjects and science subjects (not shown in the figure), we find even stronger commonality between subjects that are within the same subject area (84 percent of the variance in all math subjects and 78 percent of the variance in all science subjects is accounted for by a single factor). These evidences lend further support to the assumption that students performing well in one subject are also likely to perform well in other subjects, and that any teacher–student sorting based on ability in one subject probably will follow similar patterns if such sorting were based on student ability in any other subjects.

The lack of an initial student performance measure in a specific subject has another important implication for our cross-subject student fixed effects model. Since education is a cumulative process, academic performance depends not only on contemporaneous inputs but also on inputs from all previous time periods. Levels of academic performance at the beginning of the current time period capture students' cumulative education experiences up until that point. As a result, value-added models are typically used to estimate teacher effects on student performance. Without initial test scores for high school EOC subjects, we are not able to specify a model that controls for lagged student performance on the right-hand side of the equation (or the construction of a gain score).

In effect, our model without pretest information assumes complete “decay” of prior input; that is, initial academic preparation in a specific subject at the time of class enrollment has a negligible effect on EOC test scores. What the cross-subject model does account for is the overall level of performance across eight subjects. Clotfelter, Ladd, and Vigdor (2007) argue that a model with a missing lagged term leads to downward bias in estimates, and that the less the decay, the larger the downward bias.8 9 The TFA effect estimated using the cross-subject model, therefore, is likely to provide the lower bound of the true effects.

RESULTS

DISCUSSION

The research reported here is related to larger education policy and practice concerns about teacher quality, especially teacher quality for disadvantaged students. Teach For America taps into a nontraditional pool for teachers. The teachers TFA recruits and selects differ from traditional teachers, on average, in a number of ways. They tend to have stronger academic credentials; they have not been prepared in traditional teacher training programs; they are more likely to teach for only a few years (Boyd et al., 2006); and they are assigned to the most challenging schools in the country. Given these differences, the program has been controversial. Research providing guidance on the merits of the program to policymakers and to local education administrators has been limited at the elementary school level and nonexistent at the high school level. This study represents the first study at the high school level.

Our findings show that high school TFA teachers are more effective than the teachers who would otherwise be in the classroom in their stead. While these other teachers are a diverse group in terms of background and training, for policy purposes they are an appropriate comparison group. Other things being equal, the findings suggest that disadvantaged students taught by TFA teachers are better off than they would be in the absence of TFA.

But there are additional policy questions. Suppose we raised the bar on teacher qualifications and required that all high school teachers be fully licensed in their field, particularly teachers of math and science. Raising the bar may also mean we would have to raise salaries to attract sufficient numbers of qualified teachers. But under these conditions, would students be better or worse off with a TFA teacher? To examine this question we restricted the comparison to traditional teachers who were fully certified in field. The TFA advantage still held.

Or suppose we required that all teachers teaching disadvantaged high school students have, say, three years of prior experience. Would students be better or worse off with TFA teachers on average? The findings show that TFA status more than offsets any experience effects, particularly in teaching high school science subjects. Disadvantaged secondary students would be better off with TFA teachers than with fully licensed in-field teachers with similar or more years of experience.

We should note that the findings here do not necessarily mean that there is no value to teacher training. It is possible that the teachers that TFA recruits and selects would be even more effective with more pedagogical training. Our findings also do not mean that there is no value to teaching experience. As reported in Boyd et al. (2006), TFA teachers can become even more effective with experience at a rate that is comparable to the return to experience for non-TFA teachers.

The findings have important implications for the recruitment and selection aspects of human resource management in education, at least for high school teachers. They stress the likely importance of strong academic backgrounds for high school teachers: Once we control for PRAXIS scores and college selectivity, the estimated “TFA effect” is significantly reduced. For example, the TFA effect (controlling for experience and compared with all other teachers) is reduced from 0.132 to 0.035 in math subjects and from 0.189 to 0.151 in science subjects.14 -31 The findings also suggest that policymakers probably should focus more on issues of teacher selection, and less on issues of teacher retention, if the concern is the performance of disadvantaged high school students, especially in math and science. In short, they suggest that programs like TFA that focus on recruiting and selecting academically talented recent college graduates and placing them in schools serving disadvantaged students can help reduce the achievement gap, even if teachers stay in teaching only a few years.