Tag Archives: education

TFA Effectiveness

Several people have sent me articles discussing Mathematica’s recent research study that examined Teach For America (TFA) math teacher effectiveness.  This study is significant because, to my knowledge, it is the first large-scale study on TFA to randomly assign students to classrooms.  Its experimental design provides fairly convincing support for the idea that TFA teachers’ students perform no worse than the students of non-TFA teachers at TFA placement schools.  But this finding is consistent with the findings of previous research and does not support the assertion that Teach For America teachers can close the achievement gap.

Anyone who continues to argue that TFA teachers yield worse educational outcomes than other teachers (generally citing pretty old research and ignoring the much larger body of research that contradicts that claim) are just plain wrong.  While there are some methodological concerns with recent studies, enough evidence exists for me to state confidently that TFA teachers, on average, do not harm student achievement.  At the same time, TFA and its proponents must also stop using misleading data and insisting that studies like this one prove more than they actually do.  Despite articles’ claims, the new Mathematica research does not suggest that TFA teachers’ students outperform non-TFA teachers’ students in a meaningful way.

The study showed a difference between TFA teachers and all comparison teachers of 7% of one standard deviation.  To put that number in context, a difference of 7% of one standard deviation in home runs between two baseball players in 2012 would be a difference of less than one home run over the course of the entire 162-game season.  Or, if you aren’t a baseball fan, a difference of 7% of one standard deviation between two students on the math section of the SAT in 2012 was equivalent to a difference of less than one correctly answered question.  The authors of the Mathematica study and just about every article quoting the study claim 7% of one standard deviation in this context is equivalent to 2.6 months of learning, using this 2007 research paper as justification, but that number is invalid and based on an inappropriately applied heuristic.  The average student in a non-TFA classroom scored in the 27th percentile on the tests administered while the average student in a TFA classroom scored in the 30th percentile; moving from the 27th percentile on a test to the 30th percentile does not represent, on average, 2.6 months of learning.  Furthermore, 40% of classrooms with TFA teachers scored lower than comparison classrooms taught by non-TFA teachers.  The study’s results were statistically significant, sure, but the advantage they show for TFA teachers is remarkably slight at best.

To me, the most important takeaway from the Mathematica study is that students at TFA placement schools, in general, perform terribly on standardized tests no matter who happens to be teaching them.  The reasons for that fact, as I alluded to in my last post, have a lot less to do with teaching and school quality than reformers would have us believe.  Most teachers, whether from TFA or any other program, want to help kids learn and are working hard towards that end most of the time.  But despite our best efforts, in-school reforms alone do little to impact the achievement gap.  The Mathematica study suggests that educators can only succeed if we simultaneously address economic inequality and other outside-of-school factors that disadvantage low-income students. I’d like to see critics and proponents of TFA alike stop quibbling about marginal improvements on standardized tests and start concentrating on the larger-scale advocacy that can really make a difference.

1 Comment

Filed under Education

The Teach Like a Champion Paradigm

At the middle school where I currently serve as Math Instructional Coach, part of our professional development consists of reading Doug Lemov’s Teach Like a Champion.  This book provides a list of “49 techniques that [ostensibly] put students on the path to college” and is a Teach For America (TFA) favorite.  I have nearly finished the book and many of the techniques are great – I believe it can provide considerable value to teachers.  The book has a few major flaws worth discussing, however.  Lemov makes two assumptions that are representative of what I believe to be some of the biggest problems with TFA and the broader education reform movement.

First, Lemov assumes “the highest levels of student performance” are the goal of education and measures performance with standardized test scores.  He briefly acknowledges that “state test results…are not sufficient” but then buries that comment in a bunch of prose defending the value of what test results measure.  As one of my best former teachers argues in a recent article, reformers far too often concentrate on academic concerns “to the complete exclusion of the characteristics that make a difference in kids’ lives.”

The vast majority of TFA staff and corps members I have encountered care deeply about their students, and I imagine Lemov does as well.  But I’d like to see educators focus at least as much time and energy discussing how to inspire kids to be better people as Lemov does explaining how “a walk to the bathroom is a perfect time for a vocabulary review.”  The singular focus on academic achievement and test scores in the education reform debate has also contributed to the nationwide decline in art and physical education classes since No Child Left Behind.

Second, and much more dangerously, Lemov uses his charter network’s test scores to assert: “Great teaching, [our] teachers have proven, is strong enough to close the achievement gap.”  As evidence for this claim, Lemov cites the scatterplot below, which documents the scores of every public school in New York state on a 2009 English Language Arts assessment:

New York State ELA Scatterplot

FRPL stands for free and reduced-price lunch, an indicator of student poverty levels. The circled data point is Rochester Prep, a school from Lemov’s charter network, Uncommon Schools.

Lemov’s declaration – that this chart and others like it demonstrate that teaching effectiveness closes the achievement gap – is the fuel for much of the national conversation blaming the achievement gap on ineffective teachers and schools.  Reformers argue that dismantling teachers unions, linking teacher evaluation to student test score data, shutting down “failing” public schools in favor of charters, and a whole host of other changes would drastically improve educational outcomes in this country.  TFA and books like Teach Like a Champion, both of which suggest educator skill causes the results in the graph shown above, empower these reformers.  Sadly, not only is a causal link between teacher effectiveness and student performance data unestablished and the data itself unreliable, but data presented on charter school success is also incredibly misleading.

In-school factors have never been shown to explain more than one-third of the achievement gap and recent research suggests income inequality is the predominant cause of the achievement gap.  Additionally, outliers are expected in any dataset, Uncommon Schools is one of several that fall well above the trendline in the scatterplot above, and it’s possible (though I believe it unlikely) that the school just got lucky.  But let’s leave those considerations aside for a second and ask whether the student populations at Uncommon Schools are truly representative of the populations the schools draw from.  Lemov states,

Our students are selected at random from the districts where we work, have a higher poverty rate than the districts from which we draw, and, contrary to myth, are often the least, rather than the best, prepared students in those districts (one of the major reasons parents exercise school choice is that their students are struggling and increasingly at risk in their original schools; they are moving from as much as to).

His claim that his students are selected at random is just plain untrue and contradicted by his description of school choice at the end of the sentence.  While students who apply for admission are admitted via a random lottery, any introductory statistics student can tell you that a random sample from a group of volunteers in an area is most definitely not a random sample from the entire area.  Lemov provides no evidence to support his claim that his students are less prepared than their counterparts who don’t apply because there is no evidence to support this assertion; in fact, families who have the knowledge and commitment to seek new schools for their students are more likely to find success in any environment.  Lemov’s statement, if not willfully deceptive, reflects a poor understanding of statistics.

Gary Rubinstein, a 1991 TFA corps member and seasoned educator, has extensively studied data presented by “miracle” schools, schools like those in the Uncommon Schools network that achieve seemingly unbelievable results.  Rubinstein has found that in most cases those results are unbelievable because they are, well, not worthy of being believed.  Whether charters lose students by expelling them, by not-so-subtly suggesting to their families that kids won’t be successful at the school (a practice a friend of mine who worked at a KIPP school called “counseling out”), or through some other method (perhaps unintentional), the student attrition rates at these charter schools are very high and the percentages presented in most cases don’t account for this fact.  When a “miracle” school’s change in the student body is taken into account, the school’s results are hardly noteworthy.  These schools often help students, to be sure, but the students they help are generally small, unrepresentative subsets of students in poverty.

Schools in the Uncommon Schools network are quite possibly awesome schools for the students they serve, and the teachers Lemov observed and wrote about are most likely excellent teachers.  Their results, however, despite Lemov’s and reformers’ claims, do not indicate that great teaching can level the playing field for children in poverty.  Teachers can and should work hard to change students’ lives and there are undoubtedly cases where we succeed.  But when reformers advance changes based on misleading data and/or false claims from people like Lemov, our students suffer and the goal of educational equity becomes harder to realize.

5 Comments

Filed under Education