Tag Archives: Teach For America

by | September 22, 2015 · 7:45 am

A Closer Look at TFA’s Alumni Survey and Data Practices

Teach For America (TFA) recently released a report describing some of the results from its alumni survey.  The report provides interesting information about many alums’ careers.  At the same time, however, TFA’s misleading presentation of the results and the organization’s unwillingness to share the underlying data are troubling.

Consider the following three charts:

TFA Chart 1TFA Chart 2TFA Chart 3

The first of these charts claims to show “the mean value on the measure of total years teaching for each cohort,” the second ostensibly shows both the “mean and median total years teaching, by cohort,” and the third purports to show “the percentages of alumni in each Teach For America cohort that reported teaching 3 or 4, 5 or 6, or more than 7 years.”  A reader could easily think these statistics apply to everyone who joined Teach For America between 1990 and 2010.  But the charts depict statistics only for most TFA alumni who responded at some point to a TFA alumni survey; the two biggest categories of people they ignore are alumni who never responded to a survey and former corps members who did not complete their original two-year teaching commitments (while Raegen Miller and Rachel Perera, the authors of the brief, are clear that only alumni are their focus, the word “cohort” invokes images of all corps members that began teaching during a given year).  In other words, the charts depict results for an unrepresentative subset of the teachers who join Teach For America.

To their credit, Miller and Perera do discuss these issues in footnote 4:

The 23,653 alumni represented here represent about 85 percent of the alumni from the 1990 to 2010 cohorts. While alumni who don’t respond to the survey may be systematically different than those who do respond, the response rate is high enough to bolster the case that our statistics speak pretty well to the population of alumni. The alumni survey is administered only to those who complete their initial commitment to teach for two years. Alumni reporting fewer than two years of teaching on the survey, or more than the number of years possible since their corps year, were excluded from these analyses.

Still, most consumers of these misleading charts, especially when a chart is tweeted or included in a summary blog post, are unlikely to see footnote 4.  The footnote also doesn’t address the fact that TFA’s presentation of their survey results is in some cases downright untrue, as is the case in the following sentence from this TFA media piece: “Among more than 42,000 Teach For America alumni, some 84 percent report working in education or other capacities serving low-income communities.”

This statement is false because, as noted above, TFA does not have reports from the full population of “more than 42,000 Teach For America alumni” (TFA alumni may also incorrectly report that they work in “other capacities serving low-income communities” when they don’t in fact do so, but that’s a different conversation).

The charts below thus attempt to describe what the survey results actually tell us.

Revised TFA Chart 1

Revised TFA Chart 2

Because TFA won’t share their underlying data and declined to comment on this section of my post, the charts present a likely lower bound based on some simplifying assumptions I made: namely, that TFA has usable responses for at least 70 percent of alumni in each cohort (I’m told that response rates are typically lower in earlier cohorts), that fewer than 15 percent of corps members quit TFA before completing their second year of teaching, and that these corps members who quit average 0.75 years of teaching.

We cannot assert with confidence, as the report does, that “of those alumni from cohorts having had abundant opportunity to teach at least 5 years—say, those before 2001— approximately half have done so.”  Instead, with worst-case assumptions about alumni who didn’t take the survey, we know that at least between 34 and 39 percent of alumni from these cohorts taught for that amount of time.  If we include non-alumni in our calculations, as my revised charts show and seems appropriate, the lower bound drops even further (to a little less than one third of the teachers in each corps year).  It also appears to be possible that the majority of TFA corps members teach for two years or fewer.

That doesn’t mean these lower bounds are the actual percentages – in fact, I suspect that the true percentages, while likely lower than the report’s estimates, are closer to what Miller and Perera report than to the lower bounds.  Unresponsive alumni aren’t necessarily less inclined to continue or return to teaching than those who respond to surveys.  I also imagine that some corps members who leave before becoming alumni eventually return to the classroom through other channels.

Furthermore, even if the lower bound is closer to the truth, it’s clear that a good chunk of TFA alumni have extended teaching careers.  And while TFA’s attrition is greater than teacher attrition in general (which is still pretty high), critics sometimes forget that creating lifelong teachers is not TFA’s purpose.  A key part of TFA’s mission is to expose people interested in other careers to the obstacles facing low-income kids – to turn alumni who will go on to work in other fields into lifelong advocates for disadvantaged populations (this goal can backfire – prominent alumni too often undermine social justice efforts – but I believe it is an admirable goal).

The thing is, a thorough and honest look at TFA’s statistics isn’t damning.  The organization has many problems and needs to improve, but its teachers certainly aren’t harming students.  What is damning, however, is TFA’s lack of transparency.

When I requested information on the number of respondents from each cohort, the number of total alumni from each cohort, and the number of total corps members from each cohort – explaining that I wanted to conduct the exercise above – I had a good conversation with Miller that unfortunately culminated in the following response:

I’m afraid we won’t be posting the cohort-level n’s for the main analytic sample featured in Unsung Teaching. The brief does enough to defend the findings as is. In particular, the take-away points are broad ones, spanning many cohorts. And the footnotes document the rather conservative approach. Over 85 percent of alumni from the cohorts portrayed are represented in the analytic sample. At the cohort level, the percentages vary in a pretty predictable way, but none of them is small. Hard generalizations at the cohort level would be defensible, for the most part, but we didn’t go there. The little social media exchange has totally confounded the issue of response rate on the annual survey and the n’s you seek. The reason 85 percent of alumni (from cohorts 1990 to 2010) are involved is that some of the responses come from older surveys than the 2014 one. Using some dated responses improves the generalizability, but it does bias the statistics—downward. People can dispute the value of the work, but there’s not much of an argument to make with the n’s, at least about the brief. Thus, the decision not to share.

Miller makes some good points about the interpretation of the results.  The size of the sample – over 85 percent of total alumni from the studied cohorts – is impressively large.  Additionally, Miller is right to point out that the inclusion of older survey data could bias the results downwards (an alum who had taught for only two years when she filled out the 2010 survey, for example, might have taught more in the time since despite failing to fill out another survey).  But neither of these points provides a justification for declining my request.

The data I (and others, in the social media exchange I believe Miller was referencing) asked for should not be particularly difficult to provide.  Refusing to share it just doesn’t make sense if the data is legitimate – it serves only to make people wonder if TFA has something to hide.

On some level, TFA’s decision here is part of a broader pattern of data misrepresentation.  Many of my colleagues and I observed it on a small scale at TFA’s Institute in 2010 and it’s apparent on a larger scale when TFA propagates bogus statistics about corps members’ impact. Key TFA decision-makers still seem in too many cases to care more about promoting TFA as something that “works” than about honest and accurate assessment of evidence.

Since TFA proclaims that they “welcome independent research efforts to assess [their] impact and inform the continuous improvement of [their] program,” I find these actions to be pretty hypocritical.

Because the vast majority of the TFA staff I know (including Miller) care deeply about educational equity, I also find TFA’s organizational mindset baffling.  As I’ve written before, our students “depend on us to combat misleading claims by doing our due diligence, unveiling erroneous interpretations, and ensuring that sound data and accurate statistical analyses drive decision-making.”  TFA’s disinterest in data sharing does such analyses – and thus low-income students – a disservice.

The simplest fix to these problems is for TFA to appropriately caveat alumni survey results and, more generally, to improve the way they report statistics.  TFA could also use random sampling to obtain more representative data.  We used stratified random sampling for one of our member surveys when I was on the Executive Board of the San Jose Teachers Association (SJTA) and it paid major dividends; we could for the first time speak confidently about the generalizability of our results.

Interestingly, the results from SJTA’s random sample closely resembled the results from our pure volunteer survey.  The same could conceivably be true for the findings from Miller and Perera.  It’s just impossible to know without more research.

More importantly, TFA’s misleading data presentation and lack of transparency, combined with TFA’s silence in response to thoughtful critiques in other domains, make it difficult to trust the organization.  For an alum like me who deeply respects almost everyone on TFA staff I know and who thinks TFA could potentially be a strong ally in the fight for social justice, that’s a real shame.

3 Comments

Filed under Education

Tagged as , , , , ,

by | January 10, 2015 · 9:35 am

Social Justice Unionism, Education On Tap Style

I recently discussed why teachers unions are important agents of social justice on “Education On Tap,” a Teach For America (TFA) podcast created by Aaron French. I really enjoyed the conversation – French goes above and beyond his promise to make the show “a little bit of fun” – and appreciated TFA’s continued (though still very young) efforts to deconstruct myths about organized labor and education reform ideas.

You can listen to the podcast below:

A couple additional details on two of the topics we discussed:

1) How we refer to education stakeholders: We often use the phrase “reformer” to describe people on one “side” of the education debate. As Nick Kilstein explains, we typically think “reformers” do the following:

1. Support market forces including choice and competition as a mechanism to improve all schools. This is usually done through vouchers and charter schools.

2. Support business practices including evaluation, promotion and merit pay to motivate and attract teachers

3. Hold that teachers and schools should be accountable for student achievement, usually measured by standardized testing

4. Support alternate paths to the classroom through programs like Teach For America

5. Affiliate themselves with no-excuses charter schools

However, neither French nor I (nor Kilstein) are crazy about this term, for a few reasons. First, the group of people we call “reformers” sometimes have drastically different views on these topics. For example, opinions about the appropriateness of suspending students vary widely among people who support the rapid expansion of charter schools. Because “reformers” don’t hold monolithic views, it doesn’t make a ton of sense to lump them all into the same category.

Second, using the term “reformers” erroneously suggests that only a certain group of people support school improvements. However, teachers in unions and other critics of typical education reform efforts fight for school reforms themselves; they just have a different (and, on balance, more evidence-based and theoretically sound) perspective about which reforms we should pursue on behalf of students in low-income communities. Despite misleading claims to the contrary, very few people actually support the “status quo” in education. Though the word has become associated with negative imagery for a lot of education stakeholders, nearly everyone is a “reformer” to some extent.

Third, the use of a term like “reformers” reinforces the notion that there are two polarized “sides” in education debates, the “reformers” and their opponents. As I discussed with French, I believe the “sides” are much less in opposition than they sometimes appear to be, and that most people in education are in general agreement on the vast majority of issues. The more we can deconstruct the notion of “sides,” the better.

That said, I don’t have a great solution to either the first or third problems (for the second, I’d recommend that we use clunkier phrases more like “proponents of market-driven reforms to education” and “advocates for a comprehensive social justice approach to education policy” when we can). Categories can be useful for brevity’s sake, and as is evident below, it’s hard to construct an argument while avoiding categorization altogether. Still, I think it’s worth reflecting on our naming conventions as we endeavor to be more nuanced.

2) Why unions are power-balancing advocates for low-income kids: French explained during our discussion that many people believe the San Jose Teachers Association (SJTA, the local union for which I served as an Executive Board member from 2012 to 2014) to be an atypically progressive union. In reality (and I believe French agrees), the vast majority of unions, including national teachers unions like the National Education Association (NEA) and American Federation of Teachers (AFT), are some of the most power-balancing institutions out there.

Recent research by Martin Gilens confirms this fact: unions consistently advocate on behalf of less advantaged populations on a wide range of social justice issues. They serve as an important counterbalance to wealthy interests and exploitative policies, and have made extremely important gains for working Americans throughout their history. It’s probably not a coincidence that the steep decline in unionization over the past thirty years has coincided with a steep increase in earnings, income, and wealth inequality.

That doesn’t mean unions can’t be wrong on certain issues. We should absolutely condemn the behavior of police unions that defend racist positions, for example, and demand that they be held accountable and change. Teachers unions shouldn’t be immune from criticism, either, and it’s imperative that we confront them when we believe their positions are misguided. Not all teachers unions have realized their potential as social justice unions just yet, and while I firmly believe that a different approach from the education community would help more of them do so, organized labor must also proactively analyze and revise practices that don’t fit its mission.

Yet we must also remember that teachers unions have very strong track records on behalf of low- and moderate-income families, and more credibility as advocates for low-income kids than many of the people and organizations who malign unions. Even if you think certain teachers unions are wrong about aspects of education policy, it’s completely inaccurate to argue that their existence harms low-income kids. The empirical evidence is clear (much clearer, in general, than the evidence about education policy ideas) that teachers unions are a major net positive for low-income populations.

There’s joint responsibility to change the tone of education conversations, and union members must avoid becoming reflexively defensive when confronted with criticism. We do ourselves and our students a disservice when we react by ignoring people outright or slinging insults right back; instead, we should try to understand the legitimate elements of critiques, address them, and educate people on where they’re wrong and how to have more productive dialogue.

At the same time, union members and leaders are understandably offended when proponents of market-driven reforms (making an attempt!) imply that union opposition to these reforms is borne of laziness, selfishness, and/or incompetence. Everyone needs to remember that teachers in unions, who are directly student-facing and who will actually implement education reform ideas, typically have good ideas about what students need, and that both private and public sector unions are important advocates for low-income people in general. While there is some shared responsibility, the tone of the debate cannot change until proponents of market-driven reforms acknowledge these facts. The sooner anti-union messaging becomes a thing of education reform conversations past, the sooner we can collaboratively develop great policies for students.

A big thank you to French for having me on the show, and hope you enjoy the podcast!

1 Comment

Filed under Education, Labor

Tagged as , , , , ,

by | November 2, 2014 · 7:07 pm

TFA, CTA, and What It Means to Be a Union

A former instructional coach and one of only five people selected nationwide as a 2012 recipient of the Horace Mann Award for Teaching Excellence, Jen Thomas is now President of the San Jose Teachers Association (SJTA).  In this post, also destined for the next issue of the California Educator, Jen discusses the California Teachers Association’s (CTA’s) recent cover story about Teach For America (TFA) and the responsibility that comes with being part of a union.

SJTA President Jen Thomas

Jen Thomas

Like any president would be, I was delighted when I received the October edition of the California Educator and saw one of San Jose TA’s members smiling from the cover. Clinton Loo was not only a very talented math teacher, but a member of our local’s governing body: he spent the 2013-2014 school year as our Secretary-Treasurer.

My excitement turned quickly to concern, though, when I saw the title of the article in which Clinton was featured: “Teach for America: Do-gooders or school Rhee-formers?” My concern was the rhetorical choice this framing implied. My colleagues and friends from TFA are either “do-gooders” with the saccharine naiveté that implies, or agents of Michelle Rhee and her intolerable demagoguery.

The October issue of California Educator, featuring TFA alum and former SJTA Secretary-Treasurer Clinton Loo on the cover.

The October issue of California Educator, featuring TFA alum and former SJTA Secretary-Treasurer Clinton Loo on the cover.  

As CTA, this article highlights two serious problems: inadvertently undermining our union brothers and sisters who came to us from the TFA program, and not resolving the problems generated by the organization.

1) CTA members who come from Teach for America should feel that they are as valued and supported as any other teacher entering the classroom.  First and foremost, a teacher is our colleague. We must be united in support of one another, and that starts with being extremely careful with how we frame important questions about the changing political landscape in our profession when these questions can lead to division in our ranks.

2) What are we doing about these issues and are they unique to Teach for America members?

  • High TFA turnover is an issue, but about 50% of all teachers leave in their first five years, driven out by workload, wage stagnation, and the abject failure of our society to prioritize education.  Many TFA corps members stay in San Jose for long past their two-year mandate and often they leave for the same reason any teacher leaves: The job is entirely unsustainable. Our compassion for that should be where we anchor this conversation.
  • No, five weeks training is not enough time to make a quality educator. We’ve also seen teacher training programs of a year or even two years that do not produce teachers ready to face the real strains and struggles of the classroom.  Poor preparation puts a terrible burden on our system; what are we going to do about it?
  • That TFA members don’t become actively involved in the union because they see themselves as education transients is a broad statement and contradicted by our experience in San Jose. Perhaps we are unique, but TFA corps members and alumni don’t deserve to all be painted with the same brush.
  • Where’s our plan to be as strong as Leadership for Educational Equity? Let’s build on our political strength and create a powerful support and training program to elect public officials from the teaching ranks.

Issues of training, policy, and politics; issues of values, arrogance, and teaching as a hobby – all of these are valid and worth a discussion aimed at remedy rather than rhetoric. In the meantime, every CTA member past and present – regardless of how they came to the classroom – should believe that we are united together in support of the work we do for our students, our colleagues, our communities, and our futures.

That’s what it means to be a union.

10 Comments

Filed under Education, Labor

Tagged as , , , , , , , , ,

by | April 25, 2014 · 7:06 pm

Teachers Unions: What We Do and How Students Benefit

The National Labor Relations Board (NLRB) recognized the Northwestern University football team’s right to unionize in February and the players just held a unionization vote. Quarterback Kain Colter began the program’s union movement primarily to address player safety, but supporters of the team’s efforts believe unionization will improve college athletes’ experiences across the board. Northwestern management, however, opposes granting a collective voice to their underlings; the university tried to convince players not to unionize and has appealed the NLRB’s ruling. The NCAA, meanwhile, has begun a fear-mongering campaign to obfuscate the plethora of issues with the way it conducts business, issues that a union can help the players address.

If that story sounds familiar, it’s because the NCAA’s behavior in this case resembles that of wealthy interests in most other industries. Misinformation about union purpose and impact abounds in education especially; prominent education reformers have successfully hoodwinked large swaths of the intelligent public into believing that teachers unions undermine student interests. I am often taken aback by the inaccurate, negative comments about teachers unions I still hear from otherwise well-meaning members of charter school networks, education advocacy groups, and Teach For America (TFA).

I am encouraged, however, by the initial efforts taken by TFA and Leadership for Educational Equity (LEE) staff in the Bay Area to debunk member misconceptions about teachers unions. A few weeks ago, LEE invited me to speak to a group of current corps members, alumni, and LEE and TFA staff at an event called “Unions Matter.” During the event, I described the difference between social justice unionism and industrial unionism and laid out five important roles that unions play:

1) The traditional union role – Most people are familiar with this category of union activity; it covers salary, benefits, working conditions, and grievances. Anti-labor interests often denigrate teachers unions that focus on this “industrial” role, arguing that it has little to do with student interests, but they’re wrong for three primary reasons. First, there’s an extremely high correlation between good working conditions for teachers and good learning conditions for students. Unions that advocate for adequate classroom resources, a sustainable work day, and functioning air conditioning systems do so as much for their students as for their members. Second, the families of many students in low-income communities benefit significantly from the strength of the organized labor movement. The ability of unions to collectively bargain for fair wages and benefits is essential for the well-being of low-wage workers who are frequently exploited by their employers. Third, people mimic what we do more than what we say. If we want teachers to inspire their students to take collective action and advocate for themselves, district leadership needs to model that approach with teachers.

2) Collective voice – Unions provide a forum for educators to band together, prioritize action items, and communicate with management about those items. For example, the San Jose Teachers Association (SJTA) has helped teachers identify professional development and resource needs during the rollout of the Common Core State Standards, and San Jose Unified School District (SJUSD) has worked hard to respond to teachers’ collectively expressed requests. When district leadership unintentionally overlooks the impact initiatives have on students’ classroom experience, union members can use their collective voice and collaborate with administration to quickly resolve problems. Unions also foster a sense of community among members, connecting teachers across the district and thus building school and district culture.

3) Community and family outreach – SJTA helps coordinate teachers during Read Across America, sponsors little league baseball teams, works with parents from community service organizations like Sacred Heart, partners with Vision to Learn to bring eye doctors and prescription eye glasses to students who might not otherwise have them, and awards scholarships to aspiring teachers in our high schools. Unions can, should, and often do engage parents and advocate for students and public education at community events.

4) Political advocacy – School boards, other elected officials, and ballot initiatives matter significantly for students, and unions work hard to elect candidates and pass propositions that will positively impact kids’ lives. Without the efforts of the California Teachers Association and local California teachers associations in 2012, Proposition 30 would likely have failed, an outcome that would have resulted in a significantly shorter school year, increased class sizes, layoffs, a reduction in extracurricular programs, and/or a reduction in elective offerings in most California schools. SJTA also helped put two excellent SJUSD school board members in office in 2012. In addition to education-specific issues, teachers unions can advocate for a broader set of social justice policies that make a difference in our students’ lives; that purpose explains why SJTA joined the South Bay Labor Council in supporting San Jose’s minimum wage increase in 2012 and why we consider endorsements for the Santa Clara County Board of Supervisors and the San Jose City Council.

5) Education reform – Though typical uses of the phrase “ed reform” conjure anti-labor images, teachers unions can and often do drive smart, ethical modifications to education policy that improve opportunities for teacher satisfaction and student learning. SJTA and SJUSD recently co-developed a new teacher evaluation system (see Article 16000 of our contract) that, though not yet fully implemented, uses multiple measures of effectiveness to help teachers of all skill levels grow professionally, requires multiple evaluators for both formal and informal observations, and grants joint control of the process to teachers and administrators. We are hoping California will grant our request to either shorten or lengthen permanent status timelines when doing so is in the joint interests of students, teachers, and the school. Our contract also allows for new teacher leadership pathways (although we currently lack the funding necessary to implement our Model Teacher and Master Teacher Leader positions). We co-developed several other teacher-empowering, student-centered policy decisions with our school district and other unions can and often try to do so as well.

The current and former teachers I talked to at the LEE event were, as most teachers I encounter from both TFA and other programs are, thoughtful, intelligent, and passionate about improving the lives of low-income kids. They astutely noted that they don’t see all of these roles pursued by their unions all the time and wondered how SJTA became so proactive. Their question is a great one, and while I haven’t been involved in SJTA long enough to see the process unfold firsthand, I believe I can lend some insight.

Throughout history, labor-management relationships have typically involved some combination of management withholding information, misappropriating money, imposing unreasonable working conditions, and lying to the media about the effects of negotiations and employee objectives (the NCAA is currently engaged in all of this behavior in its attempt to prevent the Northwestern football team’s attempt to unionize). It’s important to note that, anytime one perceives intransigence from a teachers union, that intransigence is typically in response to irresponsible and/or unethical behavior from the school district. The district and union have a joint obligation to behave responsibly, but unions are the less powerful entity in the union-district relationship and the onus is therefore more on districts to create the conditions – transparency, openness to union ideas, respect for union membership, and a willingness to work together – under which a union can adopt the social justice approach described above. SJTA can function as we do in large part because SJUSD has demonstrated its commitment to honest, collaborative negotiations and messaging. Most seemingly obstinate union positions, on the other hand, arise in response to corrupt and/or incompetent management decision-making processes.

That said, unions must also work proactively to define themselves as social justice organizations. I believe establishing a positive mission statement (SJTA’s is to “empower teachers to educate, inspire, and change lives through public education”) can go a long way. We should try to develop contract structures, like salary formulas (see Appendix A of our contract), that enable us to spend a smaller percentage of collective bargaining time on salary and benefits. We must also consider innovative ideas that have a compelling rationale and research base behind them.

It’s important to remember that members of teachers unions work directly with students every day – we are students’ most credible advocates. We care deeply about educational equity and the learning that takes place in our classrooms. Education reformers who are likewise passionate about helping students succeed will therefore stop bashing unions and start working with us to develop the intelligent, ethical policies that can benefit students most.

6 Comments

Filed under Education, Labor

Tagged as , , , , , , , ,

by | December 25, 2013 · 9:14 pm

Approaching Education Data the Nate Silver Way

My girlfriend’s very hospitable and generous family gave me some great gifts for the holidays when I stayed with them in upstate New York.  As I rocked my new Teach For America T-shirt in the Rochester airport on Christmas Eve, my cursory overview of Nate Silver’s new book, The Signal and the Noise, inspired me to write this post.

While most people probably know Silver for his election predictions and designation in 2009 as one of the world’s 100 Most Influential People, Silver has been my baseball stat guru for considerably longer than he’s been doing political analysis.  In one of my favorite books of all time, Baseball Between the Numbers, Silver penned a brilliant examination of clutch hitting that I still quote at least four or five times a year.  I have generally found Silver’s arguments compelling not just because of his statistical brilliance, but also because of his high standards for data collection and analysis, evident in the following passage from the introduction of his book:

The numbers have no way of speaking for themselves.  We speak for them.  We imbue them with meaning…[W]e may construe them in self-serving ways that are detached from their objective reality…Before we demand more of our data, we need to demand more of ourselves.

In few fields are Silver’s words as relevant as education.  While the phrase “data-driven” has become ubiquitous in discussions of school reform and high-quality instruction, most people discussing education have very little understanding of what the statistics actually say.  As I’ve written before, many studies that reformers reference to push their policy agendas are methodologically unsound, and many more have findings very different than the summaries that make it into the news.

It’s hard to know how many reformers just don’t understand statistics, how many fall victim to confirmation bias, and how many intentionally mislead people.  But no matter the reason for their errors, those of us who care about student outcomes have a responsibility to identify statistical misinterpretation and manipulation and correct it.  Policy changes based on bad data and shoddy analyses won’t help (and will quite possibly harm) low-income students.

Fortunately, I believe one simple practice can help us identify truth in education research: read the full text of education research articles.

Yes, reading the full text of academic research papers can be time consuming and mind-numbingly dull at times, but reading articles’ full text is vitally important if you want to understand research findings.  Sound bites on education studies rarely provide accurate information.  In a Facebook comment following my most recent post about TFA, a former classmate of mine referenced a 2011 study by Raj Chetty to argue that we can’t blame the achievement gap on poverty.  “If you leave a low value-added teacher in your school for 10 years, rather than replacing him with an average teacher, you are hypothetically talking about $2.5 million in lost income,” claims one of the co-authors of the study in a New York Times article.  Sounds impressive.  Look under the hood, however, and we find that, even assuming the study’s methodology is foolproof (it isn’t), the actual evidence can at best show an average difference of $182 in the annual salaries of 28-year-olds.

As I’ve mentioned before, there’s also a poor statistical basis for linking student results on standardized test scores to teacher evaluation systems.  Otherwise useful results can give readers the wrong impression when they gloss over or omit this fact, a point underscored by a recent article describing an analysis of IMPACT (the D.C. Public Schools teacher evaluation system).  The full text of the study provides strong evidence that the success of D.C.’s system thus far has been achieved despite a lack of variation in standardized test score results among teachers in different effectiveness categories.  Instead, the successes of the D.C. evaluation system are driven by programs teachers unions frequently support, programs like robust and meaningful classroom observations that more accurately measure teacher effectiveness.

Policymakers have misled the public with PISA data as well.  In a recent interview with MSNBC’s Chris Hayes, Michelle Rhee made the oft-repeated claim that U.S. schools are failing because American students, in aggregate, score lower on international tests than their peers in other countries.  Yet, as Hayes pointed out, it is abundantly clear from a more thorough analysis that poverty explains the PISA results much better than school quality, not least because poor US students have been doing better on international tests than poor students elsewhere for several years.

I would, in general, recommend skepticism when reading articles on education, but I’d recommend skepticism in particular when someone offers a statistic suggesting that school-related changes can solve the achievement gap.  Education research’s only clear conclusion right now is that poverty explains the majority of student outcomes.  The full text of Chetty’s most recent study defending value-added models acknowledges that “differences in teacher quality are not the primary reason that high SES students currently do much better than their low SES peers” and that “differences in [kinder through eighth grade] teacher quality account for only…7% of the test score differences” between low- and high-income schools.  In fact, that more recent study performs a hypothetical experiment in which the lowest-performing low-income students receive the “best” teachers and the highest-performing affluent students receive the “worst” teachers from kinder through eighth grade and concludes that the affluent students would still outperform the poor students on average (albeit by a much smaller margin).  Hayes made the same point to Rhee that I made in my last post: because student achievement is influenced significantly more by poverty than by schools, discussions about how to meet our students’ needs must address income inequality in addition to evidence-based school reforms.  We can’t be advocates for poor students and exclude policies that address poverty from our recommendations.

When deciding which school-based recommendations to make, we must remember that writers and policymakers all too often misunderstand education research.  Many reformers selectively highlight decontextualized research that supports their already-formed opinions.  Our students, on the other hand, depend on us to combat misleading claims by doing our due diligence, unveiling erroneous interpretations, and ensuring that sound data and accurate statistical analyses drive decision-making. They rely on us to adopt Nate Silver’s approach to baseball statistics: continuously ask questions, keep an open mind about potential answers, and conduct thorough statistical analyses to better understand reality.  They rely on us to distinguish statistical significance from real-world relevance.  As Silver writes about data in the information age more generally, education research “will produce progress – eventually.  How quickly it does, and whether we regress in the meantime, depends on us.”

Update: Gary Rubinstein and Bruce Baker (thanks for the heads up, Demian Godon) have similar orientations to education research – while we don’t always agree, I appreciate their approach to statistical analysis.

Update 2 (6/8/14): Matthew Di Carlo is an excellent read for anyone interested in thoughtful analysis of educational issues.

Update 3 (7/8/14): The Raj Chetty study linked above seems to have been modified – the pieces I quoted have disappeared.  Not sure when that happened, or why, but I’d love to hear an explanation from the authors and see a link to the original.

7 Comments

Filed under Education

Tagged as , , , , , , , , , , , , ,

by | November 8, 2013 · 8:34 am

Working Together for Educational Equity: What’s Missing from the TFA Debate

Teach For America (TFA) articles are all the rage right now.  Over the past month and a half, the four articles linked below have received particular attention:

“I Quit Teach for[sic] America” by Olivia Blanchard (The Atlantic, September 23)

“Remember the ‘I Quit Teach for[sic] America’ essay?  Here’s the counterpoint. ‘I stayed.’” by Maureen Downey and Tre Tennyson (The Atlanta-Journal Constitution, October 3)

“Why I Stopped Writing Recommendation Letters for Teach for[sic] America” by Catherine Michna (Slate, October 9)

“I Almost Quit Teach for[sic] America” by Eleanor Barkhorn (The Atlantic, October 14)

Though I generally hesitate to suggest that truth lies somewhere towards the middle of two extremes, the majority of both pro- and anti- TFA articles in this case contain inaccurate claims and arguments that unnecessarily pit people with the same goals against each other.  This post is my attempt to debunk the inaccuracies presented in these articles and identify the true benefits and drawbacks of TFA.  I also hope to identify how TFA and opponents of TFA can find common ground in their work for educational equity.

Before I make those arguments, a little bit about my educational background: I attended a traditional public school in a working class, mostly white neighborhood in southern New Jersey from first grade through sixth grade.  From seventh grade to twelfth grade, my parents sent me to Moorestown Friends School (MFS), a high-performing private Quaker school twenty-five minutes from my house.  I moved across the country to attend Stanford University for college and joined TFA right afterwards. San Jose Unified School District (SJUSD) paid TFA a few thousand dollars to hire me to teach at San Jose Community Day School (SJCDS), a school for students expelled from other schools for drug, weapon, violent, or other behavioral offenses.  I taught at SJCDS for three years, the second year of which I served as my school’s Site Representative for the San Jose Teachers Association (SJTA), the union that represents around 1,700 professional educators in SJUSD.  In my third year at SJCDS, I was appointed to the SJTA Executive Board.  I still serve as the SJTA Outreach Director in my new role as an instructional coach in SJUSD and also run professional development sessions for first and second year TFA corps members in San Jose.  I feel connected to both SJTA and TFA, though I tend to hear more compelling arguments from my colleagues at SJTA than I hear from the TFA staff members I know.  I hope you find this context valuable as I address the claims either directly made or implied in the above articles by answering the questions below:

Are TFA teachers prepared for their teaching assignments?

The short answer to this question is no.  As both Blanchard’s and Barkhorn’s articles note, TFA’s summer Institute, besides being short and often unrelated to a teacher’s upcoming teaching assignment, focuses far too much on theory and vision and far too little on tangible skills.  However, criticisms of TFA along these lines are, as another alum puts it, “a moot point” – nobody does a particularly good job preparing first-year teachers for assignments in low-income neighborhoods.  As I mentioned in an earlier blog post, nearly all the evidence suggests that there is very little, if any, difference, on average, between the standardized test results of students who have had TFA teachers and students who have had teachers with different backgrounds.  One of my friends, fellow 2010 TFA alum Connor O’Steen, summarized the problem with the “lack of preparation” critique in response to that post on Facebook:

…[W]hat does it mean when (at least) two years and 40-50,000 dollars of ed school has you performing ever so slightly worse on average than someone who’s done a six week crash course over a summer? Certainly you’d expect ed school–this long and formal educational experience which usually culminates in a Master’s degree–to add more value, more human capital? I think a lot of the criticism of TFA comes from stakeholders in the traditional ed pipeline who are made genuinely uncomfortable by the fact that all the training and apprenticeships seem to put people solely on par with beginning TFA corps members. Granted, there are more ways to measure achievement than standardized tests, but I don’t think many people would see percentile scores this low and think there’s *not* a problem here.

While I know several teachers (both within and outside of TFA) who believe their training contributed value to their teaching, I know many more, from a variety of preparation programs, who believe their training was practically useless.  Studies suggest corps members and other teachers have similar attitudes about their preparation and there’s no escaping the fact that there’s no well-established statistical correlation between time spent in a teacher preparation program and teacher effectiveness.  I proposed three possible explanations for this fact in my response to Connor’s post:

1. TFA and traditional teacher education systems are similarly ineffective at preparing teachers for placements in low-income schools.
2. TFA is less effective than traditional teacher education systems at training teachers but recruits better “talent” on average than those programs. One of the more interesting findings from the Mathematica study was the lack of correlation between a teacher’s undergraduate background and student achievement. But Dana Goldstein has an alternate theory (http://www.danagoldstein.net/…) that work ethic, a strong orientation to a mission, and intense focus on data and testing all explain the results.
3. TFA is more effective than traditional teacher education systems at training teachers but traditional teacher education systems recruit better “talent” than TFA. There are few people who make this argument.

Whichever of the above three options is most accurate, it’s hard to indict TFA for putting poorly prepared teachers in schools unless you indict every single teacher preparation program for the same fault.  I actually believe both traditional teacher preparation programs and TFA’s program (which is very similar in content to traditional programs) could improve significantly, but my point is that this critique is not valid when used to compare TFA to other programs (the only exception to this rule may be special education.  As one member of the SJTA Board pointed out to me, TFA teachers typically lack the legal knowledge necessary to succeed as special educators.  While two of the best special education teachers I know in SJUSD are TFA alums who have remained in the classroom well after their TFA commitment expired, I think that argument is valid).

Does the relatively short two-year commitment negatively impact students?

Most studies suggest that common sense is correct and teacher turnover is bad for students.  Though TFA placement regions have high turnover rates for first and second year teachers in general, attrition rates for TFA corps members are in the same ballpark during those two years and are significantly greater in subsequent years.  I personally believe TFA should not recommend corps members for positions for which there are other qualified candidates more likely to remain in education long-term.  In SJUSD, for example, TFA has placed a number of corps members at schools that are relatively low-poverty and easy-to-staff, which seems antithetical to the TFA mission.

At the same time, and contrary to Michna’s claims, extremely hard-to-staff positions with high turnover rates exist.  Also in SJUSD, which I believe to be one of the best large urban school districts in the country, we still have several open positions and are nearly three months into the school year.  TFA focuses primarily on these hard-to-staff positions and explicitly tries to select people unlikely to quit on commitments (they obviously failed in the case of Blanchard, but I think they’re pretty justified in excoriating her for her decision; TFA asks applicants outright in the final interview if they would quit under any circumstances and I find it hard to believe she answered this question honestly).  TFA in many places effectively addresses teacher shortages.

Do TFA teachers, on average, help level the playing field for children in low-income communities (do TFA teachers close the achievement gap)?

The short answer to this question is also no; as I mentioned above and discussed in an earlier post, TFA teachers seem to guide students to roughly equivalent standardized test results as all other teachers.  Those results are overwhelmingly poor compared to the results for affluent students.

Tennyson states in his article that, in his first year, “100 percent of [his] students passed the ELA exam and 90 percent were proficient or above in reading.  Down the hall, Donna Jenkins, the third corps member at [his] school, led [her] fifth graders to a 95 percent pass rate in math and 97 percent in science.”  These numbers sound great, but there are several possible explanations for them.  While it’s certainly possible that Tennyson and Jenkins were two of the best teachers in America during their first years of teaching and were able to single-handedly change the academic trajectories of their students in one year, I think it’s more likely that these statistics are misleading.  Perhaps their students weren’t all that disadvantaged before fifth grade.  Perhaps some out-of-school factors were contributing to the success of these students.  Perhaps these teacher-designed assessments don’t tell the whole story of student performance.  I’d bet a fair amount of money that Tennyson was at least a pretty good teacher based on what he wrote, but I’d bet even more money that the results he lists have a lot less to do with excellent teaching than he makes it sound.  We’d have to see his tests and get significantly more context and data about his students and classroom to know for sure, but while I’m sure he genuinely believes he can teach kids out of poverty, nearly all the externally verified data we have suggests that’s highly unlikely (again, check out my previous post here for a summary of research findings).  Even if Tennyson and Jenkins did work miracles with their students, they’d be incredibly unique within TFA.  There’s no reason to believe their success would be replicable on a large scale because, if it were, TFA would be teaching their best practices to all new teachers and getting results better than what they’re getting.

Again, none of that is to say TFA teachers (and other teachers, for that matter) can’t make a difference and change students’ lives – they definitely can and I know a number of people who were very good teachers as corps members – but many TFA teachers, like many charter networks, have a tendency to overstate their impact.  In the case of individual teachers, I’m inclined to believe the misleading information they present is unintentional, though I am less predisposed to think that misinformation coming from organizational leadership is so innocuous.

How do TFA’s leadership development, political work and alignment, and brand affect low-income students?

TFA’s mission includes developing leaders who work “to ensure that all children can receive an excellent education” outside of the classroom.  Critics sometimes forget this purpose.  The hope is that even people who join TFA solely to build their resumes will see the obstacles low-income children face during their two-year stints in the corps and will then advocate for those children long after they have left the teaching profession for their careers in law, medicine, or business.  I believe this goal is admirable.  At the same time, however, TFA’s brand often develops leaders and political outcomes that actively harm students in poverty.

Blanchard formulates a pretty accurate summary of the problem.  Pervading TFA is

…the unspoken logic that current, non-TFA teachers and schools are failing at the task of closing the achievement gap, through some combination of apathy or incompetence. Although TFA seminars and presentations never explicitly accuse educators of either, the implication is strong within the program’s very structure: recruit high-achieving college students, train them over the summer, and send them into America’s lowest-performing schools to make things right. The subtext is clear: Only you can fix what others have screwed up.

Her analysis gels with my TFA experience – most people within TFA are hesitant to explicitly blame the achievement gap on bad teachers and schools, but most also perpetuate a negative narrative about public education at least implicitly.  When Blanchard asked a TFA spokesperson about TFA’s views on traditionally trained teachers, she received the response that “[i]f anything, teachers are victims of more structural problems: inequitable funding; inadequate systems of training and supporting teachers; the absence of strong school and district leadership.”  Notice that this response still implies that teachers aren’t doing a very good job; it just blames the problem on inequitable school funding, poor training, and bad leadership instead of laying the proximate culpability at teachers’ feet.  I really like nearly everyone I know on TFA staff, but I have never gotten a single one of them to admit the well-established fact that in-school factors explain, at most, 33% of student achievement.

This mindset – that teachers and schools have nearly total control over student outcomes – has two really problematic implications.  The first implication is that schools that serve low-performing students are bad schools, that some combination of the teachers and leadership at those schools are doing a terrible job that someone else could do significantly better and mass firings and closings are warranted.  The second implication is that we can focus our political energy away from solving poverty directly; if education can fix poverty, as Teach For America suggests, poor children can succeed without a drastic overhaul of society.  School-based reforms are all we need.  The reality, though, is that education cannot solve society’s problems.  Education can make a difference, but the main reasons low-income students perform poorly compared to their affluent peers have nothing to do with school and everything to do with the gamut of obstacles they face from birth.  When you break down school performance in the US by free and reduced-price lunch rate before comparing it to school performance internationally, “low-performing” US schools with high numbers of poor students have higher test scores than schools in countries with similar concentrations of disadvantage.

The best critique of Teach For America, in my opinion, is based on political affiliations and impact.  The organization produces a large number of influential alumni who support the expansion of charter schools, changes to teacher employment law, and making student standardized test scores increasingly more important in teacher and school evaluations.  There is, unfortunately, very little evidence that these reforms help poor students.

Yet a lot of politicians who couldn’t care less about poor kids rally around TFA’s “unspoken logic.”  Chris Christie, the governor of New Jersey, is a prime example.  Christie uses the cover of an education “reform” agenda – he promotes closing schools, opening more charters, eliminating tenure, and introducing “merit pay” based on student test score data – to hide the fact that tax cuts for the rich are a higher priority for him than poor students eating breakfast or lunch (see this link for a more extensive list of Christie’s cuts to education).  TFA obviously doesn’t support cutting school breakfast money, but the concept that educator and school-related changes are most important for poor students enables people like Christie to further disadvantage low-income kids, bust unions, enrich the wealthy even further, and receive credit for supposedly student-oriented ideas at the same time.

How can TFA, teachers unions, and other proponents of opportunities for low-income children work together for educational equity?

In the end, most people within Teach For America and most other people working in education have very similar goals; to use the words of the San Jose Teachers Association, most of us want to “educate, inspire, and change lives through public education.”  As I recently discussed with my older sister, the biggest shame about the TFA debate is that, while people who care about kids are arguing with each other about teacher and school quality, people like Christie are exacerbating poverty and directly destroying the lives of low-income students.

So what should TFA and people like Michna and Blanchard do differently to better support their stated missions?

First, and most importantly, TFA should acknowledge that the achievement gap is caused by poverty, not by bad teachers and schools.  School-related changes alone can address only some of poverty’s symptoms.   TFA should thus publicly advocate for policies that address poverty, policies like single-payer health care, increased taxes on the wealthy, wraparound services for low-income kids, and more environmentally and socially responsible food standards.  This advocacy will lose TFA money – I highly doubt Arthur Rock and many of Teach For America’s “National Corporate Partners, Sponsors, Supporters, and Investors” will continue to support the organization if TFA begins to promote reducing income inequality – but if TFA is really “students first,” TFA will worry about that funding later and start working now for the change most likely to actually benefit poor students.  Quality teaching matters, but what matters more is the overall environment in which the student grows up and lives.

Second, everyone in education should promote further research on the link between various reform ideas and student outcomes.  Until other reform ideas are supported by strong evidence, however, we should focus on the school-related change everyone agrees about: teacher support.  Though more study and experimentation is needed, research suggests that teachers can benefit greatly from ongoing professional development in the form of one-on-one coaching.  TFA already has a structure for coaching corps members and, when it comes to TFA teachers, believes in development instead of dismissal.  Many traditional school districts, like SJUSD, have coaching models as well for the same purpose.  I believe directing energy and policy focus towards making these systems more effective and aligned with this purpose should be the primary goal of education reform.  Focusing on evidence-based support first and other evidence-based reforms second is both the ethical way to treat the teaching workforce and a way to encourage the development of strong teachers interested in remaining in the profession.

At the same time, educators must consider additional reforms pending future research.  While student test scores, for example, are not yet a valid or reliable indicator of effectiveness, we should continue to study them.  Teachers unions can get behind that idea; unions only oppose linking test scores to teacher evaluations because doing so currently provides an inaccurate picture of a teacher’s effectiveness.  Unions believe in robust evaluation systems that more accurately assess teachers’ contributions.  If empowered by a change in the education narrative and given adequate support, I also believe the vast majority of teachers would buy into respectful, evidence-based discussions about revised layoff procedures and expedited dismissal processes for the small fraction of teachers not doing their jobs.  Those discussions present a problem now mainly because reformers like Michelle Rhee continue to promote unproven reforms and focus on teacher blame and dismissal rather than substantive, constructive criticism and support.

In general, critics of TFA should stop harping on illegitimate complaints about TFA teachers’ lack of preparedness.  A lot of TFA teachers turn out to be very good teachers, even in their first years, and targeting well-intentioned, hardworking, and talented individuals for the problems of the larger organization is counterproductive.  Teachers should also remember that we do make a difference – though we can’t close the achievement gap, we can markedly improve our students’ lives.  And TFA should stop sending the sometimes explicit and frequently implicit message to its corps members and the general public that educational changes alone can fix poverty, since they can’t.

All educational stakeholders should be able to agree that we must continuously improve our schools and practices to better serve our students.  But to truly put our low-income kids first, TFA and other stakeholders must simultaneously band together with teachers unions and advocate for social justice policies that address economic inequality.

Note: Thanks to Jack Schneider, this post was updated to include the most recent data on teacher attitudes about their preparation programs.

Update 2 (2/21/14): The second-to-last paragraph of this piece originally referred to critics of TFA as “the anti-reform crowd.”  This reference has been changed because of a thoughtful comment by Serge Vartanov.

Update 3 (3/2/14): The text above originally included a parenthetical aside that referenced a flawed study on teacher preparation programs.  Thank you to Demian Godon for prompting me to reexamine it.

Update 4 (9/26/15): In reading back through this post, I realized that the text originally said the following:

“When you break down school performance in the US by poverty rate before comparing it to school performance internationally, ‘low-performing’ US schools with high poverty rates do better than schools in every other country with similar rates.”

The link, however, does not break down US schools by the official poverty rate, but by the percent of students who receive free and reduced-price lunch.  I have updated the text to more accurately reflect this fact, though it’s worth noting that the official poverty rate in the US is very low and that the percent of students receiving free and reduced-price lunches is probably a better proxy for disadvantage.

11 Comments

Filed under Education

Tagged as , , , , , , , , , , , , ,

by | September 16, 2013 · 10:29 pm

TFA Effectiveness

Several people have sent me articles discussing Mathematica’s recent research study that examined Teach For America (TFA) math teacher effectiveness.  This study is significant because, to my knowledge, it is the first large-scale study on TFA to randomly assign students to classrooms.  Its experimental design provides fairly convincing support for the idea that TFA teachers’ students perform no worse than the students of non-TFA teachers at TFA placement schools.  But this finding is consistent with the findings of previous research and does not support the assertion that Teach For America teachers can close the achievement gap.

Anyone who continues to argue that TFA teachers yield worse educational outcomes than other teachers (generally citing pretty old research and ignoring the much larger body of research that contradicts that claim) are just plain wrong.  While there are some methodological concerns with recent studies, enough evidence exists for me to state confidently that TFA teachers, on average, do not harm student achievement.  At the same time, TFA and its proponents must also stop using misleading data and insisting that studies like this one prove more than they actually do.  Despite articles’ claims, the new Mathematica research does not suggest that TFA teachers’ students outperform non-TFA teachers’ students in a meaningful way.

The study showed a difference between TFA teachers and all comparison teachers of 7% of one standard deviation.  To put that number in context, a difference of 7% of one standard deviation in home runs between two baseball players in 2012 would be a difference of less than one home run over the course of the entire 162-game season.  Or, if you aren’t a baseball fan, a difference of 7% of one standard deviation between two students on the math section of the SAT in 2012 was equivalent to a difference of less than one correctly answered question.  The authors of the Mathematica study and just about every article quoting the study claim 7% of one standard deviation in this context is equivalent to 2.6 months of learning, using this 2007 research paper as justification, but that number is invalid and based on an inappropriately applied heuristic.  The average student in a non-TFA classroom scored in the 27th percentile on the tests administered while the average student in a TFA classroom scored in the 30th percentile; moving from the 27th percentile on a test to the 30th percentile does not represent, on average, 2.6 months of learning.  Furthermore, 40% of classrooms with TFA teachers scored lower than comparison classrooms taught by non-TFA teachers.  The study’s results were statistically significant, sure, but the advantage they show for TFA teachers is remarkably slight at best.

To me, the most important takeaway from the Mathematica study is that students at TFA placement schools, in general, perform terribly on standardized tests no matter who happens to be teaching them.  The reasons for that fact, as I alluded to in my last post, have a lot less to do with teaching and school quality than reformers would have us believe.  Most teachers, whether from TFA or any other program, want to help kids learn and are working hard towards that end most of the time.  But despite our best efforts, in-school reforms alone do little to impact the achievement gap.  The Mathematica study suggests that educators can only succeed if we simultaneously address economic inequality and other outside-of-school factors that disadvantage low-income students. I’d like to see critics and proponents of TFA alike stop quibbling about marginal improvements on standardized tests and start concentrating on the larger-scale advocacy that can really make a difference.

1 Comment

Filed under Education

Tagged as , , , , , , ,

by | September 9, 2013 · 10:43 pm

The Teach Like a Champion Paradigm

At the middle school where I currently serve as Math Instructional Coach, part of our professional development consists of reading Doug Lemov’s Teach Like a Champion.  This book provides a list of “49 techniques that [ostensibly] put students on the path to college” and is a Teach For America (TFA) favorite.  I have nearly finished the book and many of the techniques are great – I believe it can provide considerable value to teachers.  The book has a few major flaws worth discussing, however.  Lemov makes two assumptions that are representative of what I believe to be some of the biggest problems with TFA and the broader education reform movement.

First, Lemov assumes “the highest levels of student performance” are the goal of education and measures performance with standardized test scores.  He briefly acknowledges that “state test results…are not sufficient” but then buries that comment in a bunch of prose defending the value of what test results measure.  As one of my best former teachers argues in a recent article, reformers far too often concentrate on academic concerns “to the complete exclusion of the characteristics that make a difference in kids’ lives.”

The vast majority of TFA staff and corps members I have encountered care deeply about their students, and I imagine Lemov does as well.  But I’d like to see educators focus at least as much time and energy discussing how to inspire kids to be better people as Lemov does explaining how “a walk to the bathroom is a perfect time for a vocabulary review.”  The singular focus on academic achievement and test scores in the education reform debate has also contributed to the nationwide decline in art and physical education classes since No Child Left Behind.

Second, and much more dangerously, Lemov uses his charter network’s test scores to assert: “Great teaching, [our] teachers have proven, is strong enough to close the achievement gap.”  As evidence for this claim, Lemov cites the scatterplot below, which documents the scores of every public school in New York state on a 2009 English Language Arts assessment:

New York State ELA Scatterplot

FRPL stands for free and reduced-price lunch, an indicator of student poverty levels. The circled data point is Rochester Prep, a school from Lemov’s charter network, Uncommon Schools.

Lemov’s declaration – that this chart and others like it demonstrate that teaching effectiveness closes the achievement gap – is the fuel for much of the national conversation blaming the achievement gap on ineffective teachers and schools.  Reformers argue that dismantling teachers unions, linking teacher evaluation to student test score data, shutting down “failing” public schools in favor of charters, and a whole host of other changes would drastically improve educational outcomes in this country.  TFA and books like Teach Like a Champion, both of which suggest educator skill causes the results in the graph shown above, empower these reformers.  Sadly, not only is a causal link between teacher effectiveness and student performance data unestablished and the data itself unreliable, but data presented on charter school success is also incredibly misleading.

In-school factors have never been shown to explain more than one-third of the achievement gap and recent research suggests income inequality is the predominant cause of the achievement gap.  Additionally, outliers are expected in any dataset, Uncommon Schools is one of several that fall well above the trendline in the scatterplot above, and it’s possible (though I believe it unlikely) that the school just got lucky.  But let’s leave those considerations aside for a second and ask whether the student populations at Uncommon Schools are truly representative of the populations the schools draw from.  Lemov states,

Our students are selected at random from the districts where we work, have a higher poverty rate than the districts from which we draw, and, contrary to myth, are often the least, rather than the best, prepared students in those districts (one of the major reasons parents exercise school choice is that their students are struggling and increasingly at risk in their original schools; they are moving from as much as to).

His claim that his students are selected at random is just plain untrue and contradicted by his description of school choice at the end of the sentence.  While students who apply for admission are admitted via a random lottery, any introductory statistics student can tell you that a random sample from a group of volunteers in an area is most definitely not a random sample from the entire area.  Lemov provides no evidence to support his claim that his students are less prepared than their counterparts who don’t apply because there is no evidence to support this assertion; in fact, families who have the knowledge and commitment to seek new schools for their students are more likely to find success in any environment.  Lemov’s statement, if not willfully deceptive, reflects a poor understanding of statistics.

Gary Rubinstein, a 1991 TFA corps member and seasoned educator, has extensively studied data presented by “miracle” schools, schools like those in the Uncommon Schools network that achieve seemingly unbelievable results.  Rubinstein has found that in most cases those results are unbelievable because they are, well, not worthy of being believed.  Whether charters lose students by expelling them, by not-so-subtly suggesting to their families that kids won’t be successful at the school (a practice a friend of mine who worked at a KIPP school called “counseling out”), or through some other method (perhaps unintentional), the student attrition rates at these charter schools are very high and the percentages presented in most cases don’t account for this fact.  When a “miracle” school’s change in the student body is taken into account, the school’s results are hardly noteworthy.  These schools often help students, to be sure, but the students they help are generally small, unrepresentative subsets of students in poverty.

Schools in the Uncommon Schools network are quite possibly awesome schools for the students they serve, and the teachers Lemov observed and wrote about are most likely excellent teachers.  Their results, however, despite Lemov’s and reformers’ claims, do not indicate that great teaching can level the playing field for children in poverty.  Teachers can and should work hard to change students’ lives and there are undoubtedly cases where we succeed.  But when reformers advance changes based on misleading data and/or false claims from people like Lemov, our students suffer and the goal of educational equity becomes harder to realize.

5 Comments

Filed under Education

Tagged as , , , , , , , , , ,