Tag Archives: education research

A Closer Look at TFA’s Alumni Survey and Data Practices

Teach For America (TFA) recently released a report describing some of the results from its alumni survey.  The report provides interesting information about many alums’ careers.  At the same time, however, TFA’s misleading presentation of the results and the organization’s unwillingness to share the underlying data are troubling.

Consider the following three charts:

TFA Chart 1TFA Chart 2TFA Chart 3

The first of these charts claims to show “the mean value on the measure of total years teaching for each cohort,” the second ostensibly shows both the “mean and median total years teaching, by cohort,” and the third purports to show “the percentages of alumni in each Teach For America cohort that reported teaching 3 or 4, 5 or 6, or more than 7 years.”  A reader could easily think these statistics apply to everyone who joined Teach For America between 1990 and 2010.  But the charts depict statistics only for most TFA alumni who responded at some point to a TFA alumni survey; the two biggest categories of people they ignore are alumni who never responded to a survey and former corps members who did not complete their original two-year teaching commitments (while Raegen Miller and Rachel Perera, the authors of the brief, are clear that only alumni are their focus, the word “cohort” invokes images of all corps members that began teaching during a given year).  In other words, the charts depict results for an unrepresentative subset of the teachers who join Teach For America.

To their credit, Miller and Perera do discuss these issues in footnote 4:

The 23,653 alumni represented here represent about 85 percent of the alumni from the 1990 to 2010 cohorts. While alumni who don’t respond to the survey may be systematically different than those who do respond, the response rate is high enough to bolster the case that our statistics speak pretty well to the population of alumni. The alumni survey is administered only to those who complete their initial commitment to teach for two years. Alumni reporting fewer than two years of teaching on the survey, or more than the number of years possible since their corps year, were excluded from these analyses.

Still, most consumers of these misleading charts, especially when a chart is tweeted or included in a summary blog post, are unlikely to see footnote 4.  The footnote also doesn’t address the fact that TFA’s presentation of their survey results is in some cases downright untrue, as is the case in the following sentence from this TFA media piece: “Among more than 42,000 Teach For America alumni, some 84 percent report working in education or other capacities serving low-income communities.”

This statement is false because, as noted above, TFA does not have reports from the full population of “more than 42,000 Teach For America alumni” (TFA alumni may also incorrectly report that they work in “other capacities serving low-income communities” when they don’t in fact do so, but that’s a different conversation).

The charts below thus attempt to describe what the survey results actually tell us.

Revised TFA Chart 1

Revised TFA Chart 2

Because TFA won’t share their underlying data and declined to comment on this section of my post, the charts present a likely lower bound based on some simplifying assumptions I made: namely, that TFA has usable responses for at least 70 percent of alumni in each cohort (I’m told that response rates are typically lower in earlier cohorts), that fewer than 15 percent of corps members quit TFA before completing their second year of teaching, and that these corps members who quit average 0.75 years of teaching.

We cannot assert with confidence, as the report does, that “of those alumni from cohorts having had abundant opportunity to teach at least 5 years—say, those before 2001— approximately half have done so.”  Instead, with worst-case assumptions about alumni who didn’t take the survey, we know that at least between 34 and 39 percent of alumni from these cohorts taught for that amount of time.  If we include non-alumni in our calculations, as my revised charts show and seems appropriate, the lower bound drops even further (to a little less than one third of the teachers in each corps year).  It also appears to be possible that the majority of TFA corps members teach for two years or fewer.

That doesn’t mean these lower bounds are the actual percentages – in fact, I suspect that the true percentages, while likely lower than the report’s estimates, are closer to what Miller and Perera report than to the lower bounds.  Unresponsive alumni aren’t necessarily less inclined to continue or return to teaching than those who respond to surveys.  I also imagine that some corps members who leave before becoming alumni eventually return to the classroom through other channels.

Furthermore, even if the lower bound is closer to the truth, it’s clear that a good chunk of TFA alumni have extended teaching careers.  And while TFA’s attrition is greater than teacher attrition in general (which is still pretty high), critics sometimes forget that creating lifelong teachers is not TFA’s purpose.  A key part of TFA’s mission is to expose people interested in other careers to the obstacles facing low-income kids – to turn alumni who will go on to work in other fields into lifelong advocates for disadvantaged populations (this goal can backfire – prominent alumni too often undermine social justice efforts – but I believe it is an admirable goal).

The thing is, a thorough and honest look at TFA’s statistics isn’t damning.  The organization has many problems and needs to improve, but its teachers certainly aren’t harming students.  What is damning, however, is TFA’s lack of transparency.

When I requested information on the number of respondents from each cohort, the number of total alumni from each cohort, and the number of total corps members from each cohort – explaining that I wanted to conduct the exercise above – I had a good conversation with Miller that unfortunately culminated in the following response:

I’m afraid we won’t be posting the cohort-level n’s for the main analytic sample featured in Unsung Teaching. The brief does enough to defend the findings as is. In particular, the take-away points are broad ones, spanning many cohorts. And the footnotes document the rather conservative approach. Over 85 percent of alumni from the cohorts portrayed are represented in the analytic sample. At the cohort level, the percentages vary in a pretty predictable way, but none of them is small. Hard generalizations at the cohort level would be defensible, for the most part, but we didn’t go there. The little social media exchange has totally confounded the issue of response rate on the annual survey and the n’s you seek. The reason 85 percent of alumni (from cohorts 1990 to 2010) are involved is that some of the responses come from older surveys than the 2014 one. Using some dated responses improves the generalizability, but it does bias the statistics—downward. People can dispute the value of the work, but there’s not much of an argument to make with the n’s, at least about the brief. Thus, the decision not to share.

Miller makes some good points about the interpretation of the results.  The size of the sample – over 85 percent of total alumni from the studied cohorts – is impressively large.  Additionally, Miller is right to point out that the inclusion of older survey data could bias the results downwards (an alum who had taught for only two years when she filled out the 2010 survey, for example, might have taught more in the time since despite failing to fill out another survey).  But neither of these points provides a justification for declining my request.

The data I (and others, in the social media exchange I believe Miller was referencing) asked for should not be particularly difficult to provide.  Refusing to share it just doesn’t make sense if the data is legitimate – it serves only to make people wonder if TFA has something to hide.

On some level, TFA’s decision here is part of a broader pattern of data misrepresentation.  Many of my colleagues and I observed it on a small scale at TFA’s Institute in 2010 and it’s apparent on a larger scale when TFA propagates bogus statistics about corps members’ impact. Key TFA decision-makers still seem in too many cases to care more about promoting TFA as something that “works” than about honest and accurate assessment of evidence.

Since TFA proclaims that they “welcome independent research efforts to assess [their] impact and inform the continuous improvement of [their] program,” I find these actions to be pretty hypocritical.

Because the vast majority of the TFA staff I know (including Miller) care deeply about educational equity, I also find TFA’s organizational mindset baffling.  As I’ve written before, our students “depend on us to combat misleading claims by doing our due diligence, unveiling erroneous interpretations, and ensuring that sound data and accurate statistical analyses drive decision-making.”  TFA’s disinterest in data sharing does such analyses – and thus low-income students – a disservice.

The simplest fix to these problems is for TFA to appropriately caveat alumni survey results and, more generally, to improve the way they report statistics.  TFA could also use random sampling to obtain more representative data.  We used stratified random sampling for one of our member surveys when I was on the Executive Board of the San Jose Teachers Association (SJTA) and it paid major dividends; we could for the first time speak confidently about the generalizability of our results.

Interestingly, the results from SJTA’s random sample closely resembled the results from our pure volunteer survey.  The same could conceivably be true for the findings from Miller and Perera.  It’s just impossible to know without more research.

More importantly, TFA’s misleading data presentation and lack of transparency, combined with TFA’s silence in response to thoughtful critiques in other domains, make it difficult to trust the organization.  For an alum like me who deeply respects almost everyone on TFA staff I know and who thinks TFA could potentially be a strong ally in the fight for social justice, that’s a real shame.

3 Comments

Filed under Education

The Political Lens: What Global Warming and Wright v. New York Have in Common

During the 2003-2004 school year, my chemistry teacher told my class that global warming wasn’t occurring.  I believed her.  When I attended New Jersey’s Governor’s School of International Studies in the summer of 2005, a professor told me the opposite – the evidence for global warming, and for the human contribution to it, was virtually incontrovertible.  Confused about what to think, I began to research the issue.  I also reached out to some of my other former teachers to ask for their input.

Three things became immediately clear.  First, most popular articles about global warming contained more empty rhetoric than useful information.  The mainstream media, as it far too frequently does, focused not on the truth but on grandstanding and a false sense of balance.  Second, I didn’t know enough climate science to look through a given study’s results and determine their legitimacy.  Third, I didn’t have to – a different approach could tell me everything I needed to know about each study’s likely veracity.

Global warming research falls into two categories: research by legitimate scientists and “research” funded by big energy interestsLegitimate scientists, who have no economic incentive to lie, conclude that global warming is a manmade crisis deserving our immediate action.  The few studies that suggest otherwise are normally sponsored by organizations like Exxon and the American Petroleum Institute, interest groups with billions of dollars invested in the activity responsible for global warming.

As with global warming, knowledge of the individual and organizational incentives behind opposing “sides” of any debate provides us with critical information.  This “political lens,” though not completely foolproof, reminds us that certain claims deserve a larger dose of skepticism than others.  The agendas behind a movement are especially important to consider when we lack in-depth knowledge of a particular issue.

In education policy debates, “reformers” far too often selectively and inaccurately apply the political lens or dismiss its importance.  That dynamic surfaced after Stephen Colbert interviewed former CNN anchor Campbell Brown on July 31. Brown’s organization, Partnership for Educational Justice, had filed Wright v. New York three days before the interview.  Wright, modeled after Vergara v. California, challenges several aspects of teacher employment law.

A small group of teachers, parents, and grassroots organizers showed up to protest Brown’s appearance on the show.  Colbert, responding to the protesters and the Twitter hashtag #questions4campbell, asked Brown about her organization’s funding sources.  Brown refused to disclose her donors.  Amidst the criticism that followed, various stakeholders have rushed to Brown’s defenseThey continue to argue that a focus on Brown’s donors and political affiliations is a “desperate effort to distract from the real conversation” about teacher employment law.

The truth of the matter, however, is that educators would love to focus on substantive conversations about teacher employment law.  Teacher “tenure” and dismissal and layoff procedures, though they are intended to protect both student and teacher access to a positive, productive educational experience, don’t always work as intended.  Unions recognize this problem and recommend legislative improvements that simultaneously address issues with the execution of the laws and preserve their important components.  We also frequently discuss the laws on their merits.  Additionally, student advocates would love to see reformers, unions, and legislators engaged in substantive conversations about how to unite behind and fight for causes that matter considerably more for the lives of low-income students: in-school causes like funding equity and improved teacher support and out-of-school causes like the living wage and immigrant rights.

Unfortunately, pro-Wright propaganda, featured much more prominently in the mainstream media than legitimate arguments for the defense, often drowns out these “real conversations.”  No teacher has a job for life, competent school districts can and do dismiss bad teachers, and there is absolutely no evidence that teacher employment law causes inequities between low-income and high-income schools***, yet relatively large swaths of the American public have bought Brown’s misleading narrative and harbor severe misconceptions about the statutes and their effects.  Brown isn’t leading her crusade with a rigorous analysis of the facts and sound logical argument; instead, she “addresses” the lawsuit’s substantive critiques by ignoring inconvenient statistics and logic and implying that disagreement indicates a disregard for the well-being of children.  It’s hard for the public to understand the nuances of education law and research when Wright supporters prominently and erroneously equate opposition to the lawsuit with the defense of horrible teachers.

Thus while education law and research is arguably less complicated than the science behind global warming, the political lens is equally important to consider in this debate.  It’s theoretically possible that the unions who defend teacher employment law do so to protect teachers who call students names and sleep in class.  And it’s theoretically possible that Campbell Brown and her unnamed donors care more about the lives of low-income kids than do the unionized teachers who work with them every day.  It’s also theoretically possible that Exxon produces more honest research about global warming than does the entire scientific community.  But these theoretical possibilities are all extremely unlikely.

Instead, it’s significantly more likely that Campbell Brown’s donors, like the people who funded Vergara v. California, actively exacerbate economic inequality.  That Wright v. New York and Vergara conveniently allow them to undermine organized labor and distract us from the ways their business and political activities harm the families of the very same low-income students they purport to help.  That teachers in unions care deeply about delivering an excellent education to their students, and that their opposition to the lawsuit stems from its negative narrative, erroneous claims and premises, and failure to provide solutions to the actual causes of teacher quality issues.  In other words, looking through our political lens reminds us that there are literally billions more “adult interests” in support of Wright v. New York than in its defense.

Educators must continue to clarify facts about teacher employment law and support responsible reforms.  Most proponents of challenges to the statutes are well-intentioned, and a focus on agendas alone would not do the issues justice.  It is also entirely legitimate, however, to call attention to the profit and political motives behind lawsuits like Wright and Vergara.  Knowledge of donors and allies helps us understand why, when unions and Campbell Brown present conflicting information about the law’s intent and effects, Campbell Brown’s claims warrant significantly more suspicion.

Campbell Brown graphic

***While the plaintiffs in Wright, unlike those in Vergara, do not erroneously contend in their complaint that the laws cause inequities between low- and high-income schools, the idea that low-income students are disproportionately impacted by bad teachers was mentioned by Brown in her appearance on The Colbert Report and still surfaces in discussions of the lawsuit.

Update: A version of this post appeared on The Huffington Post on October 2.

1 Comment

Filed under Education, Labor, Philosophy

On Education and Poverty, and How We Talk About Them (Part 3b)

StudentsFirst Vice President Eric Lerum and I recently began a debate about approaches to teacher evaluation.  During Part 2 of that debate, the conversation touched on the relationship between anti-poverty work and education reform.  We resume that conversation below.

Here were the relevant parts of our original exchange, in case you missed it:

Lerum: The larger point that is made repeatedly is that because outside factors play a larger overall role in impacting student achievement, we should not focus on teacher effectiveness and instead solve for these other factors. This is a key disconnect in the education reform debate. Reformers believe that focusing on things like teacher quality and focusing on improving circumstances for children outside of school need not be mutually exclusive. Teacher quality is still very important, as Shankerblog notes. Improving teacher quality and then doing everything we can to ensure students have access to great teachers does not conflict at all with efforts to eliminate poverty. In fact, I would view them as complementary. But critics of these reforms use this argument to say that one should come before the other – that because these other things play larger roles, we should focus our efforts there. That is misguided, I think – we can do both simultaneously. And as importantly in terms of the debate, no reformer that I know suggests that we should only focus on teacher quality or choice or whatever at the expense or exclusion of something else, like poverty reduction or improving health care.

Spielberg: I believe you discuss [a] very important question…Given that student outcomes are primarily determined by factors unrelated to teaching quality, can and should people still work on improving teacher effectiveness?

Yes!  While teaching quality accounts for, at most, a small percentage of the opportunity gap, teacher effectiveness is still very important.  Your characterization of reform critics is a common misconception; everyone I’ve ever spoken with believes we can work on addressing poverty and improving schools simultaneously.  Especially since we decided to have this conversation to talk about how to measure teacher performance, I’m not sure why you think I’d argue that “we should not focus on teacher effectiveness.”  I am critiquing the quality of some of StudentsFirst’s recommendations – they are unlikely to improve teacher effectiveness and have serious negative consequences – not the topic of reform itself.  I recommend we pursue policy solutions more likely to improve our schools.

Critics of reform do have a legitimate issue with the way education reformers discuss poverty, however.  Education research’s clearest conclusion is that poverty explains inequality significantly better than school-related factors.  Reformers often pay lip-service to the importance of poverty and then erroneously imply an equivalence between the impact of anti-poverty initiatives and education reforms.  They suggest that there’s far more class mobility in the United States than actually exists.  This suggestion harms low-income students.

As an example, consider the controversy that surrounded New York mayor Bill de Blasio several months ago.  De Blasio was a huge proponent of measures to reduce income inequality, helped reform stop-and-frisk laws that unfairly targeted minorities, had fought to institute universal pre-K, and had shown himself in nearly every other arena to fight for underprivileged populations.  While it would have been perfectly reasonable for StudentsFirst to disagree with him about the three charter co-locations (out of seventeen) that he rejected, StudentsFirst’s insinuation that de Blasio’s position was “down with good schools” was dishonest, especially since a comprehensive assessment of de Blasio’s policies would have indisputably given him high marks on helping low-income students.  At the same time, StudentsFirst aligns itself with corporate philanthropists and politicians, like the Waltons and Chris Christie, who actively exploit the poor and undermine anti-poverty efforts.  This alignment allows wealthy interests to masquerade as advocates for low-income students while they work behind the scenes to deprive poor students of basic services.  Critics argue that organizations like StudentsFirst have chosen the wrong allies and enemies.

I wholeheartedly agree that anti-poverty initiatives and smart education reforms are complementary.  I’d just like to see StudentsFirst speak honestly about the relative impact of both.  I’d also love to see you hold donors and politicians accountable for their overall impact on students in low-income communities.  Then reformers and critics of reform alike could stop accusing each other of pursuing “adult interests” and focus instead on the important work of improving our schools.

Lerum: So I’m beginning to understand where some of the miscommunication is coming from. You speak a lot about how you view StudentsFirst’s (and other reformers’) discussion of poverty from the perspective of what you expect us to talk about, rather than from the perspective of our stated objectives. That is, what you deem as “lip service” is merely an acknowledgement of something that is not our primary focus. There are many folks in education reform – I have a few on my team – who could spend hours talking about poverty reduction and could very easily work in another field that more traditionally aligns with what you think of as efforts geared toward reducing poverty. But the route we’re taking is one where reducing poverty, achieving social justice, lifting the long-term opportunities for our country – they all intersect. And therefore what we focus on at StudentsFirst are the policy levers – what we think of as levers for reform or change. For example, creating the conditions for other reforms to flourish or for educators and school leaders to use their resources more wisely (fiscal transparency, structuring smarter compensation systems, creating more school-level autonomy) are levers, whereas something like instituting a STEM program or increasing funding for social and mental health services would be specific programs or initiatives. Both are great for kids. Both are needed in order to ultimately reduce poverty. But we’re squarely focused on the former, while critics seem to be expecting we would focus on the latter. This disconnect is made worse though because critics seem to believe that an approach that involves initiatives is the only way to combat poverty. There’s a lack of appreciation and understanding of what’s intended by reform efforts that target levers.

Spielberg: I actually wasn’t talking about the distinction between levers and initiatives; I was talking about accurate messaging and political activity.

My two critiques from above (rephrased and with my questions for you added) were:

1) StudentsFirst leaders and board members frequently suggest that education can improve the lives of low-income kids as much or more than alleviating poverty.  That suggestion is demonstrably false.  You could say the following, but don’t: “Research is clear that school-related factors cannot fix the achievement gap, but it’s also clear that schools make a difference.  They seem to account for about 20% of student achievement, and our organization believes we can maximize the impact of this 20% with an intense focus on certain policy levers.  We fully support other organizations that work on the anti-poverty efforts that are most important for low-income kids.”  Why won’t you speak honestly about the limitations and relative importance of the reforms you push when compared with other efforts?

2) Relatedly, StudentsFirst supports politicians (besides just Chris Christie, who I discussed above) who substantially harm some of the neediest kids: your preferred candidates have rejected the Medicaid expansion, slashed education spending, tried to prevent immigrants from enrolling in school, and actively discriminated against LGBT youth (though you finally withdrew support for your 2013 “education reformer of the year” after intense public pressure).  StudentsFirst says on your website that the candidates you support “have demonstrated a commitment to policies that prioritize student interests;” I find this assertion at best myopic, and at worst deliberately misleading.  How can you reconcile StudentsFirst’s candidate support with the fact that, on the whole, many of these candidates cause significant harm to low-income and minority students?

I appreciate, as you mentioned in a comment on Part 2 of this conversation, that you “created a school-based mental health program and piloted a half-dozen evidence-based mental/social/emotional health programs” in DC, and I’d love to talk more about the other issues you raised in your response, but I think your thoughts on the above points and questions are most relevant to typical reformer critiques.

Lerum: On the policy discussion, I would just end with this then. Saying that education can’t solve the achievement gap is demonstrably false only works if you base it on the education system we have now. To say that today’s education system cannot and has not solved the problem of poverty or the problem of the achievement gap thus far is correct. It’s also correct that in 60 years we haven’t solved the problem of segregation. But I got into this work because, like every reformer I know, I believe completely that we can do better than this. We don’t even know what’s possible because we haven’t actually tried. We’ve never run a public school system at scale completely differently. We’re not very good at breaking the mold of a model that hasn’t worked. But there are reasons – an increasing body of research – to believe that if we do, we just might get somewhere. That’s a theory of change. You can disagree with it. But you do not have the evidence that it won’t work because everything that’s been tried or done thus far has been done within some confines or under some of the restraints of the existing system. There are many limitations, that’s true. I think we’ve done a pretty good job of talking about those limitations through our advocacy work.

I would also add that there’s little evidence that other approaches that are championed as counters to reform will have a tremendous impact on kids either. I would love to see this “comprehensive assessment of de Blasio’s policies” that you spoke of earlier – but it doesn’t exist. Rather, there is simply a different theory of change – that certain other levers, be they class size or overall funding or whatever will have a greater impact than reforms we’re advocating for. What we need is a way to model, using rigorous research, what the potential impact of various reforms would be. That doesn’t exist right now either. But what I’m trying to get you to agree to here is that by attacking one side as only having a theory that’s not proven while not acknowledging that the anti-reform side isn’t exactly operating with a track record of success seems to me to be disingenuous, but more importantly, allows opponents to occupy this space wherein they own the debate on what’s good for solving poverty, what the right approach is to combat social ills, etc. And I just believe that way of thinking hasn’t gotten us very far and doesn’t advance social change.

As to the political issues you raise – we consistently say that we will support public officials who support the policies we believe are right for kids. I understand you have issues with our agenda – but there’s nothing inconsistent about a single-issue organization supporting candidates that support and will advocate for their issues. That almost always means as an organization we will support candidates with whom I may not agree with on a personal level when it comes to any number of other issues. But that is not unique to StudentsFirst and I do not think it is reasonable to expect us to answer for their stances on other issues or to ask them to change their stance on other issues. The issues we prioritize are those on our policy agenda and we work to stick with that approach, as do countless other organizations in other fields.

Spielberg: While I would agree with you (and said in Part 1 of our conversation) that the research on many in-school reforms is mixed, the suggestion that you seem to be making – that school-based reforms alone could potentially solve the opportunity gap – is contradicted by existing research and logic.  Research has never attributed more than one-third of the variation in student outcomes to school-based factors, we know that “children from rich and poor families score very differently on school readiness tests when they enter kindergarten,” and there is even “some evidence that achievement gaps between high- and low-income students actually narrow during the nine-month school year, but they widen again in the summer months.”  Though I suppose it’s theoretically possible that these studies are wrong, that could be said about almost anything, and the findings you link about teacher attrition and charters in no way support that conclusion.  Our knowledge about the disadvantages of growing up in poverty and the past several decades of research suggest that this theoretical possibility is negligible, which is why I called that statement demonstrably false.

I certainly understand the sentiment behind what you’re saying – we are in agreement that we haven’t yet maximized education’s contribution to anti-poverty efforts, and I think it’s important to remember and highlight that fact – but all the evidence points to a relatively low upper bound on what education reforms alone can accomplish.  Recognizing that anti-poverty work matters more than schools does not preclude us from arguing that what happens in schools is very important for low-income kids.

I really appreciate having had this conversation and want to thank you again for going back-and-forth with me, but I believe we’re at a bit of an impasse.  My questions deal with the out-of-school factors, like having access to health care, that very clearly matter for low-income students, and I don’t think your response addresses the issues I raised.  You’re absolutely right that StudentsFirst isn’t alone in narrowing its policy focus, but the fact that other organizations also do so doesn’t qualify as a defense of that approach.  Talking about what’s “right for kids” means considering more than just education policy.

As I’ve pointed out to critics of typical reform efforts before, I think it would be reasonable for reform organizations to focus their professional advocacy on school-based approaches to the opportunity gap if you did two things:

1) Acknowledge that the best school-based reforms imaginable, while important, would likely only be able to solve 20% to, at most, 33% of the problem.

2) Avoid undermining the anti-poverty work that can address a larger percentage of the opportunity gap.

I don’t believe that StudentsFirst currently does those two things, but I will leave it up to our readers to decide which arguments they find more compelling.

Leave a comment

Filed under Education, Poverty and the Justice System

Vergara v. California Panel Discussion with Leadership for Educational Equity

Leadership for Educational Equity (LEE), Teach For America’s (TFA’s) partner organization that focuses on alumni leadership development, held an online panel for members interested in learning more about Vergara v. California on June 26.  I was excited to receive an invitation to speak on the panel – I enjoyed talking to LEE members about how teachers unions benefit low-income students at an earlier event and appreciate LEE’s recent efforts to include organized labor in their work.  LEE received over 100 RSVPs from TFA corps members and alumni who tuned in to hear our discussion of the case.

LEE Panel Vergara

USC Professor of Education & Policy Katharine Strunk, Georgetown Professor of Law Eloise Pasachoff, and former Assistant Secretary of Civil Rights for the US Department of Education Russlynn Ali joined me for an engaging hour-long session.  Each of the panelists had ample time to make opening and closing remarks and to respond to each other’s points.  You can listen to the full audio for yourself below, but I also wanted to summarize two points I made at the end of the session:

1) It’s important to read the full text of education research articles because the findings are frequently misconstrued.  As I mentioned during my initial remarks, there’s a pretty strong research basis behind the idea that teachers are the most important in-school factor related to student success (though it’s important to remember that in-school factors, taken together, seem to account for only about 20% of student achievement results).  Nobody disagrees that teacher quality varies, either – it’s clear that low-income students sometimes have teachers who aren’t as high-quality as we would like.  Additionally, there’s broad consensus that improving teacher quality and addressing inequities between low-income and high-income schools are both important objectives.  The research does not suggest, however (and the plaintiffs did not show at trial), that there is a causal link between teacher employment law and either teacher quality issues or inequities between low-income and high-income schools.  There’s plenty of rhetoric about how employment law causes inequity but no actual evidence supporting that claim.  The other panelists and I unfortunately didn’t have enough time to engage in substantive conversations about the validity of the research we discussed, but I hope we have the opportunity to do so in the future.

2) Most union members and most people working within reform organizations have the same goals and should be working together.  We should therefore consider our rhetoric carefully.  Instead of insinuating that the unions who defend teacher employment law care more about protecting bad teachers than helping students, reformers could ask unions how more sensible reforms could make sure the execution of the laws aligns with the ethical, student-oriented theory.  Reformers could then signal their support for organized labor and work with unions to address the real root causes of teacher quality issues and inequities between schools.  The other panelists indicated their belief in reasonable due process protections, improved teacher evaluation and support, and equitable school funding, and kids would benefit if reformers and unions united behind these causes and pursued them with the same vigor with which some have jumped on the Vergara bandwagon.

You can hear more of my thoughts beginning about 22 minutes and 30 seconds into the clip, though I’d encourage you to listen to the whole thing if you have the time.  I’d also love to discuss the case more in the comments with anyone interested.  Hope you enjoy the panel!

Note: An earlier version of this post called LEE “Teach For America’s alumni organization.”  The reference has been changed to reflect that, while LEE focuses on leadership development for TFA alumni, they are an independent organization.

Update (7/19/14): The following sentence was modified to clarify that addressing teacher quality issues and addressing inequities between low-income and high-income schools are distinct tasks: “Additionally, there’s broad consensus that improving teacher quality and addressing inequities between low-income and high-income schools are both important objectives.”  The original sentence read: “Additionally, there’s broad consensus that improving teacher quality and addressing inequities between low-income and high-income schools is important.”

6 Comments

Filed under Education

Approaching Education Data the Nate Silver Way

My girlfriend’s very hospitable and generous family gave me some great gifts for the holidays when I stayed with them in upstate New York.  As I rocked my new Teach For America T-shirt in the Rochester airport on Christmas Eve, my cursory overview of Nate Silver’s new book, The Signal and the Noise, inspired me to write this post.

While most people probably know Silver for his election predictions and designation in 2009 as one of the world’s 100 Most Influential People, Silver has been my baseball stat guru for considerably longer than he’s been doing political analysis.  In one of my favorite books of all time, Baseball Between the Numbers, Silver penned a brilliant examination of clutch hitting that I still quote at least four or five times a year.  I have generally found Silver’s arguments compelling not just because of his statistical brilliance, but also because of his high standards for data collection and analysis, evident in the following passage from the introduction of his book:

The numbers have no way of speaking for themselves.  We speak for them.  We imbue them with meaning…[W]e may construe them in self-serving ways that are detached from their objective reality…Before we demand more of our data, we need to demand more of ourselves.

In few fields are Silver’s words as relevant as education.  While the phrase “data-driven” has become ubiquitous in discussions of school reform and high-quality instruction, most people discussing education have very little understanding of what the statistics actually say.  As I’ve written before, many studies that reformers reference to push their policy agendas are methodologically unsound, and many more have findings very different than the summaries that make it into the news.

It’s hard to know how many reformers just don’t understand statistics, how many fall victim to confirmation bias, and how many intentionally mislead people.  But no matter the reason for their errors, those of us who care about student outcomes have a responsibility to identify statistical misinterpretation and manipulation and correct it.  Policy changes based on bad data and shoddy analyses won’t help (and will quite possibly harm) low-income students.

Fortunately, I believe one simple practice can help us identify truth in education research: read the full text of education research articles.

Yes, reading the full text of academic research papers can be time consuming and mind-numbingly dull at times, but reading articles’ full text is vitally important if you want to understand research findings.  Sound bites on education studies rarely provide accurate information.  In a Facebook comment following my most recent post about TFA, a former classmate of mine referenced a 2011 study by Raj Chetty to argue that we can’t blame the achievement gap on poverty.  “If you leave a low value-added teacher in your school for 10 years, rather than replacing him with an average teacher, you are hypothetically talking about $2.5 million in lost income,” claims one of the co-authors of the study in a New York Times article.  Sounds impressive.  Look under the hood, however, and we find that, even assuming the study’s methodology is foolproof (it isn’t), the actual evidence can at best show an average difference of $182 in the annual salaries of 28-year-olds.

As I’ve mentioned before, there’s also a poor statistical basis for linking student results on standardized test scores to teacher evaluation systems.  Otherwise useful results can give readers the wrong impression when they gloss over or omit this fact, a point underscored by a recent article describing an analysis of IMPACT (the D.C. Public Schools teacher evaluation system).  The full text of the study provides strong evidence that the success of D.C.’s system thus far has been achieved despite a lack of variation in standardized test score results among teachers in different effectiveness categories.  Instead, the successes of the D.C. evaluation system are driven by programs teachers unions frequently support, programs like robust and meaningful classroom observations that more accurately measure teacher effectiveness.

Policymakers have misled the public with PISA data as well.  In a recent interview with MSNBC’s Chris Hayes, Michelle Rhee made the oft-repeated claim that U.S. schools are failing because American students, in aggregate, score lower on international tests than their peers in other countries.  Yet, as Hayes pointed out, it is abundantly clear from a more thorough analysis that poverty explains the PISA results much better than school quality, not least because poor US students have been doing better on international tests than poor students elsewhere for several years.

I would, in general, recommend skepticism when reading articles on education, but I’d recommend skepticism in particular when someone offers a statistic suggesting that school-related changes can solve the achievement gap.  Education research’s only clear conclusion right now is that poverty explains the majority of student outcomes.  The full text of Chetty’s most recent study defending value-added models acknowledges that “differences in teacher quality are not the primary reason that high SES students currently do much better than their low SES peers” and that “differences in [kinder through eighth grade] teacher quality account for only…7% of the test score differences” between low- and high-income schools.  In fact, that more recent study performs a hypothetical experiment in which the lowest-performing low-income students receive the “best” teachers and the highest-performing affluent students receive the “worst” teachers from kinder through eighth grade and concludes that the affluent students would still outperform the poor students on average (albeit by a much smaller margin).  Hayes made the same point to Rhee that I made in my last post: because student achievement is influenced significantly more by poverty than by schools, discussions about how to meet our students’ needs must address income inequality in addition to evidence-based school reforms.  We can’t be advocates for poor students and exclude policies that address poverty from our recommendations.

When deciding which school-based recommendations to make, we must remember that writers and policymakers all too often misunderstand education research.  Many reformers selectively highlight decontextualized research that supports their already-formed opinions.  Our students, on the other hand, depend on us to combat misleading claims by doing our due diligence, unveiling erroneous interpretations, and ensuring that sound data and accurate statistical analyses drive decision-making. They rely on us to adopt Nate Silver’s approach to baseball statistics: continuously ask questions, keep an open mind about potential answers, and conduct thorough statistical analyses to better understand reality.  They rely on us to distinguish statistical significance from real-world relevance.  As Silver writes about data in the information age more generally, education research “will produce progress – eventually.  How quickly it does, and whether we regress in the meantime, depends on us.”

Update: Gary Rubinstein and Bruce Baker (thanks for the heads up, Demian Godon) have similar orientations to education research – while we don’t always agree, I appreciate their approach to statistical analysis.

Update 2 (6/8/14): Matthew Di Carlo is an excellent read for anyone interested in thoughtful analysis of educational issues.

Update 3 (7/8/14): The Raj Chetty study linked above seems to have been modified – the pieces I quoted have disappeared.  Not sure when that happened, or why, but I’d love to hear an explanation from the authors and see a link to the original.

7 Comments

Filed under Education