Monthly Archives: December 2013

Approaching Education Data the Nate Silver Way

My girlfriend’s very hospitable and generous family gave me some great gifts for the holidays when I stayed with them in upstate New York.  As I rocked my new Teach For America T-shirt in the Rochester airport on Christmas Eve, my cursory overview of Nate Silver’s new book, The Signal and the Noise, inspired me to write this post.

While most people probably know Silver for his election predictions and designation in 2009 as one of the world’s 100 Most Influential People, Silver has been my baseball stat guru for considerably longer than he’s been doing political analysis.  In one of my favorite books of all time, Baseball Between the Numbers, Silver penned a brilliant examination of clutch hitting that I still quote at least four or five times a year.  I have generally found Silver’s arguments compelling not just because of his statistical brilliance, but also because of his high standards for data collection and analysis, evident in the following passage from the introduction of his book:

The numbers have no way of speaking for themselves.  We speak for them.  We imbue them with meaning…[W]e may construe them in self-serving ways that are detached from their objective reality…Before we demand more of our data, we need to demand more of ourselves.

In few fields are Silver’s words as relevant as education.  While the phrase “data-driven” has become ubiquitous in discussions of school reform and high-quality instruction, most people discussing education have very little understanding of what the statistics actually say.  As I’ve written before, many studies that reformers reference to push their policy agendas are methodologically unsound, and many more have findings very different than the summaries that make it into the news.

It’s hard to know how many reformers just don’t understand statistics, how many fall victim to confirmation bias, and how many intentionally mislead people.  But no matter the reason for their errors, those of us who care about student outcomes have a responsibility to identify statistical misinterpretation and manipulation and correct it.  Policy changes based on bad data and shoddy analyses won’t help (and will quite possibly harm) low-income students.

Fortunately, I believe one simple practice can help us identify truth in education research: read the full text of education research articles.

Yes, reading the full text of academic research papers can be time consuming and mind-numbingly dull at times, but reading articles’ full text is vitally important if you want to understand research findings.  Sound bites on education studies rarely provide accurate information.  In a Facebook comment following my most recent post about TFA, a former classmate of mine referenced a 2011 study by Raj Chetty to argue that we can’t blame the achievement gap on poverty.  “If you leave a low value-added teacher in your school for 10 years, rather than replacing him with an average teacher, you are hypothetically talking about $2.5 million in lost income,” claims one of the co-authors of the study in a New York Times article.  Sounds impressive.  Look under the hood, however, and we find that, even assuming the study’s methodology is foolproof (it isn’t), the actual evidence can at best show an average difference of $182 in the annual salaries of 28-year-olds.

As I’ve mentioned before, there’s also a poor statistical basis for linking student results on standardized test scores to teacher evaluation systems.  Otherwise useful results can give readers the wrong impression when they gloss over or omit this fact, a point underscored by a recent article describing an analysis of IMPACT (the D.C. Public Schools teacher evaluation system).  The full text of the study provides strong evidence that the success of D.C.’s system thus far has been achieved despite a lack of variation in standardized test score results among teachers in different effectiveness categories.  Instead, the successes of the D.C. evaluation system are driven by programs teachers unions frequently support, programs like robust and meaningful classroom observations that more accurately measure teacher effectiveness.

Policymakers have misled the public with PISA data as well.  In a recent interview with MSNBC’s Chris Hayes, Michelle Rhee made the oft-repeated claim that U.S. schools are failing because American students, in aggregate, score lower on international tests than their peers in other countries.  Yet, as Hayes pointed out, it is abundantly clear from a more thorough analysis that poverty explains the PISA results much better than school quality, not least because poor US students have been doing better on international tests than poor students elsewhere for several years.

I would, in general, recommend skepticism when reading articles on education, but I’d recommend skepticism in particular when someone offers a statistic suggesting that school-related changes can solve the achievement gap.  Education research’s only clear conclusion right now is that poverty explains the majority of student outcomes.  The full text of Chetty’s most recent study defending value-added models acknowledges that “differences in teacher quality are not the primary reason that high SES students currently do much better than their low SES peers” and that “differences in [kinder through eighth grade] teacher quality account for only…7% of the test score differences” between low- and high-income schools.  In fact, that more recent study performs a hypothetical experiment in which the lowest-performing low-income students receive the “best” teachers and the highest-performing affluent students receive the “worst” teachers from kinder through eighth grade and concludes that the affluent students would still outperform the poor students on average (albeit by a much smaller margin).  Hayes made the same point to Rhee that I made in my last post: because student achievement is influenced significantly more by poverty than by schools, discussions about how to meet our students’ needs must address income inequality in addition to evidence-based school reforms.  We can’t be advocates for poor students and exclude policies that address poverty from our recommendations.

When deciding which school-based recommendations to make, we must remember that writers and policymakers all too often misunderstand education research.  Many reformers selectively highlight decontextualized research that supports their already-formed opinions.  Our students, on the other hand, depend on us to combat misleading claims by doing our due diligence, unveiling erroneous interpretations, and ensuring that sound data and accurate statistical analyses drive decision-making. They rely on us to adopt Nate Silver’s approach to baseball statistics: continuously ask questions, keep an open mind about potential answers, and conduct thorough statistical analyses to better understand reality.  They rely on us to distinguish statistical significance from real-world relevance.  As Silver writes about data in the information age more generally, education research “will produce progress – eventually.  How quickly it does, and whether we regress in the meantime, depends on us.”

Update: Gary Rubinstein and Bruce Baker (thanks for the heads up, Demian Godon) have similar orientations to education research – while we don’t always agree, I appreciate their approach to statistical analysis.

Update 2 (6/8/14): Matthew Di Carlo is an excellent read for anyone interested in thoughtful analysis of educational issues.

Update 3 (7/8/14): The Raj Chetty study linked above seems to have been modified – the pieces I quoted have disappeared.  Not sure when that happened, or why, but I’d love to hear an explanation from the authors and see a link to the original.


Filed under Education

Aphorisms and a short story

I have a collection of aphorisms to present that I’ve been working on. The art of the maxim has gone out of style somewhat, lamentably. The less we say, the more others hear — if we say it well. It is so easy to rant on the Internet or in private with our friends, but more rewarding to first organize our thoughts first and distill what we have to say. Brevity is the soul of wit, they say, but if the brief is unnecessary, the wit is superfluous. Though we are loathe to admit, Twitter has its advantages; constraint demands creativity. By limiting ourselves, the eternal can be reached, through finite means. Isn’t this the whole purpose of art? — to explain the utter chaos of the human condition in dense, honest flavor?

Ok, back to the point. I hail from the tradition of Nietzsche, La Rouchefoucauld, Chamfort, Chateaubriand, Vauvenargues, Heraclitus, the comments section of the NYT, ATCQ, Nasir, squirrels, and the most plump of bumblebees. Below are some of my original aphorisms. Enjoy, if you can. Think, if you enjoy. Scoff, if you think.


The poor who want wealth redistribution will change their mind if they are rich, as will the rich who are against it if they are poor.

Few are nice enough to be your harshest critic.

Those who can’t handle criticism don’t deserve praise.

Price is set high to obscure quality.

To know wants from needs, and adjust expectations accordingly.

The love of money is always unrequited, yet its lovers abound. Fame, too.

Confidence concerns character; arrogance concerns reputation.

Conviction suffocates intellect.

We spend more time convincing others we are happy than being happy; happiness is its own pursuit.

We despise bragging for fear of its accuracy.

We mistrust praise — and yet desire it — because of our insecurity.

We are not afraid of heights, but only afraid of falling.

To know who you envy, who envies you, the cause and effect of both: a rare wisdom.

We are most insecure when we abuse love and love abuse.

The worst type of guilt is feeling guilty for not feeling guilty.

Politicians begin as sophists and end as demagogues.

A politician’s success depends on how well he can fool two parties: the people and himself.

Every god gets the atheists it deserves.

You want to be unique? So does everyone else. Join the crowd!

We hate being misunderstood, unless we are also admired.

The stylish are simply following the trends. And concern with being ‘stylish’ is a trend many follow.

Give unexpectedly without expectations.

High expectations and low patience: quick to marry, quicker to divorce.


finally, a short story on the power of excuses:

Every day, Jerome was stoned to death by his excuses. Yet the very next day, he always got back up, like the little train that could, worshipping at the altar of “Lights, Camera, Inaction!” His excuses gave him great comfort — something had to fluff his pillows at night, putting his fear in words made him feel invincible; as long as his excuses could find no obvious detractors, they were the only eulogists that he would ever need. Besides, excuses were renowned for their jealousy; they forbade other idols and gifted brimstone to those who doubted their omnipotence.


Filed under Philosophy

The Prisoner’s Dilemma

“I used to be fast, man. I’ll race you to that gas station.”

There is only one group of American citizens that are constitutionally guaranteed the right to medical services, and you may be surprised to find that it isn’t politicians, law enforcement, the elderly, or even veterans—it’s prisoners. The U.S. has always had a love-hate relationship with its law-breakers. We romanticize the old-timey gangsters, the outlaws, and the reclamation stories about overcoming turmoil and coming out stronger in the end (plus, when was the last time that there was a U.S. president that didn’t admit to ingesting illegal drugs?). And we hate the inner-city gun violence, gangs, and “thug culture” that seems so pervasive in the media today. The prison-industrial complex has been getting a lot of flak in the U.S. for well over a decade (and rightfully so). Often lost in the talk about profit motives and unjust penalties for petty crimes are the effect it has on the prisoners themselves once they are released into a different world from when they last left.


I began running with Back on My Feet (BoMF) a couple months ago. Back on My Feet is a non-profit organization that aims to combat homelessness by having daily runs at 5:30am in which residents of homeless shelters are grouped together with non-homeless runners, and they run about 3-5 miles around Philadelphia. The program is designed for former drug users, ex-convicts, or otherwise down-and-out types to get a regiment centered around fitness and a routine. There are milestones that once reached entitle the members to new goods – new shoes, running equipment, etc.—and services—resume editing and interview workshops. The shelter that I run with is Ready, Willing, and Able (RWA), and consists mainly of ex-cons that are being reintroduced into society.

I don’t normally like waking up early. In fact, I only do it when I have to. It’s even harder to wake up early when it’s cold out, which it always is at 4:45am. When I first began running with BoMF I basically tried to get each session over with quickly so that I could go back to the friendly confines of my own bed. I soon realized that no matter how fast I ran to the meeting place, or how fast I sprinted during our morning jog with the RWA members I would still get home at the same time. I might as well try enjoying this I figured. Soon waking up early was still difficult, but rather than lament that I felt like I was one of the only people awake on the east coast I felt that I should embrace it. As I leave my apartment and jog towards the meet up I will frequently stop to walk and look at the sky above me. There are stars out and lots of them. Just about all of the neon signs are turned off in Philadelphia in those early hours and I can see as far into the universe as my contact lenses allow me. There is something quite surreal about looking up and seeing distant, tiny lights that you have never noticed before through your own warm breath walking alone in a city of one and a half million. Perhaps it is this feeling of isolation and quiet that allows strangers to share things about their life, or perhaps it is just frustration.

Technically the program is designed so that one has no idea who are the ex-homeless and who are there for support—but it’s pretty evident who’s in each camp. I met ‘Steve’ during my first session of BoMF and we exchanged the usual pleasantries that are the norm for a “res-member” (someone who is a resident of one of the shelters), and a “non-res-member” (someone like me, just an outsider looking to run). At about mile 2 we had exhausted talk of weather and sports and so we just ran side by side quietly. My body had acclimated to the temperature by now but Steve was sweating profusely, his balding head spewing a trail of steam in his wake as we glided alongside the Schuylkill River. He turned to me asked if I had any plans for the end of the year. I wasn’t really sure what he meant by that but I could tell it was his way of asking me to ask about his own plans. Steve then began to discuss how hard it is to find a job that pays well enough to allow him go back to school. Already equipped with a GED, he always wanted to go to a technical school to become a certified mechanic in a two-year program. This December he was finally moving out of the halfway house and onto another phase of his life, but it did not come without plenty of meetings with social workers and temp agencies.

He seemed to have done the math thoroughly because he was spouting out figures on how much he would need to make per week, per day, and per hour in order for him to pay for technical school. He said that he was beginning to look at rent in different areas in Philly for when he leaves RWA. He also said that a lot of the places he was looking at were in areas that are too expensive and so he would have to choose between living in a decent area but barely having enough money left over to eat after savings, or to live in “the ‘hood” but have some extra money. He went on to say that living in the latter areas is what led him to prison in the first place, and that going back there would be a recipe for disaster. I asked what he thought he would do and he said that he would try to live in a nicer area but try to take lots of side jobs and see how much money he could save for a year or two and hopefully he could gauge the probability of him going to school. We made that final turn down Bainbridge St. and headed towards the Hess station that was light up like E.T.’s spaceship in the forest that is Broad St. at 6am.

How does someone get out of a cycle like this? Steve is lucky in a way—he has no kids, no child-support, and no wife, but he will still face an uphill battle for the foreseeable future. Just supporting himself and hitting the reset button will take the better part of a decade, and for a forty-something year old with a history of drug abuse and living a hard life, that’s looking to be on the other side of the halfway point. It is easy to talk about statistics to report how our prison system is failing on so many levels. And how (like many other things in our society) the profit-motive doesn’t align with the mission of the industry, which is namely to rehabilitate those that are incarcerated, and should therefore be taken out of the hands of those that seek to make the profit. We are so inundated with numbers and reports that they have virtually no meaning. What is the number that will create outrage? 2.3 million adults are currently incarcerated, which is about 740 per 100,000 people or 0.7% of the US population. Prison private telecom systems, health insurance contracts, and the bail industry are making profits in the billions each year. Does it matter if it is 1 billion, 2 billion, or 50 billion? At what point do we stand up and say something is wrong here when nearly 40% of those incarcerated are African-American? Is it when the percentage creeps up to 43, or how about a nice round 50%? These are people, plain and simple. It’s easy for me to say this now only because I have met some of these people, but this system is self-sustaining and every generation that has a high percentage of its members in jail will surely have even higher percentages in the next generation. There are some members of RWA that are in their early 20s that have multiple children who they no longer see. Where are these kids going to end up in 15 years?

The black sky was finally half purple, with what almost looked like blue along the Ben Franklin Bridge. For all the things not going his way, Steve doesn’t seem to mind as much when he’s running. He beat me to the gas station—by a lot actually—then turned around and gave me a big smile while I gasped for the frigid air that was no longer being warmed by my throat sufficiently. This isn’t about me realizing that if there’s a will there’s a way. This isn’t a “things will get better if we work at it” story and it certainly isn’t Jim and Huck. This is about awareness and understanding. This is about making the connection from Steve’s life story to the millions of others who are similar only because they are all victims of the prison industrial complex. This is about realizing that even a positive ending to Steve’s story, which we’re all hoping for, is insignificant compared to the systemic changes that are needed to avoid creating millions more down-and-out Steves generation after generation.


Filed under Poverty and the Justice System