Pro-voucher “researcher” isn’t above distorting results

The American Prospect’s weblog, Tapped, had a terrific item yesterday about some controversial research indicating that publicly-financed tuition vouchers to religious and other private schools helps African American children.

As Tapped explained, Paul Peterson, a long-time advocate of public aid for private schools, published a study showing “that black students who had won in a 1997 lottery for privately financed school vouchers scored several points higher than those who hadn’t.” The study, which claimed to track the success of students in New York City, has been used frequently over the last year by other voucher advocates to suggest their ideological opponents are standing in the way of African American students getting a better education.

Princeton economist Alan Krueger did some follow-up on the Peterson data and came to very different conclusions. In fact, Krueger has published a study of his own indicating that Peterson’s results have been “blown out of proportion” and that “For the most representative sample of black elementary school students…offering a voucher had no statistically discernible impact on achievement scores in the New York City experiment.”

What Tapped failed to mention is that this is the latest in a series of serious questions surrounding Peterson’s poor scholarly work. In fact, a close look at Peterson’s work on vouchers over the last decade reflects an awful record when it comes to quality social science.

Paul Peterson frequently appears in the media and at conferences as an expert in education reform. Despite his academic credentials, however, Peterson is more an advocate than a scholar. His research, which is sometimes published before it goes through a peer review process, is more often than not funded by advocacy groups hoping to produce materials to bolster their political agenda.

Peterson’s research in New York, for example, was called into question just weeks after his study was published. Mathematica Policy Research, the company that gathered the data for the New York students in Peterson’s research, issued a statement calling Peterson’s conclusions about the benefits of vouchers “premature” and saying the data shows no difference in academic performance between voucher students and public school students.

“[S]tudents who were offered scholarships to attend private schools as part of one of the nation’s largest private voucher programs performed about the same on standardized reading and mathematics tests as students who were not offered scholarships,” Mathematica said in a statement. Mathematica also noted that many of the low-income parents who were offered vouchers did not use them because they still could not afford private school tuition or were unable to find a private school willing to accept their children, a fact that Peterson conveniently omitted from his report.

This wasn’t the first time Peterson’s results have been called into question. Four years ago, Peterson published a report claiming that Milwaukee’s voucher experiment had boosted test scores for students who left public schools to attend private schools with a voucher.

Soon after, a University of Wisconsin professor hired to evaluate Milwaukee’s program concluded the opposite — vouchers failed to improve the academic performance of participating students.

John Witte, a professor of political science at the University of Wisconsin and the state’s official evaluator of the voucher program from 1990 to 1995, reported his findings in the October 1999 issue of Phi Delta Kappa magazine. He reported that predictions by voucher boosters that students in the program would do better than their counterparts in the public schools had not come to pass.

Concluded Witte, “Thus there was no pattern of superiority of choice students over [public school] students or vice versa. And these results held up with a range of more complex and diverse statistical models.” As for the conclusion reached by Peterson, Witte said, “[T]he problems with this result are so numerous that I don’t think anyone really believes it. At least, no one should.”

But voucher advocates did believe Peterson’s results. They took his faulty conclusions and ran with them. Indeed, Peterson’s data was used to justify passing a voucher scheme in Florida in 2000 and is still being used by advocates today in states such as Texas and Colorado.

The problem, at its root, is that Peterson’s work on this issue is tainted by an obvious bias. He breaks the Cardinal Rule of social science — starting with the conclusion you hope to find and then work backwards to prove yourself right.

In Peterson’s work in Milwaukee, he arrived at his chosen results by comparing a carefully selected sample of voucher students to a disproportionately small group of inner city youths in public schools. As Bruce Fuller, a professor at UC Berkeley said, Peterson’s research “is based on the less than 80 students who lasted four years in just three choice schools. Complete data are not available on the other 2,900 children who spent less than four years or who could not be tracked. Would we believe a scientist who claimed that smoking has no harmful health effects based on a study that simply tracked smokers who were still alive?”

This is terribly frustrating, to say the least. A serious debate has been underway for over a decade about whether to transfer tax dollars from public schools to private schools. Questions about education, religious liberty, race, and economics are raised by this public policy discussion and reliable social science data can be useful in reaching the correct conclusions.

Peterson, however, has abandoned the role of neutral researcher. It’s a shame.