Recently, NCES/ETS published a paper on public/private correlated data, "Comparing Private Schools and Public Schools Using Hierarchical Linear Modeling" (Braun, Jenkins, Grigg). It's an extremely technical paper, obviously intended for statisticians, given comments like, "And now I'll make it easier for you, here are the formula:..."
Shortly thereafter, there came along a refutation from Harvard Education Policy faculty, "On the Public-Private School Achievement Debate" (Peterson, Llaudet).
The debate is, in essence, not a debate. The data were analyzed and somewhat informative as to what truly influenced success, and the surprising results were that it actually wasn't whether the school was private or public; the Harvard faculty, who has a history of policy recommendations that are pro-private school, basically lampooned it.
The Harvard paper began, "On a quiet Friday afternoon in July of 2006, the U. S. Department of Education’s National Center for Education Statistics (NCES) released a study (Braun and others 2006 [hereinafter referred to as NCES]) that compared the performance in reading and math of fourth and eighth graders attending private and public schools." Cue the ominous music!
I can't fully follow the statistics in the NCES/ETS paper, but I don't agree with any of the substitutions in the Harvard paper, such as the statement -- I'm paraphrasing: "'books in the home' is too subjective and falsifiable a data set; we're using 'education level' to proxy for the student's home environment" because, you know, no one has ever lied about education level... and "forget Title I because that's biased towards public schools that have the funds to administer it, so we're replacing it with a regional "urban vs. rural" variable."
The issue of Title I came up because of course it was a way to measure what neighborhood the kids were in, as in "neighborhood public schools."
"Tuition rates at private schools are little more than half the expenditure rates of public schools, running as little as a few thousand dollars a year, making them affordable to families of only modest means."
This comes up in the context of his allegtions that people still have a "Catcher in the Rye" (his term!) worldview that private schools are for the wealthy. I'm really not sure what he thinks "modest means" are, but to be Title I eligible, a family of four makes under $25,000/year.
And the thing that really got me to post: the Harvard paper goes out of its way to point out the standard "only use these results within the bounds of experimental validity" disclaimer in the NCES report is an admission of sloppy research! As part of its executive summary, the NCES report cautioned its readers that:
“the data are obtained from an observational study rather than a randomized experiment, so the estimated effects should not be interpreted in terms of causal relationships. . . . Without further information, such as measures of prior achievement, there is no way to determine how patterns of self-selection may have affected the estimates presented.” (NCES, p. v.)
But, in the news coverage and public discussion that ensued, such cautionary words were often ignored. The politician-academics at Harvard raised the Red Herring: After all, if the data cannot be interpreted as providing information on the effect of school sector (public v. private, public v. Catholic, and so forth), then what was the point of the exercise in the first place and why would a government agency commission such a project?
I hope that the debate can emerge as a real debate. The issues are relevant, and inform policy. In the future, I hope that the leading academic institutions in the United States can refrain from politicizing their research, and instead focus on solving the issues at hand.
However, it brings out the importance of the paper in the last post, which explains that even given school choice, the choices parents make are biased by their sample set, which is in turn biased by the people in their social network with whom they have strong ties.