Apparently U.S. News has decided to pretend that its botched rankings rollout never happened. That transparency is overrated, or at least, that it only applies to law schools, not to U.S. News. And that law schools are to blame for the rankings fiasco this year.
Let's recap. On April 11th U.S. News posted a "preview" of the new law school rankings, including a list of the top 14 schools; it said the full list would be available April 18. This was likely an attempt to (1) preempt possible leaks, and certainly (2) a ploy to generate hype for the new rankings. On April 14th, U.S. News announced it was delaying the rankings release to April 25th due to "an unprecedented number of inquiries from schools." On April 19th U.S. News announced it was delaying the rankings release indefinitely, blaming "requests from law and medical schools to update data submitted after the collection period" and pointedly noting "[t]he level of interest in our rankings, including from those schools that declined to participate in our survey, has been beyond anything we have experienced in the past." On Monday U.S. News emailed law schools with the new rankings, and they were publicly released last night accompanied by a bland statement. Today, their letter to prospective students says nothing about the problems.
U.S. News has spent the last month or so suggesting that the fault for the delayed release lay at the feet of law schools. If only those pesky schools had done our work for us and given us their data, everything would be fine - but they didn't. So hypocritical of the elite, free-speech hating ivory tower class, implied U.S. News.
In fact, that was not the problem at all. For the second time in three years, U.S. News had serious problems with its data, and with its methodology.
Thanks to the boycott, U.S. News couldn't use its old methodology. With over 60 schools boycotting, they simply didn't have the data they relied on to generate the rankings. So they moved to a system that relied only on publicly available data, and their assessment survey results. In theory, this should have made creating the rankings quite simple. Everything that U.S. News needed was here, here, and in their voter responses.
However, being the mess they are, U.S. News managed to screw up what should have been a simple, straightforward exercise. For one, defying all rational explanation, they decided that for some metrics they would use the data submitted by law schools who still participated in their survey; for non-participants, they would input the results from ABA data. There's no justification at all for a mixed system. First, you are inviting possible errors from law schools submitting their data to U.S. News. Second, you are giving law schools an opportunity to submit data which is better than what is reported by the ABA- when the data should be the exact same. Third, now you're manually combining data from two different sources, which is just asking for transcription or arithmetic errors.
Which is exactly what happened. There were substantial discrepancies between the data U.S. News had, and the data available through the ABA. These discrepancies occurred across multiple metrics. Schools were being listed with job outcomes 5% below what they really had. First-time bar passage was off by 10% for some. Student faculty ratios were both over and under-estimated. We're not even certain everything has been fixed- the University of Wisconsin might want to check if U.S. News is using the 90% first-time bar passage rate listed for them instead of the 99.6% number it should be.
These errors weren't the fault of law schools. Non-participating law schools didn't send U.S. News the wrong data. The University of California- Los Angeles did not submit data to U.S. News; its non-participation didn't force U.S. News to seemingly include library staff in its calculation of UCLA's student to faculty ratio. That was because of U.S. News' sloppy quality control. So of course when law schools saw that U.S. News was using the wrong data, they reached out. Even if your school doesn't participate in the rankings, it's entirely reasonable to be annoyed that U.S. News will be listing factually incorrect information about you on its website.
Because of that sloppiness, there were major changes between the initial embargoed rankings and the final public version. 27 schools moved up or down by at least 4 ranks. One school saw its ranking jump up by 17 spots from where it had been on April 11th's first release. And embarrassingly for U.S. News, there were changes in the top 14 rankings it initially previewed, with Harvard, Duke, and NYU each moving.
After six months of being under fire, of all years, U.S. News had to know that its reputation was at stake with this release. And with its reputation on the line, this is the quality of product it put out. We're not even talking about issues with the methodology (you can read some thoughtful analysis of the methodology here). We are just talking about the data collection and calculation. U.S. News could not copy data over from a spreadsheet someone else already made for them. And this is the organization that describes itself to prospective students as having "earned...your trust" with its "quality journalism, data and information, and consumer advice..."?
U.S. News tried to shift the blame to law schools with its statements. How is this the fault of law schools? Were some schools wrong in asserting U.S. News weighted job outcomes incorrectly? Yes. But it would take all of five minutes for U.S. News to clarify that wasn't a problem. Instead, it ignored the issue for weeks, no doubt hoping the scale of its errors wouldn't force it to change its previewed top 14 schools. When it did, U.S. News seems to have decided that avoidance would be the best strategy.
Silence is an interesting decision from an organization which claims it is committed to "fight for...transparency and accountability...". We wonder if it is either transparent or accountable not to acknowledge your errors, explain what happened, and show how you plan to avoid such problems in the future. After all, U.S. News claims that "tens of millions" rely on its rankings. Don't they deserve to know whether the organization they allegedly rely on is taking steps to ensure basic multiplication and division problems aren't going to screw up the results?
Of course, this has never been about transparency or accountability. It has been, and always will be, about U.S. News' profit and market dominance. But maybe if U.S. News spent less time willfully misconstruing the Constitution, and more time focused on its product, it wouldn't hurt its own ability to make money with embarrassing public gaffes. As for anyone who might think the rankings are a useful tool in their decision about where to go to school, ask yourself: in light of all this, are you really sure you want to rely on what U.S. News puts out?
U.S. News wants to pretend the past month never happened. It shouldn't be allowed to do so.