This is a great guest blog post from Warren Buff, who took it upon himself to crunch the numbers and look at the data behind Law School Numbers.
Like many law school applicants, one of the first resources I found when I began my quest for admission was Law School Numbers ("LSN"). I pored over their scatter charts trying to discern which schools would be open to me and what kind of LSAT score I would need to be a viable candidate for admission. Once I found the aggregate data at MyLSN, I dug even deeper. But now that all my applications are in, I devoted a little time to a question I (and many other applicants) should probably have asked up front: Just how representative are the applicants on LSN?
A simple glance should be enough to know that not every applicant posts on LSN, but even without complete data, some strong patterns emerge: some schools have GPA floors, some are open to candidates with the right LSAT regardless of GPA, while others accept, waitlist, and reject similar candidates with enough frequency that you pretty much have to believe them when they say they take a holistic approach (or at least that additional factors are in play). Aside from the very strongest patterns, though, LSN data points can only confirm that admission is possible, and do little to nothing to explain why one candidate in a numerical range was accepted while another was rejected. Furthermore, comparing LSN data to the LSAT/GPA tables some schools publish through LSAC, some of the admissions percentages are just plain wrong.
These sorts of observations led to the question: How much of the applicant pool is on LSN? An old post from Mike Spivey indicated that LSN represents around 4-5% of the applicant pool. As he stated when this came up, though, that number is based on research a school did around six years ago, and the percentage could well have changed in that time. That was enough to prompt me to bring hard data to the problem, and I spent a day counting applicants and acceptances (remembering to go back for the WL, Accepted applicants, who sort separately from the Accepted applicants), then putting them into a spreadsheet along with the total numbers of applicants and acceptances the schools reported to the ABA. I did this for the top fifty schools in USNWR’s most recent rankings – check out all of the data here.
The first obvious conclusion we can reach is that larger numbers of LSN users are applying to higher-ranked schools. That isn’t completely generalizable, as LSN applicants represent a smaller percentage of Yale applicants than at other top schools, but from Stanford to Cornell, they generally represent 12% or more of total applicants, and maintain 10% or more down through around USC. After that, the applicant percentages trail off, and they do so more quickly at schools that aren’t in major coastal cities. By the time you get down to the end of the top fifty, it gets a lot easier to believe that LSN might truly represent only 4-5% of the total applicant pool across all schools. Depending on what segment of schools you are interested in, though, that 4-5% figure could be misleading.
Secondly, the applicants on LSN represent a disproportionately large sample of admitted applicants. This is most pronounced with Duke, where the 15% of applicants on LSN make up over 35% of admitted students, but the pattern of LSN applicants being more successful than the complete applicant pool holds for almost every school I researched (BYU is the lone counter-example, where the tiny sample on LSN represents 4% of both applicants and admitted students).
So what does it mean for us as applicants? Just how big a grain of salt do we need to take with our LSN data? Fundamentally, it tells us more about what successful applicants look like than unsuccessful ones. Take Virginia, for example. We know what 25.4% of admitted students looked like last cycle, but we only have information on 11.3% of applicants who were rejected or languished on the waitlist. That’s a tremendous gap in our understanding – our picture of admitted students is more than twice as complete as our picture of unsuccessful applicants.
Even acknowledging this limitation, at the top of the USNWR rankings, LSN can tell us a lot. Certain numerical patterns emerge in the accepted applicants, and while seeing that 80% of the hundred or more similar applicants on LSN were admitted to[say hypothetically] Princeton Law over the past few years most definitely does not tell us that we have an 80% chance of admission, it does tell us that our numbers make us strong candidates, and that the hard work of putting together a top-notch application to Princeton Law is well worth our time, and that we might want to prioritize this effort over an application to a school where only 15% of similar applicants were admitted. We might well decide that a fishing expedition to that more selective school – or even one where only a single historical applicant with our profile has been accepted – is worth the time and money, but we should do so with the clear understanding that the application is a longshot.
Once we get outside of the top segment of the law school rankings, however, we have to use an increasingly large number of years to find a decent sample size of similar applicants. With such a broad net, we have to include more data of questionable value to our current cycle. We should approach smaller data sets with greater skepticism, especially in light of the bias in favor of success on LSN. Following the same simple math we did for Virginia, when we get down to the paucity of data around a school like Utah, while we know what 6.3% of admitted students look like, we only have a picture of a mere 2% of unsuccessful applicants. Even at a school like Fordham, where we have significantly more data, we can examine 10.5% of admitted students as opposed to only 4.7% of unsuccessful applicants. When we know so much more about successful applicants than failed ones, we have to acknowledge that we may not really have a complete enough picture to understand the difference between the two, and as these examples demonstrate, the bias towards admitted students is compounded by the bias towards top schools.
So there are the two fundamental flaws with LSN as a source for predictive data regarding our own application cycles, even before we consider the differences brought about by changes in school leadership and applicant pools from cycle to cycle. We know much more about applicants who were admitted than about applicants who weren’t, and we know much more about applicants to the top dozen and a half schools than about applicants to schools outside of that elite range. Even with those limitations, LSN remains one of the best resources available, but we owe it to ourselves to be realistic about the quality of the data we are using and its limitations as a predictive tool.