Guest writer Corey McMahon is a Northwestern computer science student and former journalism student. He has covered NU sports and also writes a blog applying statistical analysis to sports. Check out his blog — blog.coreymcmahon.net — to see more of his work, particularly his work with baseball statistics. Follow him on Twitter at @cdmcmahon.
Over the last few decades, our ability to understand what causes success in sports has increased remarkably. Baseball is the clear winner in this regard, but football, currently our nation’s most popular sport, has lagged behind.
On Monday, Bob Valvano spoke at length on his ESPN radio show about the poor job that so-called expert pundits — a group of which I would consider him a member — do when it comes to forecasting team success early in the year. The Preseason AP Poll, conducted among sports journalists and released every year in the build up to the first kickoff, rarely looks anything like the end-of-season AP Poll, released following bowl season.
Last year, in fact, among teams listed in the Preseason AP Poll, they ended up more than 12 spots away from where they started[1]. That’s not very precise for something that carries such attention and even financial stake. Furthermore those misses leaned heavily in one direction. The pollsters showed a clear bias to overrate teams[2].
I have a theory as to why that is, though with no way to test it I couldn’t say for sure. The majority of news that comes out of a program over the course of the offseason is positive; the quarterback is developing his mechanics or the coaches and trainers are implementing a new conditioning system in camp. And while the logical jump is there that this would lead to success is made, there really hasn’t been and conclusive evidence that these things lead to success — and there probably can’t ever be such evidence because it’s difficult or impossible to test this kind of evidence[3]. This combined with all kinds of historical and regional biases lead to poor forecasts[4].
Bob Valvano’s conclusion in his radio show was to “let the cake bake” — that is, don’t make forecasts or predictions until the teams have played a significant portion of the season. I say the real solution is to use a different cake recipe. I don’t believe (though I could be wrong) that there doesn’t exist a decent way to predict football teams’ success better than the Preseason AP Poll (admittedly, a relatively low bar).
I’ve started building up a database from previous years to test these ideas and already, even though my sample size remains small at this point[5], I can start to see that there are factors that correlate well with how a team performs. For starters, there’s the previous year’s performance. Below is a scatterplot identifying the relationship between a team’s win-loss record (identified as wins minus losses) and the same measure from the previous year.
A positive correlation is clear. There’s a lot of variance, so alone it’s not useful, but if I can find many variables that correlate with team success, I can hopefully begin to think about building an accurate forecasting model. Another variable that correlates well with a team’s win-loss record is their yardage differential. But that’s not very useful, of course, because you don’t know a team’s yardage differential until the end of the year. Let’s look at the team’s yardage differential from the year before.
It’s much noisier, but there is still actually a slight correlation there, and one that I hope might tighten up as I add more data. The best correlation I found was using a statistic from sports-reference.com that they came up with in 2006 and aptly called Simple Ranking System. They have a great, lengthy write-up about the whole thing:
One equation for each team. The number just after the equal sign is that team’s average point margin. In plain English, the first equation says:
The Colts’ rating should equal their average point margin (which was +12), plus the average of their opponents’ ratings.
So every team’s rating is their average point margin, adjusted up or down depending on the strength of their opponents. Thus an average team would have a rating of zero.
Plotting a team’s win-loss record against their previous season SRS was better than against their previous season’s win-loss record or yardage differential.
SRS is even better at predicting itself for the next year — likely because the pitfalls that may cause a team to over- or underperform one year often stay the next — but that doesn’t tend to be important to most fans who want to see their team win on Saturday, so it’s not as interesting to try and predict.
All of this is to say that I believe there are factors that can be used to forecast success in a much more systematic and bias-free way than the AP Poll. Even though there is a lot of turnover in college football teams, we can use a team’s history — and likely the history of key returning players — to forecast the future. I say let’s give it a shot.
1. 12.6 to be exact and that’s using the generous assumption that every team that started on the Top 25 and ended the year receiving no votes was the top team among non-vote receivers, which means in reality, this number should be higher.
2. Part of this is unavoidable: you can’t miss high on whatever team is rated number one. But even on teams 10-25, they missed overwhelming by overrating teams.
3. Even more difficult is to determine whether what teams say about this kind of thing is even true, or whether they just want to say good things and avoid looking bad.
4. I do understand that the AP Poll has other value besides forecasting, that it serves as a “power rankings” list of sorts.
5. At this point, I’ve gathered data from the previous five years for Big Ten teams only. This allowed me to identify a few broad trends and figure out what exactly I want to look at. Going forward, I want to build up the database using power conference teams and eventually all of Division I, but doing so is slow and tedious given the stratification of college football data online.