What Stat Geeks Don't Get
As stat geeks start to take over football commentary on the web, Aaron asks for a bit of reflection from the Stat-obsessed fandom that seems to think that numbers are always the answer.
By PackerAaron
This is good fodder for the weekend.
Recently, a bunch of sports-related talking heads got up on a stage at M.I.T. for something called the Sloan Sports Analytics Conference. The panel of note, at least for me, was "What Geeks Don't Get: The Limits of Moneyball" which featured Jonathan Kraft and Bill Polian along with Mark Cuban and Bill Simmons.
More interesting? The take of Aaron Schatz, founder of Football Outsiders, a site that most serious football fans are familiar with. Schatz, needless to say, was not very happy with the proceedings. Money quote:
The most annoying part, for me, is when Polian basically says "there's nobody doing analysis worth listening to." This is about an hour after his son had personally told me that everyone in the Colts front office reads FO and they generally feel we're on the right track. So I was a bit pissed off.
OK, first things first.
I've been reading Football Outsiders for just about the entire time they've been on the web. I'm a big fan. They were the first people to give you something more than "you need to establish the run" as commentary. I met Aaron at the Combine, we had dinner and got along famously. He's a passionate football fan, as dedicated to the game as he is his work, which is superb.
But that doesn't mean Polian doesn't have a point, even if he made it in his characteristically abrasive fashion.
Even someone as dedicated to statistical analysis as Schatz is has to recognize that there's a limit to what we can do as fans. No matter how many times we watch our DVR copies of games, no matter how many times we log in to NFL Game Rewind, no matter how many incredibly interesting though thoroughly perplexing stats we come up with on our own, we are essentially hobbyists dabbling in someone else's profession.
That's not to say we don't have things of consequence to say and it doesn't mean we shouldn't be listened to. I understand Aaron's frustration - he's worked long and hard over the course of the last decade and people that matter in NFL circles know his work. I talked to tons of people in Indy who are regular readers, both in the NFL and in the journalistic community. And to be fair, his site isn't strictly numbers - what gets lost is that there is some downright amazing writing on there. (Tanier's Walkthrough and Farrar's Cover-3 are simply weekly must-reads for any football fan)
But all the writing and all the numbers in the world by fans doesn't capture a complete picture of what happens on every play of every football game. The football people have that. Polian himself even eludes to this saying:
There's nobody out there working on quantifying all that goes on, 22 guys, 60 times a game
There's a reason for this - the NFL guards the coaches tape as though it were a state secret, which, for the NFL, it kind of is. The obvious inference here? If the stat geeks ever got a hold of the coaches film, they would pull the curtian back on Oz even more than they already have.
The football people, on the other hand and armed with said coaches tape, have every tendency, every formation, every medical report on every player, every piece of history of every coordinator in the league including, at the press of a button, a cut-up of every third down throw from the right hashmark that the slot receiver caught for a first down, etc.
All of this, and they've been doing it for longer than a lot of these stat guys have been alive.
What drives me to distraction about most stat-driven analysis is how its proponents hold it up as some kind of talisman that they and they alone are capable of understanding. They bandy about numbers as though they are their own facts rather than the part-to-the-puzzle they truly are. Even worse, stat geeks often fail to provide any context to the numbers they hold so dear.
To be fair, I know I sound like I'm completely anti-stats, completely anti-FO. This is far from the truth, and I love this from Schatz:
I take pride in the fact that Football Outsiders leads the league in couching our opinions in caveats
Knowing what you do and don't know is a very important thing - some stat geeks are better at this than others.
Enjoy the video and I'd love to get your thoughts on the subject.
PLEASE SUBSCRIBE TO OUR CHEESEHEAD NATION WEEKLY NEWSLETTER HERE.
__________________________
Comments (58)
holly
March 26, 2010 at 08:46 pm
Hmm, yes. It is a frequent flaw that we stats geeks justify our numbers by looking at other numbers. I am as guilty as anyone of looking for truth behind stats, but I am also painfully aware of just how little I know about this sport I love to watch.
I wish Polian wasn't quite so blunt about saying it, but he does have a point. And you're right - Aaron - the reason there's no one "doing analysis worth listening to" is that no one's been given the chance. Maybe the mystique is good for the sport. Maybe they just like holding the cards. Who knows.
March 26, 2010 at 10:13 pm
Eh, I dunno Holly. Sure, there is a lot we don't know but I also think the league and it's personnel like to make the game more complicated than it is. I think Aaron can be a bit too reverential towards the people who run the teams (I get it, to get respect you have to not only earn it but also give it.) I personally believe Aaron shortchanges people like himself and Schatz by referring to themselves as "hobbyists."
March 26, 2010 at 10:09 pm
Polian is so full of crap, but why should he care? He just gave a great sound byte that also looks good in a dialogue box.
-----
What I think Polian, and our Aaron, missed about the stat debate as it pertains to football is sample size. Unlike a sport like baseball, where there are 162 games and tons of atbats to verify the data, football seasons (and even player's careers) go bye in the blink of an eye. It's really hard to come to any strong conclusions from such a small sample. Also, matchups matter. Since the season is so short, all teams don't get to play one another. Thus, a lot of projections rely on common opponent or similar opponent and while this is somewhat helpful its basically just advanced tea leaf reading, imo.
-----
Another issue that is unique to football is context. For example, I've argued with a friend about who had the better peak between Favre or Warner. My friend points to stats, but I quickly remind him that Warner played in a wide open offense with 2 elite wide receivers (Bruce and Holt) and a HOF RB (Faulk.) Meanwhile, Favre had to deal with a motley crew of WRs, turning guys like Bill Schroeder into Pro Bowlers. Favre also played outside, in the elements, while Warner played in a controlled environment while with the Rams. So who's right? Who knows? My point is that stats don't really do much for me in football, because success in football depends on 11 guys doing their job correctly. If one lineman misses a block it can lead to failure.
-----
It's funny you posted this today because I have spent most of today (day off from classes) prepping for my 2 fantasy baseball drafts this weekend. When I prep for fantasy baseball, I head over to Baseball Prospectus, get their PECOTA projections and rankings based on my league settings and I am good to go. When I draft, I mostly stick to the rankings until the later rounds, when I will grab some young prospects I've either seen play or read up on and who have a chance to outproduce their projections. But by and by, established players generally meet their projections. By contrast, I pretty much throw stats out the window when I draft my fantasy football team (aside from RB carries) because success on the football field depends on so many variables and is a more team oriented game than baseball.
-----
Ultimately, my point is this: I am a total stat-geek when it comes to baseball and trust the data completely, but when it comes to football I remain more of a stat atheist. I know they're out there and probably contain answers, but I'm not convinced anyone has really figured out the best way to access these answers.
March 26, 2010 at 11:01 pm
So I watched about 20 minutes and Polian and Kraft made the points that I discussed above. Obviously, I think Polian's points about sample size is the crux of the issue when it comes to stats and football.
March 26, 2010 at 11:09 pm
One more note re: sample size. 2007 is a great example of this. If you go back and listen to Bill Simmons' podcasts with Aaron Schatz where they discuss the NFL playoffs, Schatz you'll note how every round they brushed off how the Giants were playing, relying more on "the data." Well, as a Packer fan living in NYC, I was very nervous going into the NFCC because that Giants pass rush was ferocious and they finally seemed to "get" Spagnuolo's defense. Long story short: this was not the same Giants team I had watched every Sunday for 15 weeks.
-----
Not to brush off what Football Outsiders does, because I think their product is excellent. I just feel that when it comes to football the stats serve as more of a complement to watching the games where in a sport like baseball, you could go into free agency and make decisions almost solely on statistics (assuming the player isn't a sociopath ala Elijiah Dukes.)
March 26, 2010 at 10:23 pm
If Holly doesn't know football, I am a total dork loser in comparison. Well, I guess I am anyway.
To be honest, I don't pay a lot of attention to stats. I paid quite a bit to you know who's and it misled me to think he was a stronger player than he was (and yes, I still think he was/is good). I watch the game, try to keep on top of what the play is and who should be where, and yell "GO PACK GO!" a lot.
I appreciate you educating me though. Oh, and Jersey Al.
March 27, 2010 at 12:06 am
Gotta agree w/ Polian on this. FO (and other stat-geek sources) can be innovative, insightful and highly informative, but DVOA and other advanced stats are largely backward-looking and only predictive on a macro level.
For example, you can count how many times Daryn College has a brain fart and forgets how to block, but you cannot predict when it'll happen. If only...
If turnovers or anomalously poor red-zone efficiency cost a team victories despite otherwise stellar statistical performance, the coach's next gig is still going to be as a quality control asst. in St. Louis or Detroit, bitchin' DVOA or not.
Polian's reference to 22 guys, 60 times a game is really on the money. Football success is determined by the semi-subjective outcomes of gobs of 1-on-1 and 2-on-1 battles, but the quants' can only analyze the quantifiable.
Charting a 20-yard run on 2nd & 7 from your own 45 from the left hash with 7:42 remaining in the 2nd qtr while you're team's behind by 6 doesn't account for the pulling guard taking just the right angle to block the OLB, or that safety was thinking pass and put himself out of position.
In a nutshell: there are no winners in the Pythagorean playoffs.
PackerAaron
March 27, 2010 at 12:50 am
You absolutely nail it here. Well done. Had something akin to your 2nd and 7 example in mind but couldn't get it in the post while at work - you sum my thoughts up perfectly here. Thanks much :)
March 27, 2010 at 12:22 am
People can come up with statistics to prove anything, Keith. 14% of people know that.
March 27, 2010 at 01:26 am
But Frankiln,
73% of the 14% say stats only have a 25% influence on their stance... common knowledge.
GBP 4 LIFE
March 27, 2010 at 03:17 am
Was I not clear that I do not think stats work as well for football analysis as they do for baseball analysis?
March 27, 2010 at 12:23 am
Coming from soccer, stats in football are overwhelming. They say a lot, the right ones. But they'll never say enough.
Until someone comes up with stats for personality, mood, team chemistry, refs interference, and luck, a team strategy cannot be BASED by stats, rather be assisted by it.
Stats are so big in US sports that it sometimes overshadows play itself...
March 27, 2010 at 12:31 am
"Coming from soccer"...
Well, I'm glad we can offer you American football, because I for the life of me CAN'T see any redeeming qualities in "European Football".
I have tried.
Several guys I work with are Sounder FC fans. I've watched a few games with them over a few brews, needless to say the beer was all that I enjoyed.
I don't get it.
Millions of people worldwide do though, so there must be something there.
I think I was just spoiled with the best sport and the best franchise on the planet from an early age, subsequently, nothing else stands a chance.
GBP 4 LIFE
March 27, 2010 at 12:40 am
Manchester United 4 LIFE!
March 27, 2010 at 12:41 am
That's what's wrong. You've been watching too much European football. You should watch more South American football.
The only thing worth watching in Europe is Barça.
March 27, 2010 at 01:11 am
By "European Football", I just meant Soccer.
I can honestly say that I will never give Soccer another minute of my time. I'm probably just an ignorant American in that regard... it's bliss.
Draft here yet?
GBP 4 LIFE
March 27, 2010 at 03:19 am
HALA MADRID!
March 27, 2010 at 02:22 am
"Well, I’m glad we can offer you American football, because I for the life of me CAN’T see any redeeming qualities in European Football."
This happens to be inversely true for almost anybody that holds soccer as their true sport though. People who grow in their lives watching soccer, probably wouldn't understand the appeal of Football at all either, even if they tried like you do on "European Soccer".
I think all of this happens because of two major factors, which are lost in the "translation" if you will from one game to another:
1)Knowledge of the game. For example, it is important to know why each position of each player is important to the game, and not just what the general goal of the game is. You need to know why certain plays are penalized and why some others aren’t.
When you have more of a general understanding about the game, I think you can actually enjoy it.
2)Broadcasting color. This is very important, because if you don’t listen to a proper play by play commentator, then all the magic is gone. It’s just like listening to Joe Buck the guy has no spark and he does NOTHING for me in a live Football “primetime” setting. As it is the case I’ve heard some soccer broadcasts in English, in which you could just fall asleep, and I would feel sorry for the people who actually has to watch soccer that way.
“I think I was just spoiled with the best sport and the best franchise on the planet from an early age, subsequently, nothing else stands a chance.”
Now this line holds completely what I feel about the sport and the Packers, because while I’m not from the US or started watching Football until I was 15 or so, I DO think nothing else stands a chance, not even soccer.
Now about stats, I generally think that YOU HAVE TO PLAY THE GAME OUT, because stats alone aren’t going to decide the outcome of a game. I mean I love a good stat once in a while, and most of them are peculiar and entertaining, just like the “Defensive Hog Index” which Aaron pointed out little after the regular season ended. But by stats and matchups alone we had that Dallas game last year LOST. Then the Pack went to play and won superbly in a game which not many where hopeful to win.
-Carlos Jurado
Go Pack Go!
P.S. Over a year reader, first time poster. Please excuse any writing or grammatical errors, as I mostly read or listen to English, I haven’t wrote a paper for discussion in a while :)
PackerAaron
March 27, 2010 at 03:30 am
This is brilliant - and for the record, I can't wait for the World Cup to get here. ;)
March 27, 2010 at 09:57 pm
Agree Aaron, The World Cup action is out of this world. I was stationed in Germany when they won the World Cup back in 1990. The entire country all partying at once. It was a one of the greatest experiences of my life.
As for this upcoming World Cup. I can't wait for the USA/England match. England is a total mess right now.
PackerAaron
March 27, 2010 at 10:09 pm
USA/England is going to be World War III in my house...
March 27, 2010 at 11:31 pm
Simple, whomever is rooting for England... by them a plane ticket so they can go live there.
I can't think of any reason for an American to root against America in any capacity (although it IS their right... as an "American"). It's just, how shall I say... UN-American.
GBP 4 LIFE
March 28, 2010 at 12:50 am
Nice twist. First this is a stats post... The na douche makes it about soccer.
And now, out of the blue... TCHAM! BAMN! A patriotism topic!!
;)
PackerAaron
March 28, 2010 at 03:28 am
Um, my wife is English and all three of my daughters are half-English. Sending them to live England would, in a word, suck.
March 28, 2010 at 05:09 am
Haha,
I was just tryin' to get a rise. I figured it was something along those lines... if everyone liked the same team, sports would be kinda pointless.
Really though, you should raise your girls "correctly", if you get my drift. USA, USA, US...
GBP 4 LIFE
PackerAaron
March 28, 2010 at 12:02 pm
They're Packer fans. That's all that matters.
March 28, 2010 at 01:40 pm
Aaron,
on a realted topic…. my daughters are half-kuwaiti…my oldest (9) is a die-hard Packer and Brewer fan. it confuses my poor mother-in-law so much. We went to visit her in Saudi arabia last year and my daughter was talking all about Aaron rodgers and trying to explain her grandma all about football. My mother-in-law, who can’t speak a word of English was soo confused..lol
PackerAaron
March 29, 2010 at 12:01 pm
That is brilliant Matthew.
March 28, 2010 at 06:32 pm
True dat.
March 27, 2010 at 02:55 pm
great post and so true.....I never ever liked futbol (south american/european) mainly due to the fact that I never grew up watching it...no atmosphere. But this changed when I lived in Istanbul for 3 years (very close to Fenerbace Stadium). Needless to say I was a convert after going to my first match. Atmosphere is very similar to a Badger football game (in the student section).....once you watch quality futbol matches it is an amazing sport. Also was fortunate to go see some matches down in Medillin, Colombia (Postbon). Great sport and world cup will be great!!.
Go pack and Cheers to all my Packer friends from around the world
March 27, 2010 at 06:52 pm
Okay, my last post about soccer... Turkey stadiums are the closest to a South American atmosphere. It's the fans chanting for 90 minutes nonstop. IT's the colors, the flags and the lights. It's beautiful.
Until someone throws a chair at the opponent's team, that is...
March 28, 2010 at 05:22 am
CJ,
Nice post. Nobody gives 2 shitz about spelling around here if you're making sense, which you certainly did. I totally agree with your take. Kudos.
GBP 4 LIFE
March 27, 2010 at 01:14 am
On a side note,
Any MMA fans here?
Tomorrow's card is gonna be f'ing S-I-C-K!
GBP 4 LIFE
March 27, 2010 at 10:40 am
Dude, Love the ufc. Went to see 110 when they came to australia and the atmosphere was amazing. Am looking forward to 111 and although gsp is a ledgend i am rooting for dan hardy and i also hope carwin knocks frank mir's teeth out.
March 27, 2010 at 02:49 pm
GSP is amazing and one of my favorite athletes in any sports. Just a nice human being as well. I'm a huge MMA fan and have been to my share of fights..mostly supporting Rofus Sports out of Milwaukee. Eric "Red" Schaffer, Pat Berry, Razak Al-Hassan and up and commer Anthony Pettis all train out of there...and all packer fans :)
March 27, 2010 at 08:41 pm
Matthew,
Good to know those guys are GBP fans, I'll pull for them.
I'm kind of the same way with the guys from the Pat Miletich camp. It's pretty close to where I'm from originally. He's had a lot of studs go through his doors... Matt Hughes, Tim Sylvia, Jens Pulver, Robbie Lawler, Mark Coleman, Spencer Fisher, Jeremy Horn... to name several.
March 28, 2010 at 01:35 pm
FitzCore,
yeah Miletich is on a whole nother level with his stable of fighters. Go pack!
March 27, 2010 at 08:34 pm
Very cool. I was so impressed by George Sotiropoulos' total "ownage" of Joe Daddy, dude's gonna be hard to deal with.
For me, gotta go with GSP. I just love the way the guy handles his "bitniss". Should be a great fight though, Hardy's game.
With you on Carwin. I've been looking for a Carwin vs. Lesnar matchup for quite a while.
Getting pumped.
WOOT!
March 28, 2010 at 04:38 pm
G! S! P!
March 28, 2010 at 06:31 pm
That was nothing short of TOTAL DOMINATION. I just wish GSP would have stopped it, nonetheless... ASS-WHOOP'N.
PackerAaron
March 27, 2010 at 04:00 am
I have to say, having finally had the chance to watch the whole thing, I agree with Polian almost 100 percent.
March 27, 2010 at 12:28 pm
Well.. some people just appreciate stats more than others. I can't find it now, but I know that I've read that Belichick has a stats guy who spends the entire game examining statistics and probabilities of play calls and success rates. When push comes to shove and the game is tight, the only guy that Belichick wants to hear from is the stats guy. That's why he makes calls like going for it on 4th and 2 at the end of the game to try to close out the Colts, instead of punting and playing the field position game. Granted, it doesn't always work, but statistics aren't about what will work every time, it's what is most likely to work in a given circumstance.
Don't get me wrong, it's not all about stats, but they definitely can augment what a human can see.
http://freakonomics.blogs.nytimes.com/2009/11/16/bill-belichick-is-great/
March 27, 2010 at 02:43 pm
Bellicheat does more than monitor stats during a game!
March 27, 2010 at 01:25 pm
Everybody lets keep this in perspective. Now Bill Polian knows what he does. However, there was a SERIOUS debat to go with Manning or Leaf. This guy still has a job because he guessed right. If he guessed wrong we would not be talking about him today.
So why these guys know more than we do, they should they spend 12 hours a day EVERYDAY on this stuff with better info, more at stake, ect, these guys are not the best and brightest in the world. They are just regular guys who have an eye for talent, a ton of money, or some other good attributes. However they are just guys/girls nothing more nothing less.
March 27, 2010 at 02:12 pm
Statistics are like a bikini.
What's revealed is interesting, concealed vital.
March 27, 2010 at 02:38 pm
Just watched the whole video. Was more interesting from a business perspective. Great link, thanks.
March 27, 2010 at 02:45 pm
great topic and something very close to my heart. I agree whole-heartedly with Polian and Nagler's thesis :)
Exhibit A) Aaron Kampman against Minnesota last year. I heard many people talk about how Kampman was coming on right before he got hurt. But really he was a complete no-show against a rookie RT in the 2 most important games last year for the Packers.
March 27, 2010 at 04:35 pm
Polian really shouldn't criticise stats analysis when he uses phrases like "pretty strong likelihood" and "going to have a good chance". When talking about "4th&2 gate". Likelihood based on what?
Good chance based on what? Stats or Gut? If it's gut, just say it. Don't try to quantify it.
March 27, 2010 at 04:56 pm
Bingo. There are many people whom abhor stats, and down play them, but yet rely on statistical analysis for many crucial decisions and functions- whether they fully realize it or not.
The world runs on stats and data. Yes, there can be massive failure if the stats cited aren't in proper context or the data is incomplete, or even if things just don't work out the way the odds suggest is most likely. But then, the same can be said about a NFL scout/coach/GM's appraisal of talent from film study or watching a prospect play live.
March 27, 2010 at 04:45 pm
I've been stabbed at by Aaron about stats in the past. However, I've never tried to used statistics in football as a predictor of future outcomes.
I think that often what we believe to be true is tainted by our perception at the time- we often "feel" like someone is out-performing someone else. This is why I like to see some numbers from time to time.
Last example where I cited numbers and was jeered for it; when everyone was saying Ahman Green was performing better than Ryan Grant, especially in the screen game. I understand WHY people FELT this should be true- Grant seemed lack-luster until Green stepped in, and Green looks more physical in the inside running game. He's always been regarded as a great screen RB, while Grant has been regarded as poor. But the numbers- records of what HAD transpired from last year, showed that the *average* production of Grant was better than the Average production of Green, even in the screen game. Of course, I was lambasted for the comments (I think Aaron commented on how many more attempt Grant has had compared to Green, which was why I used yardage averages instead of yardage totals.) My only point- Just as stats alone don't tell the whole story without strict context, sometimes what you perceive is tinged by personal bias as opposed to being purely what occurred.
March 27, 2010 at 05:33 pm
Oppy, like the breakdown and I think stats are great to in addition to what Polian states. i agree with biases and how sometimes it can mess up our perspective. But there is also the point that stats can mess up reality also.
March 27, 2010 at 06:33 pm
I just brought soccer in because in soccer, and any non-US sports in general, stats aren't as big a deal as it is in US. I have no idea what's the avg possession % my team has, or what's it's rank in steals... I do know how many goals it's scored, who's the leading scorer, etc...
I'm never mentioning soccer again in football posts. Sorry, people!
March 27, 2010 at 08:43 pm
Haha,
You are forgiven.
March 27, 2010 at 08:11 pm
Thank god this wasn't an ass kissing of industry insiders or an angsty rant on the under-valuation of outsiders.
PackerAaron
March 27, 2010 at 08:40 pm
Is that what you've come to expect? (Hangs head in shame...) ;)
March 28, 2010 at 01:38 am
Linear regression doesn't work for football. Variables, too damn many variables.
_____
Milano v. Madrid - guarenteed riot within the first five minutes.
_____
Austrailian football - mans sport! No one has any teeth and their nostrils are closer to their ear than the middle of their face. No concussions though!
March 28, 2010 at 07:45 am
(1) I know it's shocking, but there's no reason that qualitative and quantitative methods can't be synthesized in a single analysis. Methodological diversity is a healthy thing.
(2) I'm deeply against Aaron's women being deported to Merry Old England. Plus, I've seen her wear a Packers cap in photos. Two years ago, she probably didn't ask questions like, "Are you sure you want to keep rooting for this team? They keep losing by four points or less." My partner is an American. That's one of his favorite questions. Of course, I don't want to send my partner to England either, unless there's a ticket for me and lots of sightseeing involved.
March 29, 2010 at 07:51 pm
This is the best post in the thread.
You are exactly right. I'll take it a little further and say that stats and film review play different and complementary roles. Stats give you a macro view. It's important to have a 30,000 foot view.
You wouldn't (and shouldn't) base a critical play call primarily on statistics. On the other side of the coin, you line up all the genius GMs in the league and have them predict next years' standings, and you compare those predictions to FOs predictions. The FO guys will kick their ass every year.
Let's also keep in mind: Bob McGinn knows 3 scouts that this time last year thought that Jake Delhomme and Vince Young were better QBs than Aaron Rodgers. Did the tape lie? Did these scouts discount statistical prodution a little too much?