Pro Football Focus: How Do They Put Their Numbers Together?

In recent years, people in and outside of the NFL have tried to figure out which numbers/statistics are discoverable and relevant to a teamwork-oriented game like football. Over the last year, I've seen more and more people link to a website called Pro Football Focus. It's a website that claims (emphasis theirs): "For every game we analyze and grade every player on every play to provide you with the most in-depth statistics you can find anywhere outside the team's film room. "

The numbers they have are very detailed but beg the question: How can fans put together detailed numbers grading players based on game-charting from television angles (and not coaches' tape)? And how would you be able to ascribe blame for plays if you don't know what the play was designed and what the players were supposed to do? Their site looks at some of these questions in their grading section, and they are very good about responding to emailed questions.

For those wondering more about the Pro Football Focus people and their methods, the following is a Q&A I had with Sam Monson of PFF:

Who are the Pro Football Focus focus people?

Stephanie Stradley: Who are the people from Pro Football Focus? What is their background and how did they get into doing this work?

Sam Monson: There's a team of seven full-time staff with other contributors helping with the various things as their time permits. None of us are, have been, or will ever pretend to be, scouts; we are simply dedicated fans with a deep passion for football who utilize the skills and disciplines we learned and use in our "other" jobs to produce the information you see on the site (and a whole raft of other data you don't). We see ourselves, first and foremost, as football information provision specialists (not journalists or media), and this is the premise on which our business model is based.

Stradley: How many people do you have charting games, and how long does it take to chart each game to get such detailed data?

Monson: During the season 95 percent of the information gathered is the work of four people working in the order of 70 hours each per week. Each game takes an average of 15 hours to fully analyze, grade, error check and put on the site.

Stradley: Pardon my bluntness about this, but how on earth are you bankrolling seven full-time staffers? Even assuming extreme poverty wages, that would be some serious money. Your site has no advertising, no apparent backing from a deep-pocketed company and no book/magazine deal. Are you a front for Drew Rosenhaus or something?

Monson: Simply put, we're not bankrolling anyone anymore. Before this took off Neil Hornsby (who is effectively the site's owner) did pay a number of us until the commercial viability was understood by everyone concerned. Neil runs his own Business Consultancy Company and created and initially funded the site "as an expensive hobby." We all understood that was financially untenable moving forward, so we have formed a new company of which we are all shareholders and unpaid employees. We all love what we do, believe we are really good at it and see a huge amount of business prospects moving forward.

We need funding and are in the process of working on a host of opportunities that have arisen out of our success last year. We understand that what we do is unique, are so good at it that quite knowledgeable people (including NFL insiders) can't believe we can achieve what we do from TV and the vast majority of feedback we receive is positive. In the last year our readership has grown 25 fold and we would love to turn this into the financial stability that will allow us to continue to provide this information free of charge to the fans.

I can remember Neil telling me an NFL information director had called him to ask what methodology we used as a game he had checked only had two errors and was absolutely astounded when Neil told him it was standard TV coverage.

On accuracy and limitations of data ...

Stradley: There's been a discussion on both my Houston Chronicle Texans blog and on other sites about how accurate your data is. That with game charting data, it is much more accurate to track results than it is to evaluate specific player performance given the limitations of what you can see on a TV screen and not knowing how the play was designed. It's hard to apportion fault in a team game, and those who chart games regularly, know how difficult it is.

Monson: What we do can reasonably be split into two areas: Game Analysis or our analysis of how well each player performed on each play and Player Participation (PP) or which people were in on each individual play, which position they played and what they did generically on each play (blitz, block etc.). Although it's not entirely relevant to this question I should say that a number of NFL teams took our PP information last year and let us know it was in excess of 99 percent accurate. If we had two people doing the same game, that number rises to between 99.9 and 100 percent accuracy. Hopefully this gives you some measurable indication of the lengths we go to achieve accuracy in what we do. As far as the specific question is concerned it's about as accurate as you can be without knowing a particular assignment (more of that later) but in our opinion is at least a step change above what anyone could get by simply "tracking results". We can give you two examples:

A cornerback is beaten badly on a post route and the wide receiver drops the ball in the end zone. Tracking the result gives this as an incompletion against the cornerback (a positive) whilst we will mark this down as a significant negative.

A QB throws a perfect strike over a linebacker to hit his open slot receiver on an out. Once more, the wide receiver drops the pass and it cannons off his chest to a Safety who catches the ball even though he'd initially made a bad job of the coverage. The QB gets a INT listed against his name and the free safety gets an undeserved INT against his. How is tracking this result more accurate?

Clearly there are limitations as to what is shown on TV. The biggest issue is that of not being able to see downfield coverages on untargeted defenders and we accept this as an inherent error in what we do. That said, nobody outside the teams has access to this either so should we stop our "more accurate" analysis on the basis that it's not 100 percent perfect?

Stradley: Your website says "For every game we analyze and grade every player on every play to provide you with the most in-depth statistics you can find anywhere outside the team's film room." But then in the rules of grading under your charting discussion, it says that your charters do not guess if they do not know. And though it mentions you look at results, how can you apportion fault to a play if you don't know the play design and responsibilities? Sometimes for example, it will look like a QB is at fault when it was really the wide receivers' fault, or the defender chasing the play down wasn't the guy who missed the responsibility on the play?

Monson: As said earlier, we are not scouts. We make no attempt to document what we think should have happened on each play, how much potential we believe a player may or may not have or anything else that would take a coaches/scouts knowledge set.

We simply record what we see and the outcome of what each player attempted to do. In truth, a player may have done a good job of blocking the wrong person but as we have no knowledge of that we assume they know what they are doing and as such grade the block as a success. Clearly we accept this introduces a small error but we believe that this is still a step change in accuracy above what anyone else can achieve. When we spoke to Bengals OG Evan Mathis earlier in the season he was quick to confirm that despite this potential error margin, our grading stacks up well with the detailed feedback he was receiving internally.

At the end of the day this is the NFL and players that consistently make mental errors don't stay around very long. In the vast majority of cases the reason a play fails is because one player physically beats another and this is normally easy to see if you have the time. The reason most people don't do this is not because they can't but because they don't watch each play 5-10 times like we do and have a system for tracking it over the course of a game (never mind season). Obviously watching 60-70 hours of football a week also helps the process.

Stradley: What data do you have more confidence in, or less confidence in? I can see how you could have accurate penalty information, but some of the other stats I'm not quite sure about.

Monson: Sometimes our rankings can be misinterpreted as us saying certain players are better than others. We can only really compare people used in the same context, while leaving up to viewers to interpret the implications of our analysis.

In terms of stats, we're more confident with what you see on our site than anywhere else. Our tackle counts for example are very different to what you see recognized by the major sites, but I can say (through seeing it with my own eyes) that scorers give tackles to the wrong guys because they do things in real time and not retrospectively (it can be hard enough to do retrospectively). Countless times players who aren't even on the field are given tackles. We feel that while they may not be perfect (we do make errors and will correct them if anyone lets us know) its the best thing out there, and almost importantly all games are measured by the same standards (which any IDP player will tell you isn't the case with official scorers).

Stradley: Any question I didn't ask you that you would like to answer?

Monson: What's been the overall reaction to the site?

The general reaction has been overwhelmingly positive and interestingly (and very pleasingly) most effusive at the Professional level. We've had teams take and use our data (not just the PP information) and current players tell us that what we do is "light years ahead of most fans and media's understanding of the game". There are now far too many references of our information to mention but a quick trawl of Google will show a large percentage of online newspapers and also organizations like ESPN have used and quoted us throughout the year.

Unfortunately many of them without asking the same sensible questions that you have; I do sometimes wonder what's to suggest we're not just putting our finger in the air and saying "Hmmm Reed Doughty......feels like a +1.5".

Unsurprisingly, we've also had a small minority of general fans tell us our information is "rubbish" (and a few other things too!!), in all probability because it doesn't agree with prevailing conventional wisdom. It's extremely difficult for people who don't have the time to watch him on every play to understand Alan Faneca hasn't actually played that well for the last two years. When every commentator is mentioning how good he is every other breath, the networks are regularly only showing reruns of his positive plays and he ends up getting selected to the Pro Bowl it's a lot easier to accept he's fireproof than we have it right. If anyone has issues with an individual player's game grade we're normally very amenable to providing the audit trail of play by play analysis behind it so people can check the validity for themselves. Where people do this, those who come back to us invariably confirm our view and we've never had a single person who's taken that time and trouble criticize our methodology or overall results.

Looking at Dunta Robinson's numbers as an example.

Originally, I started discussing game charting in a Houston Chronicle blog Q&A with Bill Barnwell, managing editor of Football Outsiders, a website that puts together its own statistics. The subject came up when we were discussing cornerback Dunta Robinson's extremely poor rating by the PFF numbers. Barnwell likes the idea of what PFF does, but shares concerns that with the limitations that charters have by just televised viewing angles, plus not knowing what the player responsibilities are:

NFL teams don't chart other team's stuff like the way they're [PFF] suggesting they do, because you can't do it with anything close to reliability. Take a typical run play. Let's say the center's job is to chip the defensive tackle, get to the second level, and take out a linebacker. In reality, he doubles on the defensive tackle and it looks like he's done an effective job of sealing the tackle, but the tackle doesn't let him get off and the linebacker makes the play. How do you grade that? Do you give the center a good score? How can you know what he was supposed to be doing?

It gets even sillier when things start moving down the field -- how do you grade a route when you can't see a receiver cut? How can you grade a safety when the safety never appears on the screen? It's just absurd. If you make 100 guesses about what's happening on a play, and you get 15 of them right and 85 of them wrong, even if you mean well, you're doing the concept of charting a disservice.

Using the Robinson rating as a illustration, I figured I'd ask Sam Monson more about this subject:

Stradley: Your stats seem to suggest that Dunta Robinson is one of the worst corners in the league. But are saying is that the ordering of your numbers should not be used as player grading because it doesn't take into account things like how often he faced the #1 receiver?

Monson: The order of the players isn't necessarily a ranking system, it's simply a cumulative ordering of their various grades. If you look at Robinson's gradings most (-8 of his total -11.7 grade) of his negative comes from the 8 penalties he accumulated over the season.

Those penalties out of interest break down like this:
DPI x 4
DH x 2
Illegal Contact x 1
Facemask x 1

Just for comparison purposes Charles Woodson also had 8 penalties on the season and they look very similar:
DPI x 4
DH x 3
Facemask x 2

If you look at Robinson's coverage grade in isolation he shoots up from extremely poor to below average. Our ideal plan is for people to put their own weighting into what aspects of a position's play they think is most important. A good example at cornerback is Woodson vs Revis. Overall we have Woodson topping Revis in a close competition, but for coverage alone Revis was on another level entirely. Some people don't care much if their corner makes Deion-esque 'business decisions' if he can cover like Revis was this past season, others like their corners to be able to come up and set the tone against the run - we don't want to try and tell people which is right - it's all subjective.

We don't intend for our numbers to necessarily be a ranking system - what we present is an efficiency grading of how a player performed on the snaps he saw over the season. We then love it when smart football fans use that information together with their own knowledge to make their own conclusions.

Is it fair to grade players if you don't know what their responsibilities are?

By email, editor-in-chief of Football Outsiders, Aaron Schatz corrected a mistake I made in discussing what FOs does with their charting:

In your blog, you mentioned to someone that "FO grades on things that are possible to see on the screen--results based info." A more accurate statement would be "FO doesn't grade at all." The [Football Outsiders] game charting project doesn't grade players -- it charts events. Then we try to subjectively analyze player strengths and weaknesses, taking those stats into account the same as we do standard stats. But that analysis is clearly subjective and we don't do it with numbers -- we do it with sentences.

A good example of this is that we try to track sack reasons. We have a sack reason called "Rusher Untouched." That assigns no blame to any specific player -- because when you have a zone blitz and some guy comes in untouched, how on earth are you supposed to grade which lineman or back had blocking responsibility? You don't know the line calls! So we just say "Rusher Untouched" which means the offense as a whole sort of blew it, and the defense did something right, but it doesn't give the responsibility to any one player.

Ultimately, I don't think this is just an issue of semantics and disclaimers. I believe the Football Outsider perspective is that even with the grading methodology and normalizing that PFF does, the graders may be guessing on many things that they don't think they are guessing on but probably are. They don't want to be overly critical on the subject, but as people who have charted many games and know other people who chart games, they have a healthy skepticism about whether the data is reliable or even possible.

And from the PFF perspective, they feel that the perfect is the enemy of the good. That in focusing on a few plays you might get a few things wrong, but that if you mass large quantities of data, on balance you will get it more right than wrong. When evaluating offensive linemen for example, under their methodology if it is unclear who is to blame, the play gets a "0" for not being good or bad. But that they believe there are plenty of instances where a lineman has been overpowered by a defender straight up, and they can chart that.

One way or another, I think this is a conversation worth having. Numbers are a tool for evaluation but sometimes people make them into more than they were originally intended to be, and it is important to know their strengths, limitations and how they are put together. If you doubt me, do a brief Google search, and you can find all sorts of examples on message boards, blogs and in the media using PFF and FO data in ways beyond how they were intended.

I'd like to thank Sam, Bill and Aaron for sharing their views on this, and I thank you for reading this far. If you did, you are likely the sort of statgeek, NFL-diehard reader that will have a good comment to add to the bottom of this post. In particular, I'm interested to hear from those people who know much more about football statistics than I do, and people who have done extensive game charting.