Do Instant Draft Grades Mean Anything? Not Really

Last month you could find draft grades everywhere on the Internet (including this site). Everyone loves to read rankings, especially if they give your team an A or a B+. But do they mean anything? The old adage is it takes three or four years before you can really evaluate an NFL draft. But are the initial draft grades just for entertainment purposes or is their some value to them?

To try to find out, I looked back at the 2005 and 2006 drafts and looked at the draft grades that he handed out the day after the draft by a variety of draft experts. To get the grades from 2005 and 2006, I used the Football Outsiders' draft grade report cards posts which pulled together all of the rankings from around the web. Evaluating the drafts is made possible/easy because of Pro Football Reference's draft history. PFR is an indispensable resource for any football fan. They not only list every draft pick, but also tally how many first-team All-Pro selections, Pro Bowl appearances, seasons as a starter and games played each draftee has.

Because of that, we don't have to be subjective in slapping grades on drafts now. Instead to try to be objective, I used a simple formula that awards points for ranking best in: the number of seasons started amongst drafted players, the number of games played and career approximate value (a formula developed by Pro Football Reference), with bonuses for Pro Bowl appearances (five points per appearance) and first-team All-Pro appearances (10 points). So first, we'll look at whether instant post-draft grades mean anything, and then we'll follow up Wednesday with a much simpler way to more accurately grade drafts immediately.

For each of the rankings, being first gave the team 32 points, second gave them 31 and so on down to one point for finish last. So a team that ranked first in all three categories would have 96 points while a team that finished last in all three would have 3. The current grades are assigned in order of rank, and are weighted to equal the same average grade point average as the post-draft grades.

2005 Draft

Best Draft: The foundation of the Cowboys' moderate success in the second-half of the decade is built around this draft, which has to rank as one of the best of the decade. The Cowboys landed five significant starters (DeMarcus Ware, Marcus Spears, Chris Canty, Marion Barber and Jay Ratliff) and a sixth player (Rob Pettiti) who started for one season. Ware has been a dominating pass rusher and Ratliff has been one of the league's best nose tackles--which is quite a find for a seventh-rounder.

Worst Draft: If the Cowboys built their team around the 2005 draft, the Bills' long run of failure got a nice booster shot from this draft. Wide receiver/returner Roscoe Parrish has proven he can return punts, but he's the only long-term contributor Buffalo acquired. Center Duke Preston did get to start for most of the 2008 season, but the Bills were so unimpressed that they let him go before the 2009 season.

How The Experts Did? I'll defer to my friend, stats expert Greg Stonerock, my partner in putting together this analysis, for the following paragraph explaining how we analyzed the data.

We used a Spearman test to compare the two sets of rankings, and it looks like the draft experts did a pretty miserable job of predicting how the drafts would turn out. The Spearman test shows a correlation coefficient of .117. A coefficient of 1 (or -1) would mean post-draft grades predict NFL success perfectly; a 0 would mean the prediction is worthless. The correlation of .117 is so close to 0 that the grades cannot be considered any sort of predictor of future success. In fact, if you ranked drafts completely at random, you would get a .117 (or farther from 0) more than half the time. (52.3% of the time, to be precise.)

Two of the teams that got A's then (Dallas and Philadelphia) would still get A's now, but two of those A's are now Cs. It gets even worse as you move on. The Bengals were an A- in 2005, now you'd give them a D. The Vikings were a B+ back then, now their draft class would rank as a definite F, with Troy Williamson ranking as the best player the Vikings drafted that year.

Green Bay was rated by the consensus of draft experts as having one of the lowest grades in 2005. Now a draft that includes Nick Collins and Aaron Rodgers looks pretty good. But enough of me explaining it, here's a look at every team. The grades I doled out are based on the scoring system and are weighted to hand out the same overall GPA as the 2005 grades.

Team	AP1	PB	ST	Car AV	G	Then	Now	Best Player	Score
Dallas	4	7	19	197	467	A	A	DeMarcus Ware	166
Tennessee	1	1	19	162	566	B-	A	Michael Roos	108
San Diego	1	3	13	139	330	B	A	Shawne Merriman/Vincent Jackson	101.5
Philadelphia	0	2	18	157	451	B+	A	Trent Cole	95.5
San Francisco	0	1	19	127	490	B	A-	Frank Gore	93
New England	0	2	18	150	333	B-	A-	Logan Mankins	92.5
St. Louis	0	0	23	112	472	C	A-	Alex Barron	87
Atlanta	0	1	15	123	392	B-	A-	Roddy White	84
Seattle	1	3	13	99	259	C	A-	Lofa Tatupu	82.5
Green Bay	0	3	12	111	313	B-	A-	Nick Collins/Aaron Rodgers	81
Miami	0	1	14	98	302	B	B+	Ronnie Brown	71
Carolina	0	0	10	87	326	B-	B+	Thomas Davis	59.5
New York Jets	0	1	10	81	323	C-	B+	Kerry Rhodes	59.5
Baltimore	0	1	11	77	287	B+/A-	B	Jason Brown	54.5
Tampa	0	0	11	80	302	B	B	Barrett Rudd	52.5
Jacksonville	0	0	8	88	321	C	B	Khalif Barnes	52.5
New York Giants	1	1	8	81	237	C/C+	B	Justin Tuck/Brandon Jacobs	52
Chicago	0	0	11	87	272	B	B	Kyle Orton	52
Pittsburgh	0	0	10	86	278	B-/B	B-	Heath Miller	49
Oakland	0	0	11	64	274	B	B-	Kirk Morrison	45
Indianapolis	0	0	9	57	325	C	B-	Kelvin Hayden	44.5
New Orleans	1	2	7	64	207	C+	C+	Jamaal Brown	43.5
Cleveland	0	1	8	72	220	B/B+	C+	Braylon Edwards	35.5
Houston	1	1	6	52	229	C+	C+	C.C. Brown	35
Arizona	0	0	6	58	291	A-	C	Antrel Rolle	33
Denver	0	0	8	57	270	C-	C	Chris Myers	30.5
Kansas City	0	0	9	36	232	B-	C	Derrick Johnson	26.5
Washington	0	0	6	57	132	C	C	Jason Campbell	15
Cincinnati	0	0	5	51	208	B-/B	D	Chris Henry	14
Minnesota	0	0	3	29	219	A-	D	Troy Williamson	12
Buffalo	0	0	1	26	206	C	F	Roscoe Parrish	5.5
Detroit	0	0	2	26	150	B	F	Shaun Cody	5.5

2006 Draft

Here's how it breaks down for the 2006 draft. Chosing between the Jets and Broncos for the top draft was a very close decision.

Best Draft: The Jets picked up two-fifths of their offensive line in the first round by picking left tackle D'Brickishaw Ferguson and center Nick Mangold. Second-round pick Kellen Clemens was the team's starting quarterback for the majority of one season (10 games played and eight starts in 2007) while fourth-round pick Brad Smith has been a useful receiver. Fellow fourth-rounder Leon Washington made the Pro Bowl and was a first-team all-Pro as a kick returner as well as being a contributor in the running game. Defensive back Drew Coleman has been a consistent contributor.

The Broncos draft is arguably even better (the Jets edged out Denver for the crown because of Washington's all-pro appearance as a returner, which is admittedly a very thin difference). First-round pick Jay Cutler, fourth-round picks Brandon Marshall and Elvis Dumervill and sixth-rounder Chris Kuper have all started for three seasons and Marshall and Dumervill have turned into two of the best in the league at their positions. If we re-evaluated this a year from now the Broncos would very likely end up on top. Of course, two of those draft picks have already been dumped by Denver, which doesn't count against them in this ranking (we're looking at acquiring talent not determining where they play), but does mean the Broncos draft hasn't helped them as much as New York's picks.

Worst Draft: First-round pick Jason Allen has been a significant bust for the Dolphins (he's started only 12 games in four seasons), and he's still the best player Miami picked in 2006. He's also the only player from that Dolphins' draft who's still on the team.

How The Experts Fared: Back to Greg Stonerock:

Even worse than 2005! The Spearman test shows a correlation coefficient of .054, even closer to 0 - but neither one is any different from 0, statistically speaking. As I'll show in a follow-up post, you would do a better job of predicting who will have a good draft down the line by just counting the number of players they drafted.

Team	AP1	PB	St	Car AV	G	Then	Now	Best Player	Score
NY Jets	2	4	11	110	394	B	A+	Nick Mangold/D'Brickishaw Ferguson	120
Denver	1	4	13	149	326	B	A+	Brandon Marshall/Elvis Dumervil	116.5
Houston	0	5	16	135	307	B-	A+	Mario Williams/DeMeco Ryans	105
Green Bay	0	0	17	146	425	B	A	Greg Jennings	95
New Orleans	1	1	12	140	307	B+	A	Jahri Evans	90
Indianapolis	0	2	12	123	324	C	A-	Joseph Addai	89
Baltimore	0	1	14	93	370	B	A-	Haloti Ngata	86
Buffalo	0	0	15	106	331	C	B+	Kyle Williams	84
Tennessee	1	2	9	102	316	B-	B+	Cortland Finnegan	84
San Francisco	0	1	13	82	362	B-	B	Vernon Davis	79.5
Minnesota	0	0	12	98	320	C+	B	Chad Greenway	73
Chicago	2	2	7	89	249	C-	B	Devin Hester	71.5
San Diego	1	3	9	92	238	B-	B	Marcus McNeill	71.5
Philadelphia	0	0	11	96	307	B+	B	Broderick Binkley	65
Carolina	0	1	6	81	323	B	B	DeAngelo Williams	54
Cleveland	0	0	10	78	300	B+	B-	Kamerion Wimberly	52.5
New England	1	1	6	59	306	B	C+	Stephen Gostkowski	49.5
Cincinnati	0	0	11	70	225	C	C+	Domata Peko	45
Jacksonville	0	1	8	92	212	C+	C+	Maurice Jones-Drew	44
Arizona	0	0	10	63	246	A	C+	Deuce Lutui	41.5
Oakland	0	0	9	59	258	B-	C	Thomas Howard	39.5
Kansas City	0	0	9	66	227	C+	C	Tamba Hali	38
Tampa Bay	0	1	9	54	216	C	C	Jeremy Trueblood	35
NY Giants	0	0	6	58	273	C+	C	Barry Cofield	31
Seattle	0	0	8	48	249	B+	C	Darryl Tapp	30.5
Dallas	0	0	3	44	315	C+	C-	Anthony Fasano	29
Pittsburgh	0	0	7	68	193	B	C-	Santonio Holmes	28
Washington	0	0	7	64	209	D+	C-	Rocky McIntosh	27
St. Louis	0	0	5	48	266	C+	C-	Victor Adeyanju	25.5
Detroit	0	0	4	37	163	C+	D	Ernie Sims	8
Atlanta	0	0	1	32	166	C	F	Jerious Norwood	5.5
Miami	0	0	1	25	185	C	F	Jason Allen	5.5