Back in September, I wrote a story about the NFL's most significant offseason rule change. While most of the football-watching world was worrying about replacement refs, I felt the league's decision to review all turnovers automatically had slipped undeservingly under the radar.
I argued at the time that with all scoring plays and turnovers handled in this manner, the NFL had reached a point where it ought to do away with the coaches' challenges altogether and put them all in the booth.
With little else to do during the two week stretch between meaningful football games, I thought it would be a good exercise to revisit that story. Let's see if the statistical data and anecdotal evidence from this season backs up my points.
While I'd encourage you to go back and read the original story, I'll summarize some of the main points here:
• Automatically reviewing all scoring plays and turnovers significantly decreases the amount of plays coaches can challenge, and the importance of those plays anyway.
• With more cameras, better technology and more money at stake, it seems outdated to penalize coaches for wanting to make sure all calls are correct.
• Coaches' challenges are usually either blatant guesswork or initiated after somebody on the staff has already seen the replay.
• Coaches won't be hurt by giving up their challenge powers because refs have historically reviewed more plays when given the option.
So what happened in 2012?
The number of coaches' challenges did decrease significantly during the 2012 regular season, to 157.* The NFL saw 209 challenges in the 2011 season, dropping from 249 in 2010 (when scoring plays and turnovers were both still under the coaches' purview for the majority of the game).
While I did predict this decrease, it wasn't 100 percent inevitable. Coaches could have reacted to the rule change by lowering their standards for what type of play warrants challenging, especially if they felt it was less likely they would run out of challenges at a critical juncture of the game. Instead, with the average team attempting fewer than five challenges all season, the data signifies that the coaches' challenge wasn't growing less selective, just more obsolete.
The effect on the percentage of challenges overturned is a bit murkier. Over a three-year period from 2008-2010, 40.3 percent of calls on coaches' challenges were overturned, with the average increasing slightly each season. With the new rules implemented last year, that number ballooned to 52.6 percent. This past season, the number went back down to 47.1 percent -- very much in line with the increases from 2008-2010, but a step back from last year's mark.
When the percentage shot up last year, one conclusion could have been that shifting the burden of challenges from coaches to the booth increases the likelihood of overturning a call. If that was the case, however, adding even more automatic booth reviews should have accelerated that trend, rather than reverse it. Also, the rate at which challenges initiated by the booth were overturned stayed nearly identical: 96 out of 278 in 2012 (34.5 percent) compared to 62 out of 181 (34.3 percent) in 2011.
It might be hard to differentiate the concepts, but the percentage of calls being overturned isn't necessarily indicative of the quality of the challenges. We must separate the process (deciding the proper time to challenge a call) from the outcome (the referee's decision).
In my mind, a "good challenge" is one that's used on a close call on a play that could dramatically impact a team's win expectancy. If the calls on the field are upheld more often in a given year, it might just mean that referees did a better job of getting the close calls correct. That doesn't necessarily mean coaches were worse at challenging.
Here's the important part: If coaches were worse at overturning calls simply because the easier calls had been moved to booth reviews, the booth reviews should have seen a corresponding shift in reversal rate. If coaches were worse at getting calls overturned because they were lowering their burden of proof necessary to challenge, that should have been accompanied by the increase in challenges that I already showed to be nonexistent.
With those last two facts in mind, it's impossible to draw any conclusions from this year's percentages. If recent rule changes have an effect on the percentage of calls overturned, the last two years haven't provided a large enough sample size to detect it. And we definitely didn't see anything overwhelming enough to change my opinion about coaches' challenges.
Finally, one of my reasons for proposing change in September was "a continuing failure to understand the rules in the first place." Nothing underscored that point more than Lions head coach Jim Schwartz's Thanksgiving Day Adventure. Schwartz notoriously challenged that Justin Forsett was down by contact en route to his 81-yard touchdown run. Since the play was meant to trigger an automatic review, Schwartz's challenge was not only impermissible, but meant that the replay booth was no longer allowed to review the play. The touchdown stood and the Lions lost in overtime.
Many were outraged by the whole episode but I mostly felt a perverse sense of amusement, hopeful that an embarrassing and unpopular catastrophe might actually spur change. While most people called to change this particular rule, I think that might be treating a symptom rather than a cause. Schwartz's blunder was merely one high-profile indicator that the entire institution has outgrown its usefulness.
The challenges have decreased in frequency, with no discernible difference in accuracy, with the rules only growing more complicated for referees and coaches alike. I still firmly believe that the NFL would be best-served to remove the coaches' challenges altogether and put them all in the hands of the replay booth, for the many reasons outlined in my first story.
While I've definitely written stories that I wish I could take back, I'm happy to say this isn't one of them. In September I claimed I was challenging the coaches' challenge system. Five months later, I'm confident my challenge has held up under review.
*Note: I knew some skeptics would say that this year's data is skewed because replacement refs officiated 48 out of 256 games, so I compared the data from those 48 games to the other 208. Coaches challenged 110 calls from weeks 4-17, which extrapolates to 135 over a full season. The regular referees overturned 49 (44.5 percent) of those calls. Since both of those numbers drive down the season totals, they actually strengthen my arguments.