Speculations on play-calling on first, second, and third and ten

From Advanced NFL Stats:

All the numbers that follow are from all 10-yards-to-go scrimmage plays in the first 3 quarters of regular season games from 2000-2007. The only other limitation was that the game score was within 10 points. I wanted to exclude situations when teams exercised an abundance of either risk or caution.

Note the percentage of play types called on 1st, 2nd, and 3rd downs (with 10 yards to go). There is a fairly even split between run and pass calls on 1st and 2nd downs. On 3rd and 10, the a pass is far more expected.

% of Play Types by Down, 10 Yds To Go

Type - 1st - 2nd - 3rd - Total
Pass - 47.2 - 52.7 - 91.1 - 49.6
Run - 52.8 - 47.3 - 8.9 - 50.4

Although 91.1% isn't 100%, it's close to where the anchor point on the lower right side of the game theory graph--almost the pure pass vs. pass defense strategy combination. Now let's look at the average outcomes for these situations.

Yds Per Attempt by Down, 10 Yds To Go

Type - 1st - 2nd - 3rd - Total
Pass - 7.0 - 6.3 - 6.5 - 6.9
Run - 4.2 - 4.4 - 6.9 - 4.3
Total - 5.5 - 5.4 - 6.5 - 5.6


When passing is most predictable, it yields half a yard less than on first down, when it is less expected. Conversely, running is most successful when it is least expected.

At this point, I should point out that passing on 3rd and 10 yields slightly more yards than on 2nd and 10, which isn't completely what we'd expect. This is almost certainly because defenses will allow short complete passes on 3rd down in exchange for being relatively assured to be able to stop the gain short of 10 yards. This is part of the problem posed by the fact that yards does not equal utility. We'll have to dig a little deeper. The next table lists interception rate by down.

Interception % by Down, 10 Yds To Go

Int Rate
1st - 2nd - 3rd - Total
2.6 - 2.9 - 3.5 - 2.7

Now we see more what we'd expect--a slight increase from 1st to 2nd down, then a large jump on 3rd down, in accordance with the associated increases in passing predictability. The next table lists adjusted yards per attempt, which is YPA with a -45 yd adjustment for every interception thrown. Adj YPA, however, still exhibits the same problem as plain YPA. It underestimates the drop off from 1st to 3rd down in passing effectiveness because defenses will allow gains, as long as they're not more than 9 yards.

Adj Yds Per Attempt by Down, 10 Yds To Go

Type - 1st - 2nd - 3rd - Total
Pass - 5.9 - 5.0 - 4.9 - 5.6
Run - 4.2 - 4.4 - 6.9 - 4.3


So what we can say is, the reduction in passing effectiveness due to predictability is likely at least 1 full adjusted yard per attempt. The drop from 1st down to 2nd was 0.9 yards, so the true reduction in effectiveness from 1st to 3rd down may be far larger.

Except that there's a problem with this analysis. There's a bias in the data. Which teams are more likely to face a lot of 2nd and 10s and 3rd and 10s? The ones that stink at passing. So the 2nd and 3rd down numbers are lower than would be representative of the league as a whole. In other words, poor passing teams 'get more votes' in the analysis.

All this is intended to tee up a game-theory analysis for finding some kind of ballpark run/pass equilibrium. Do read the whole thing.

But a few brief thoughts:

• The adjusted final numbers intrigue me, particularly second down as compared to first. (As Brian notes, third down is tougher to break down since it's really a binary question of conversion versus failure.) But I'm struck that on second down the yards per pass attempt drops by nearly a full yard while the yards per run goes up only .2: why does the defense get so much better on second down? Is the data skewed to losers? Is play-calling worse on second down?
  
  
• In that vein, I wonder if the old conventional wisdom about "getting back half on second and ten" works against the offense. On first down the passing plays are likely to involve play-action as well as quick or intermediate passes -- coaches can use their full asrsenal; maybe on second coaches are too concerned with screens and quicks -- trying to just get half -- that they give up too much in the way of expected points?
  
  
• But on the other hand, what if they get this 5.0 yards per pass attempt on second and ten with more certainty and less variance than the 5.9 on first down. If so, then possibly the offense is in better position to convert third down than they would be even with a greater expected play value that carried more variance. Could cut either way; football is complicated.

Hopefully Brian can shed some light as his series develops. I look forward to it.

Football, Luck, and Noise

I received a surprising amount of pushback via email regarding my last post about Texas Tech and the Hot Hand Theory. At first I was confused, but then I realized that many readers do not share a rather fundamental assumption I hold about football: an incredible amount of the game is determined by "luck." Now, when I say luck, I do not mean fluke events, or the ol' bounce a da ball, or things like that. What I mean is that almost any and every outcome in football is not set in stone, but rather, there is some probability that the outcome will be X, another probability that the outcome will be Y, and maybe even a chance that it will be Z.


Theological questions aside, I really think this is a rule of life and not just football. But the point is that at no point in a football game, be it success of a play or even a determination of what the other side is actually doing, do you have fixed answers. Instead, you have probabilities, and even then your probabilities are merely estimates of the actual probabilities. So when I talk about "coolly flipping coins," I mean that everything is probabalistic. Just like when Michael Jordan went to the free-throw line, no matter what any sports writer tells you, he is never destined to make the shot, or destined to make the game-winner. Tiger Woods is never destined to hit the putt, and Tom Brady or Peyton Manning were neither destined to win the Super Bowl or hit any particular pass.

Instead, it was merely "highly likely" that each was going to do those things, because each is very good at what they do. But at no point is anything determinate.

Indeed, one of the criticisms of my post was that the probabilities dramatically increase regarding offensive success because you gain more information as time goes on. But that argument doesn't hold water. If Michael Jordan can only max-out his free-throw percentage to a point, then there is no way to max out offensive production in football when at all turns you have a human (or group of them) making choices on the other side in ways that shift your probabilities. That is a far too nebulous cloud to assume certitude.

And any playcaller will tell you the same thing. As Norm Chow says, you are never quite sure what coverage they are in, but instead you take pieces of the field or pieces of the defensive front and attack those, and therein lies success. Mike Leach does not even require his guys to memorize coverages in the sense of "Hey they are in Cover 4!" Instead, they group them into things they can recognize and they probe areas. But at every stage, things are probabalistic. I've even discussed the notion that a purely random approach to offensive and defensive calls might even be optimal.

When I made the point about the hot hand theory, part of it was about how you cannot always extrapolate how good an offense is versus a defense just because they scored on a drive, or even if they scored a lot in a half or game, because the standard deviation is too high. Some people argued that things would even out over the course of a game; I think that is sort-of true, but I still think the variance is higher than they account for. But that's an empirical question we can solve later.

But another (amazing) site, Advanced NFL Stats, made the point about the difficulty of extrapolating skill levels from even successful outcomes:

Consider a very simple example game. Assume both [Pittsburgh] and [Cleveland] each get 12 1st downs in a game against each other. PIT's 1st downs come as 6 separate bunches of 2 consecutive 1st downs followed by a punt. CLE's 1st downs come as 2 bunches of 6 consecutive 1st downs resulting in 2 TDs. CLE's remaining drives are all 3-and-outs followed by a solid punt. Each team performed equally well, but the random "bunching" of successful events gave CLE a 14-0 shutout.

The bunching effect doesn't have to be that extreme to make the difference in a game, but it illustrates my point. Natural and normal phenomena can conspire to overcome the difference between skill, talent, ability, strategy, and everything else that makes one team "better" than another.

And adding support for my argument about the high degree of variance, Advanced NFL Stats went on to try to nail down exactly how much in the way of outcomes can be attributed to skill versus luck in the NFL. You can read the details of the explanation there, and NFL teams obviously are closer in relative skill levels than most college teams, but the results are nevertheless striking:

...By comparing the two distributions, we can calculate that of the 160 season outcomes, only 78 of them differ from what we'd expect from a pure luck distribution. That's only 48%, which would suggest that in 52% of NFL games, luck is the deciding factor!

There might yet be more to it than these calculations, but the point is that variance is high in outcomes in football games. This is not to say that skill is unimportant, but the lesson is instead that you cannot merely look to actual statistics and actual outcomes to determine who is the best. Football games are tests of ranges of probabilities put up against one another:

Will all eleven players execute their assignments; will the quarterback make the right reads; will the coaches accurately assess the opponent's schemes; will the sun shine in the receiver's eye; will the ball become sweaty where the ballcarrier holds it; will there be an injury on the play; and if these factors randomly cut 50/50, will they work in our favor enough times in a row to get us in field goal or touchdown range.

In other words, lots of football fans, players, and even coaches suffer from a Fooled by Randomness problem when they analyze the game. Football is more quantum mechanics than it is Newtonian physics (though with a splash of game theory). Yet the belief in absolute determinism is natural: we intuitively want results to be indicative of objective truths, and it is much less complex to analyze easy to observe statistics and outcomes than it is to try to estimate the underlying probabilities. But football doesn't always give us large enough sample sizes to believe that results are as instructive as we'd like. So, if we want real answers, we have to admit that there's lots of luck around.

(And if you're a fan of the Michigan Wolverines, this gives you an (incredibly weak) excuse: "It's all the result of bad luck!")

Run/Pass Balance, Game Theory, and the Passing Premium Revisited

A few years ago I wrote some of my most popular posts, entitled the "Run/Pass and a Little Game Theory," with follow-up responses to comments One, and Two. (Some of my best ideas are in the responses.)

Although I can't say exactly that they were inspired by my work - though these posts were released in mid-2006 - there has been much serious work done in the areas since.

First, one of the phenomena I identified was that in a world without risk, coaches ought to be neutral between running and passing, and so should call them in whatever mixture produces the most yards per play. But I identified a "passing premium," which refers to the fact that in the real world, passes tend to average more yards per play than runs, and coaches generally do not throw the ball more to correct for this perceived imbalance. I argued that this made sense because passes are generally riskier than runs, but teams might still be out of wack one way or another. The other side of the coin was that supposedly imbalanced teams, be they run first or pass first teams, were often very much balanced in their play selection than people give them credit for.

Well, this idea has engendered much recent scholarship and commentary, including two articles in the Journal of Quantative Analysis in Sports:

- Benjamin Alamar, The Passing Premium Puzzle.

- Duane Rockerbie, The Passing Premium Puzzle Revisited.

- Sabermetric Research, NFL Passing Premium Puzzle Revisited.

The other phenomena I touched on in those articles was that the choice between running and passing, in terms of maximizing your expected gain, was not absolute. Instead it was a game theory problem. In other words, just because passes tends to average more than rushes does not mean that you should call 100% passes. If you pass more and more, the other team will shift their strategy. But even more importantly, the other team will adjust its strategy based on your intrinsic payout structure: If you are an excellent passing team, they will obviously call more pass defenses (or add an extra defensive back, etc) but if your payout structure changes - i.e. an excellent running back returns from injury - that does not mean that you should automatically run more and more, because the other team, knowing that he is back, will be forced to counter your running back. The exact change in optimal run/pass balance can change, but the upshot is that your passing game will often become more attractive when a great running back comes back, because the defense must shift its strategy. So when TV announcers say that just because you have a new or better running back, then that team must automatically run the ball more, do not believe it.

This idea too has engendered commentary (again, not necessarily as a direct result of my posts, but I wrote about it some years ago and I'm happy to see the ideas proliferate):

- Advanced NFL Stats, Game Theory and Run/Pass Balance.

- Advanced NFL Stats, Passing Paradox. (I mention this article too because it cites some of my older work, which attempted to account for the riskiness of runs and passes and find their optimal mix by using the Sharpe ratio from financial economics. See also here, and here.)

I look forward to more and more analysis on these topics. Football is by far the most interesting sport to analyze, but also the most difficult. So, so many variables. Each play is discrete, but interconnected to the others in intricate ways (hence, game theory). Further, to analyze it requires a certain basic competency in football, statistics, probability, and economics (along with time) that few possess. As great as Football Outsiders is, they have produced very little that is useful to the typical coach. But the ideas here transcend the fan and the practitioner. I am sure there is more to come.

Links to my past articles:

Run/Pass Balance and a Little Game Theory
Run/Pass Balance - Response to Comments I
Run/Pass Balance - Response to Comments II
The Sharpe Ratio for Football - I
The Sharpe Ratio for Football - II
The Sharpe Ratio for Football - III

Constraint Theory of Offense

A common question is: what kind of offense do you run? Often someone – both fans and coaches – respond and say: I run a system with bubble screens, play action passes, screens, and draws. This person – coach or not – would be completely wrong. These kinds of plays are not your offense; your offense consists of the zone-read, the dropback passes, or your base runs and passes. Those other plays are sort-of conditions precedent to your offense; they work as constraints on the defense. What do I mean by this?

At least in the most abstract sense, your “offense” is that bread and butter stuff you can draw on the whiteboard that should always work in a perfect world. It is the pass play that always works against Cover 3, it is the run that will always burst free against a “Bear” front. Yes, it is what works on paper. But we don’t live in a perfect world, right? Well the “constraint” plays are designed to make sure you live in one that is as close as possible.

For example, the safety might get tired of watching you break big gains up the middle, so he begins to cheat up. Now you go play action and make him pay for his impatience. The outside linebackers may cheat in for the same reason. You throw the bubble screen and the bootlegs to make them pay for their impatience. The defensive ends begin rushing hard upfield; you trap, draw, and screen them to make them pay for getting out of position. If that defensive end played honest your tackle could block him; if he flies upfield he cannot. So you have to do these “constraint plays” to keep them in check. Once they get back to playing honest football, you, in essence, go back to the whiteboard and beat them with your bread and butter.

Now, in a given game your offense might look like it is all “constraint” plays: all gimmicks, screens, traps, draws, fakes and the like. Maybe so. If the defense plays too aggressively, so what. But a coach must not lose sight of how his offense is truly structured. A great offense is structured around a core idea or a few core ideas that puts the players in position to succeed every time. The triple option can be this for some teams, a well designed dropback pass game for another. The constraints are alternatively given too much and not enough weight. But they nevertheless are what make an offense go.

So the better you are at dropback passing, the more you need these constraint plays because teams will go out of their way to prevent you from chucking it all over them. Similarly if you’re a great run team. Safeties and linebackers will all cheat by formation and post-snap effort to stop your run game. You must have the counters, the screens, the bootlegs, and the quick passes (because quick 3-step passes, at core, are most effective when used to simply take advantage of a loose defensive structure). All this comports well with a game theory approach to football. Similarly, these constraint plays will be even more important against the best teams because they will put the biggest premium on stopping your primary threat.

The upshot of all this is that when you are designing an offense you must (a) find those one or two things which you can hang your hat on and beat just about anything doing when the defense is playing honest, and (b) get good at all those little “constraint” plays which keep the defense playing honest. You won’t win championships simply throwing the bubble screen, but the bubble will help keep you from losing games when the defense wants to crash your run game. Same with draws and screens if you’re a passing team. You find ways to do what you want and put your players in position to win and score.

ADDENDUM: Fair question from the comments: Does the theory work in the other direction? What if your offense is based only on bubble screens and then you just run the ball or throw the ball as a counter to your bubble screen offense?

Response: The difference is that the bubble screen is a play that really only works when the defense has made a structural choice or is out of position. Most commonly, you'll run when the bubble only when the defense has but two defenders to cover three receivers. You thus block the two defenders and the receiver has free yards. If the defense puts a third defender there they can take the play away, intercept it, or make the tackle.

Conversely, a well designed dropback pass play, a triple option play, or certain base runs will work every time you face a normal defense. The only time the play stops working is when certain defenders cheat on their assignments, either by alignment or aggressiveness.

Here's how they fit together: You're an option team. You come out running the option, you read the defensive end and the linebacker, and you tear them up. Now the safety or outside linebacker cheats in. He blows up your play. But, voila, now they are not covering your outside receivers, so you bubble screen them.

Similarly with a play action pass. You send a receiver deep down the middle or the seam. If the safety plays honest he should drop back and take it away. But if he comes up for your run play you use his aggressiveness against him.

The distinction is subtle, but important. It relates to the idea of base plays and counter plays. The bubble is simply not a base play. It will not work against a simple and sound defense, but works great against defenses that aren't structurally sound or balanced. On the other hand, "base plays" defeat balanced "whiteboard" like defenses, but can get blown up by defenses that cheat or play games. Thus the relationship between "base plays" and "constraint" plays (or "keep-em-honest plays). The bubble, while limited in use, will have a profound influence when the defense gets out of position.