Analysis
Completion percentage over expected, and the model beneath it
CPOE explained from the arithmetic up: how a completion probability is built, why a better model makes a worse quarterback metric, and its blind spots.
By CricketTaken EditorialPublished Analysis19 min read
Two quarterbacks finish a season having completed the same share of their passes. One of them threw almost nothing beyond the sticks: screens, checkdowns, a running back in the flat on third and seven. The other threw down the field into two-high coverage, on a team that could not run the ball, behind a line that could not hold. The statistic that describes them says they are identical.
Completion percentage over expected exists to fix that, and it fixes it more strangely than most people realise. CPOE is not a measure of accuracy. It is the error term of a machine learning model, and everything interesting about it, including its two serious flaws, follows from that one fact.
The metric is only as good as the expectation it is measured against. Change the model and you change every number it produces. That is the sentence to hold on to.
What completion percentage over expected actually is
Start with a single throw, because the season figure is nothing but an average of these.
Before the outcome is known, a model assigns the pass a completion probability. Call it 0.62. The quarterback throws it. If the pass is completed he banks one minus the probability, so 1 − 0.62 = +0.38. If it falls incomplete he banks zero minus the probability, so 0 − 0.62 = −0.62.
That is the whole definition, and it is worth reading twice because it explains the metric's entire character. The open-source play-by-play dictionary states it in exactly those terms: for a single pass play, CPOE is one minus the completion probability when the pass was completed, and zero minus it when it was not.
Average those per-play values across every attempt, multiply by a hundred, and you have a figure in percentage points. Positive means the quarterback completed more passes than the model expected of the attempts he actually made. Zero means average. The model is fitted to the population it describes, so average is average by construction, not by accident.
Three consequences fall out of the arithmetic immediately.
A completion of an unlikely pass is worth far more than a completion of a likely one. A ninety per cent throw completed is worth +0.10. A thirty per cent throw completed is worth +0.70, seven times as much.
An incompletion on an easy throw is punished savagely. Miss a ninety per cent pass and the quarterback is charged −0.90, which takes nine completed ninety per cent throws to recover.
And the metric is entirely symmetric, in a way that matters later. Every unit of credit a quarterback gains is a unit the model failed to predict. CPOE does not measure how well he threw. It measures how wrong the model was, and then attributes all of that wrongness to him.
Why raw completion percentage is nearly useless on its own
Completion percentage is largely a measure of throw distance. That is not a criticism of the quarterbacks; it is arithmetic. A pass travelling three yards past the line of scrimmage to a back with a blocker in front of him is a fundamentally easier proposition than a pass travelling twenty-two yards to a receiver running a comeback with a corner on his hip.
So a quarterback can raise his completion percentage without improving at all, by throwing shorter. Offensive systems can raise it for him. A scheme built on quick game and screens will produce a high completion percentage from a mediocre passer, and the same passer in a vertical system will look considerably worse while throwing the ball exactly as well as he did before.
There is a second problem, quieter and rarely mentioned. The denominator is not what people assume it is. A pass attempt is a recorded event with rules attached to it, and several things that look like failed passes are not attempts at all. A play wiped out by a penalty produces no statistics, so a throw that would have been an ugly incompletion vanishes if a defender was holding somewhere else on the field. Conversely, a ball thrown deliberately out of bounds to avoid a sack is an attempt and an incompletion, and a ball spiked to stop the clock is an attempt and an incompletion, and neither had any chance of being caught by anybody.
Those events are noise in the denominator, and they are not evenly distributed. A quarterback who throws the ball away rather than taking sacks will carry a worse completion percentage than one who takes the sack, and taking the sack is usually the worse football decision. That is the same structural blind spot that leaves the sack out of passer rating entirely, and it survives into CPOE unless somebody filters the attempts deliberately.
The honest summary is that raw completion percentage answers a question nobody asked: what share of the passes a quarterback chose to throw were caught, given the difficulty he chose. CPOE removes the second clause. That is genuinely valuable and it is also the source of the trouble.
Where the over-expected idea came from, and what football borrowed
CPOE belongs to a family, and recognising the family explains both its strengths and the arguments about it.
The construction is always the same. Measure what happened. Build a model that says what should have happened given the circumstances. Keep the difference and call it the player's contribution. Golf does it with shot outcomes against a baseline by lie and distance. Association football does it with shooting chances. Baseball does it with batted-ball characteristics. Every one of them is a subtraction, and in every one of them the entire argument is about the second term rather than the first.
American football arrived late to this, and the reason is data. Baseball is a sequence of discrete events with a small number of participants and a century of complete records. Football is twenty-two people moving simultaneously, and until the ball's flight and the players' positions could be measured directly, there was very little to build an expectation from. The play-by-play record gave analysts one usable variable, the distance the ball travelled past the line of scrimmage, and the first completion probability models were essentially that variable and a curve.
Tracking changed the ceiling rather than the idea. It also spawned the siblings. Expected rushing yards models what a carry should have gained given where every blocker and defender stood at the handoff, and rushing yards over expected is the residual. Expected yards after catch does the same for the receiver from the moment the ball arrives. All of them inherit CPOE's central difficulty in an even sharper form: a rushing model that controls for the blocking has removed the offensive line, which is frequently the thing you wanted to know about, and a model that does not control for it is measuring the line and calling it the back.
The useful habit, applied to any metric with "over expected" in its name, is to ask what was subtracted. That question is more informative than the number, and it is almost never answered on the graphic.
What actually goes into a completion probability
There are two families of completion probability model, and confusing them produces most of the bad CPOE analysis in circulation.
The first family uses only what appears in the official play-by-play record. The main input is air yards, defined precisely as the distance in yards perpendicular to the line of scrimmage at the point where the targeted receiver caught or failed to catch the ball. Add the horizontal direction of the throw, the down and distance, the field position, and a model can already do a respectable job, because throw distance dominates completion probability so heavily that a model knowing nothing else still explains a lot.
The second family uses tracking data, and the difference in what it can see is enormous. The league's own system puts radio-frequency tags in players' shoulder pads and in the ball, reads them with receivers installed around the stadium, and records position ten times a second to within inches. The published specification describes roughly 250 tagged devices in a venue and more than two hundred new data points generated on every play. Completion probability is one of the advanced statistics the league lists as derived from that feed.
- 10Positional readings per second
- 250Approximate tracked devices per venue
- 200New data points generated per play
Published specification of the league's tracking system. These are equipment and sampling figures, not performance figures.
With that feed available, a model can be given things the play-by-play record cannot supply: how far the nearest defender was from the receiver at the moment of release and again at the moment of arrival, how long the quarterback held the ball, how many defenders rushed, whether a rusher had closed to within a defined distance, how far the receiver was from the sideline, how far the ball actually travelled through the air rather than perpendicular to the line, and the geometry of the throw relative to the quarterback's own position and momentum.
Each of those inputs makes the model more accurate. Each of them also, and this is the part that gets skipped, takes something away from the quarterback.
A better model makes a worse quarterback metric
Here is the problem at the centre of CPOE, and it is not a bug that a future version will fix. It is a structural conflict between two goals that pull in opposite directions.
The goal of a completion probability model is to predict completions accurately. The goal of CPOE is to isolate the quarterback's contribution. Every variable added to the first goal serves it and damages the second, whenever that variable is something the quarterback influences.
Take receiver separation. A model that knows the receiver was wide open will assign a high completion probability, and the quarterback gets little credit for hitting him. That seems right. But separation at the moment the ball arrives is partly created by the quarterback: by throwing on time so the receiver catches the ball at the break rather than after it, by holding a safety with his eyes, by choosing the receiver who was about to be open rather than the one who was open. A model that controls for separation has quietly removed all of that from the quarterback's account and given it to the receiver.
Take time to throw. A quick release raises completion probability, so a model that includes it credits the quarterback less for the completions that followed. But getting the ball out quickly is a skill, and one of the more valuable ones.
Take pressure. Controlling for whether a quarterback was under pressure is defensible, since pressure is largely inflicted by the offensive line. It is also partly self-inflicted, by holding the ball, by drifting in the pocket, by working through progressions slowly. Control for it completely and you have credited the quarterback for throwing well under a pressure he generated himself. The interaction between the two is genuinely hard, which is why pressure rate as a standalone measure is itself contested.
The general form of the problem is that CPOE controls for variables the quarterback chose. In most statistical contexts you control for things the subject did not choose: the weather, the stadium, the strength of the opposition. Controlling for a choice removes the consequences of the choice, and if choices are what you are trying to evaluate, you have subtracted the thing you were measuring.
So there is no single correct completion probability model, and the disagreements between analysts about CPOE are usually disagreements about this rather than about arithmetic. A minimal model, one that knows only how far the ball went, produces a CPOE that mixes accuracy with decision-making, receiver quality and scheme. A maximal model, one that knows everything the tracking feed can supply, produces a CPOE that is closer to pure ball placement and much less useful for answering the question people actually ask, which is whether a quarterback is any good.
- The ball is snapped and the tracking feed runsPosition is recorded ten times a second for every tagged player and for the ball itself. Nothing has been judged yet; this is measurement.
- The features are extracted at the moment of releaseThrow distance, direction, time held, rushers, receiver position, defender distance. Which of these the model is allowed to see is the single most consequential decision in the whole process.
- The model returns a completion probabilityA number between zero and one for this specific throw, fitted on a population of comparable throws. The quarterback's identity is deliberately not an input, or the model would predict him rather than the throw.
- The play resolvesCaught, dropped, broken up, intercepted, thrown away. The model does not care which of those it was; only completed or not completed.
- The residual is bankedOne minus the probability if it was caught, zero minus the probability if it was not. This single number is the quarterback's entire credit for the play.
- Residuals are averaged across attemptsMultiply by a hundred and the result is a figure in percentage points. Zero is the model's average, which is the league's average, which is why the metric has no natural units of football in it.
The same sequence runs on every attempt. Steps two and three are where the analytical choices are made, and where two providers' numbers diverge.
Two constructed quarterbacks, and the same completion percentage
Numbers make this concrete, so here are two invented quarterbacks with invented model probabilities. Nothing below describes any real player or season; the figures are chosen so the arithmetic is visible.
Quarterback C throws ten passes. Every one is a screen or a checkdown, and the model assigns each a completion probability of 0.90. He completes eight. His expected completions were 9.0, his actual were 8, so his CPOE is (8 − 9) ÷ 10 = −0.10, or −10 percentage points. His raw completion percentage is 80.
Quarterback D also throws ten passes. Four are quick throws modelled at 0.90, three are intermediate throws modelled at 0.60, three are downfield throws modelled at 0.30. His expected completions are (4 × 0.9) + (3 × 0.6) + (3 × 0.3) = 3.6 + 1.8 + 0.9 = 6.3. He completes eight. His CPOE is (8 − 6.3) ÷ 10 = +0.17, or +17 percentage points. His raw completion percentage is also 80.
Twenty-seven percentage points separate two passers whose conventional statistics are identical. That is the case for the metric, made as cleanly as it can be made.
- Modelled at 90 per cent4
- Modelled at 60 per cent3
- Modelled at 30 per cent3
Invented attempt counts. The labels give the completion probability the model assigned to each group of throws.
Show the numbers
| Item | Value |
|---|---|
| Modelled at 90 per cent | 4 |
| Modelled at 60 per cent | 3 |
| Modelled at 30 per cent | 3 |
Now push the same example one step further, because the second step is where CPOE starts being interesting rather than merely correct.
Suppose quarterback D's three downfield throws were modelled at 0.30 because his receivers were covered. If the reason they were covered is that his offence has no separation, then a positive CPOE is telling you he overcame a bad situation. If the reason they were covered is that he stared them down and let the coverage close, then the same positive CPOE is rewarding him for creating the difficulty he then survived. The number is identical in both cases. The model cannot distinguish them, and neither can anybody reading the season total.
What CPOE genuinely does better than anything else
Three things, and they are worth stating plainly because the criticism above is not an argument for ignoring it.
It separates accuracy from depth. This is the one it was built for and it does it well. Any comparison of quarterbacks that does not adjust for throw distance is comparing two different jobs.
It accumulates sample quickly. CPOE is a per-attempt residual on an event that happens tens of times a game. Metrics driven by rare outcomes, interceptions above all, need enormous samples before the signal exceeds the noise, and by then the player has changed. Because every attempt contributes, a CPOE figure settles down faster than most quarterback measures. It is still a rate with variance around it, and a small sample is still a small sample, but the structure is friendlier than most.
It is honest about what it does not know. A completion probability model does not take the quarterback's identity as an input. If it did, it would predict the quarterback rather than the throw, and the residual would collapse towards zero for everybody. Keeping identity out is what makes the residual meaningful, and it is a discipline that several other sports metrics have failed to maintain.
There is a fourth, quieter virtue. CPOE degrades gracefully. A crude model built from the public play-by-play still produces a number that is more informative than raw completion percentage, so the metric is usable by anyone with the free data rather than only by people with access to the tracking feed. Very few modern football metrics have that property.
What CPOE cannot see, stated flatly
The list is longer than the metric's popularity suggests.
Value. A completion is a completion. A four-yard catch on third and eleven counts exactly as much as a forty-yard touchdown, and both count more than an incompletion on first and ten that cost almost nothing. CPOE has no concept of leverage, situation or gain. This is the single largest gap, and it is why the metric is almost always shown alongside expected points added, which measures precisely the thing CPOE ignores.
Interceptions. In the standard formulation an interception is simply an incompletion. The quarterback is charged the same as if the ball had fallen harmlessly to the turf, despite having handed the opposition possession. A metric that treats those two outcomes identically is not measuring quarterback play in any complete sense.
Sacks and scrambles. Neither is a pass attempt, so neither appears. A quarterback who takes six sacks and one who throws six balls away have very different games and can have identical CPOE.
Drops. A dropped pass is recorded as an incompletion, and the quarterback is charged the full negative value of a throw the model expected to be caught. Drop-adjusted versions exist. They depend on somebody deciding, by eye, what constitutes a drop, which is a subjective judgement that differs between charting services and is not reproducible.
Everyone else. The receiver who wins at the catch point, the line that gave the quarterback a clean pocket, the coordinator who schemed a defender out of the picture. None of them appear anywhere in the calculation, and all of them moved the number. This is the same attribution problem that DVOA handles by not pretending to solve it, and it has no clean answer in any sport where eleven people cooperate on one event.
The pass not thrown. A quarterback who declines a difficult throw and takes the checkdown has protected his CPOE. A quarterback who takes the difficult throw and completes six in ten has been rewarded, but one who completes four in ten has been punished for making the correct football decision. The metric has a mild but real bias towards caution, which it shares with almost every completion-based statistic ever devised.
Why it is nearly always shown next to EPA
Pair CPOE with expected points added per play and the two cover most of each other's blind spots. CPOE measures how well the ball was delivered relative to the difficulty of the delivery. EPA measures how much the play was worth. A quarterback high in both is accurate and productive. High CPOE with low EPA describes a passer completing a lot of throws that do not matter, which is the statistical signature of a checkdown offence. Low CPOE with high EPA describes volatility: big plays and misses, with the misses invisible in the points column because incompletions are cheap.
The pairing is not a formal metric and the two are not on the same scale, which is why they are usually plotted against each other rather than combined into one figure. That is the right treatment. Anyone who hands you a single number blending them has made a weighting choice, and the weighting is doing more work than either input.
It is also worth understanding what the pairing replaced. Passer rating was built from four rates with hard ceilings and no situational awareness at all, and its persistence has more to do with familiarity than merit. CPOE and EPA between them answer the two questions passer rating gestures at and cannot address: was the throw hard, and was the play worth anything.
Completion percentage over expected in college football, and why the number travels badly
The metric gets quoted about college quarterbacks constantly, usually in the context of draft evaluation, and three things go wrong in the translation.
The first is coverage. Tracking of the kind the professional league installs is not present across the college game, so a college CPOE is generally built from a play-by-play model with a handful of variables. That is not fatal, but it means the figure is measuring the whole passing operation rather than the quarterback, and it should be read as a team statistic with a quarterback's name on it.
The second is the baseline, and it is the error people make most often. A completion probability model is fitted to a population, and zero is that population's average. Fit it on college data and zero means an average college passer. Fit it on professional data and zero means an average professional. A college figure and a professional figure are not the same quantity, are not on the same scale, and cannot be compared by size, direction or rank. Putting them in the same table is a category error, and it happens every spring.
The third is variance in the inputs. Professional teams are close together in quality by design, because the whole apparatus of the draft and the cap exists to keep them there. College football has no such mechanism, so the spread between opponents is enormous, and a model with no opponent adjustment attributes all of that mismatch to the quarterback. Scheme variance is wider too: a passing game built on the horizontal stretch and quick reads of an air raid system produces a completely different attempt profile from an option offence, and the model is trying to price both from the same coefficients.
There is now a fourth complication, and it is recent. A quarterback's college record may span three programmes in four years, because the transfer portal turned roster building into an annual market. A career CPOE aggregated across three systems, three sets of receivers and three levels of opponent is not one measurement. It is three measurements stacked, and stacking them averages away the only thing worth knowing, which is how he threw in each.
One number, two completely different quarterbacks
The last problem is the one that survives every improvement to the model, because it is not a modelling problem at all. It is an averaging problem.
Here are two more invented passers. Both throw half their passes short, three tenths at intermediate depth and a fifth deep. Both complete 66.8 per cent of their attempts against a model expectation of 64.6 per cent, so both finish on a CPOE of plus 2.2 percentage points. Every figure is constructed.
E is an exceptional short-area passer who cannot throw the deep ball. F is ordinary underneath and outstanding down the field. They are different players, they need different receivers, they suit different offences, and they would be coached in opposite directions. Their season figures are indistinguishable.
An offensive coordinator building around E would use motion, quick game and defined reads. Building around F, he would use play-action, deep shots off it and a protection scheme that buys time, which is why the play-action pass is a scheme decision rather than a call and why it suits some passers and not others. One number cannot carry that distinction, and no refinement to the completion probability model will make it.
Reading a CPOE figure without being fooled
Four questions, in order. Ask them before you accept any CPOE number from anybody, including a very confident graphic.
Whose model produced it? A figure from a tracking-based model and a figure from a public play-by-play model are not the same quantity and cannot be compared. They will not even rank quarterbacks in the same order. The number is meaningless without its provenance, and providers rarely put the provenance on the graphic.
What is in the feature set? Specifically, does it include receiver separation and pressure. A model that includes both is measuring ball placement. A model that includes neither is measuring the whole passing operation and calling it the quarterback. Both are legitimate; they answer different questions and the difference is invisible in the output.
What was filtered out? Spikes, throwaways and passes batted at the line all belong in the denominator only if you have decided they do. Different providers decide differently and none of them mention it.
How is it split? A season CPOE is an average across throw types, and the average conceals the shape. The same figure can come from a quarterback who is superb underneath and poor downfield, or the reverse, and those are entirely different players to build an offence around. Splitting by air-yard band is the single most informative thing anyone can do with the metric, and it is done far less often than it should be, because one number fits in a graphic and five do not.
Ask those four and CPOE becomes what it should have been all along: a well-defined residual with known blind spots, useful for one specific comparison and honest about the rest. Skip them and it is a leaderboard whose ordering was decided by a modelling choice nobody showed you.
The rest of the sport's measurement problems, from the ones that work to the ones everybody keeps using anyway, are collected in the American football archive.
Common questions
What is completion percentage over expected in football?
CPOE is the gap between how often a quarterback completed his passes and how often a model says a typical quarterback would have completed the same passes. It is expressed in percentage points, and zero means exactly average against that model. Because it is a residual, it exists only relative to whichever model produced the expectation, and two models will give the same quarterback two different numbers.
How is CPOE calculated?
Every pass is assigned a completion probability by a model before the outcome is known. On a completed pass the quarterback banks one minus that probability, and on an incompletion he banks zero minus it, so a completed thirty per cent throw is worth plus 0.70 and a dropped ninety per cent throw is worth minus 0.90. Those per-play values are averaged across all attempts and multiplied by a hundred to give a figure in percentage points.
Is CPOE better than completion percentage?
For comparing quarterbacks it is much better, because raw completion percentage mostly measures how far downfield a quarterback throws rather than how accurately he throws. Two passers can complete the identical share of their attempts with opposite CPOE figures, one having thrown nothing but screens and the other having thrown into coverage. CPOE is not a complete measure of quarterback play, though, since it treats every completion as equally valuable regardless of down, distance or yards gained.
What is a good CPOE?
Zero is average by construction, because the model is fitted to the league, so any positive figure means a quarterback completed more passes than the model expected of his attempts. There is no fixed threshold for good, and the practical spread depends entirely on the model's inputs: a model that knows how open the receiver was will compress everybody towards zero, while a model that knows only the distance of the throw will spread quarterbacks much further apart. Always establish whose model produced the number before judging its size.
Does CPOE account for dropped passes?
Not in its standard form. A drop is recorded as an incompletion, so the quarterback is charged the full negative value of a throw the model expected to be caught, even though the ball arrived where it was supposed to. Some analysts adjust for drops using charting data, which is subjective and inconsistent between providers, and the same blindness applies to receivers who make catches nobody should make.
Filed under American Football·nfl · american football · quarterbacks · football statistics · analytics