Rockets fan left the following comment on Andres Perezchica’s recent article for the Wages of Wins Journal concerning the D-League:
To echo a question I’ve raised elsewhere — but haven’t seen addressed — what is a reasonable estimate of the [Wins Produced model’s] error margin? There are some obvious problems with the metric. (For example, it can’t attach a number to plays where a defender’s defense makes an offensive player miss a shot, it values all assists the same, and it does not account for charges drawn.) To be clear, I’m not saying the metric is bunk. But, I think it’s beyond dispute, that it isn’t perfect. Given that it’s not perfect, how imperfect is it? Is [.07]1 really worse than [.09]1? Can the WS make such fine distinctions? I don’t know, and I’d be interested in reading an answer.
One of my biggest complaints about [The Wages of Wins Journal] is that, even though we all accept there’s some [margin of error], in nearly all posts the implicit assumption is that a higher [wins produced] necessarily means the individual contributed more wins. In other words, a player with a .150 [WP48] will be treated as obviously better than a player with a .135. I’m not sure that’s the case. Sorry to (try to) highjack a thread, but I feel like my question comes up in nearly every post — including this one.
Perhaps I can offer a bit of clarification to Rockets fan and anyone else who is unsure of the implications involved in comparing players using the Wins Produced family of production metrics.
The effect of minutes played
As a sample of minutes that a player has played increase, the WP48 as calculated for that period will more closely reflect that players ability, and it’s implications become larger.
To show this point, here’s a table that lists the difference in production (Δ Wins Produced) between two players for a given number of minutes (assumed to be the same) at various differences in their rates of production (Δ WP48).
This table shows that subtle differences in WP48 (Δ WP48 of .020 and less) don’t have a large effect on wins produced until the two players approach starters minutes. So if two starters play 2800 minutes each, and the first of the two has a WP48 of 0.100 and the second has a WP48 of 0.120, then the second player will produce 1.17 more wins over the course of the season, which I would argue is significant. But if those same players have the same WP48, but only play 400 minutes, then the second player will produce only 0.17 more wins, which certainly is not very significant.
On the precision of WP48
WP48 is a precise calculation. All else being equal, having a WP48 of 0.150 is (very slightly) preferable to a WP48 of 0.149. The reason for this is that WP48 is the best model available to describe the rate of production of players in the NBA, and an increase in WP48 in isolation is very likely to lead to more wins2 . To say that player a produced at a WP48 of .150 instead of .149 over the course of a season is akin to saying that he got 1003 rebounds rather than 1000. It’s not a big difference, but everything else being equal, you would take the 1003 over the 1000.
One of the strengths of WP48 however, is that over a season’s worth of minutes, player production as expressed in terms of WP48 is relatively consistent (unlike adjusted plus/minus, for example). This tells us that if a player is productive this year, he is likely to be productive next year3. In practical application the Wins Produced model will generally explain a teams win/loss record for a given season to within 2 wins. Usually there will be a couple outliers that under/over perform the win/loss record predicted by the Wins Produced model by about 4 wins. For more on this, see the following posts by Dr. Berri: Proof and the NBA and The Differing Stories on Durant – and a Brief Thunder Review.
In summary, to say that player 1 has a higher WP48 than player 2 is to say that, when considering only the factors included in the Wages of Wins model, that player 1 was more productive on a per 48 minute basis than player 2. This is true regardless of whether there is a difference of .001 WP48 or of .300 WP48 between the two players. There are other factors outside the scope of WP48 that could mean that player 2 is more productive than player 1 in absolute terms, but these factors are both unknown and of relatively small impact. Therefore, when evaluating the production of players in the NBA, it is best to assume that player 1 is more productive than player two, at least until the Wages of Wins model is improved to have a smaller error, or until another model with a smaller margin of error becomes available.
1 Rockets fans question actually used the numbers .7 and .9, but I’m assuming that .07 and .09 were meant as the former numbers only come about in very small sample sizes and are not really reflective of a players actual ability.
2Note that I am using this number for pedagogical purposes and in reality, if a player increases his WP48 by .001 in say 2400 minutes of play, he will have helped his team by 0.058 wins which is not likely to have any parctical effect on the teams win/loss record.
3There are some well know caveats to this generalization. Very early career production (i. e. the first couple seasons a player plays in the NBA) is often much more volatile than production from mid-career seasons. Players are also less likely to maintain production after the age of 30, and especially after the age of 32.
I’m assuming that Rockets fan’s question was actually in regards to the statistical error associated with wp48. For example, not only does Dr. Berri not like adjusted plus/minus because it doesn’t correlate well across seasons, but within a season the errors are so large that it’s difficult to compare players. I’m making numbers up, but Kevin Durant might be a +6 but the error term is +- 5, meaning he could be anywhere from amazing to average. What is that number, the +-5, like for wp48? If a player posts a .100 one year, what would he have to post the next year for me to be pretty sure he got better, as opposed to there being a good chance he played just as well? .101 seems non-significant to me, but .105? .110?
Interesting question, Alex. I don’t think that there’s a really solid answer to that. Mostly, WP48 is a summation of individual player production, so I think that my assertion that any increase in WP48 is good, all else being equal, stands. To find an area of the Wins Produced model that would allow for the possibility that a player with a .100 WP48 is really more productive than a player with a .101 WP48, you would have to look at the parts of the model that are not specifically tied to the box score numbers produced by a particular player.
The area of the model which has the largest potential to lead to some inaccuracy in a players WP48, in my opinion, is the way that individual defense is incorporated. In case you are unaware, WP48 does incorporate team defense, and distributes this among the teams players based on minutes. It should be noted however, that adding individual defense has a relatively small affect, even in extreme cases (i.e. if a player has a WP48 of 0.000, then even if that player is the best individual defender in the league, he would not be able to approach an average WP48 of .100 if individual defense were incorporated into WP48, in fact, defense in general has a relatively small impact compared to shooting efficiency, rebounds, and turnovers, all of which are well accounted for in WP48). All of the factors that most affect wins are incorporated into WP48 already. The reason that individual defense is left out of WP48 is that it would add a lot of complexity to the model without increasing it’s explanatory power by much. For more discussion on this topic, see Dr. Berri’s article Incorporating Defense from The Wages of Wins Journal. Here is a relevant excerpt:
Models are not supposed to be “perfect” (whatever that means). When I and my colleagues construct models, we are trying to construct a simplified version of reality that allows us to focus on what is important (and answer the various questions we pose in our research).That is what I think Wins Produced does. It is a simple and accurate measure of performance, based on the theoretically sound idea that wins are determined by a team’s offensive and defensive efficiency. This model ultimately tells us that wins are primarily determined by shooting efficiency, rebounds, and turnovers. Yes, other issues matter. But players who do not score efficiently, who fail to rebound (given their position), and/or turn the ball over excessively, will not help you win games.
So, my answer is that we might conservatively estimate that a players WP48 is within 0.030 of his “true” win production per 48 minutes for players who excel in, or conversely are extremely poor with regard to, all of the areas that are not considered in the calculation of WP48. Any given player’s WP48 will necessarily be close to his “true” win production per 48 minutes. If he is a great individual defender, then WP48 may slightly undervalue him. If his assists are better than the the average assists, then again, WP48 may (very, very slightly) undervalue him. If one wishes to take those areas which are not explained by WP48 into account, then it is ones prerogative to do so, but caveat emptor that you are deviating from the science, and unless you know the true impact on wins of the variable you are adjusting, you are more likely to get a less accurate picture of the player’s true production than if you had assumed that WP48 was the player’s true production.