Comparing box score statistics and ShotQuality expectation metrics to understand the randomness of single game samples to improve prediction accuracy
As we approach the new year, in the 2021-2022 NBA Season there have been over 107,587 shots taken. In a make or miss league, scoring performance has a huge influence on game outcomes and also on advanced metrics like points per possession (PPP) and true shooting percentage (TS%). One of the reasons that many highly regarded power ranking systems had the 2020 New York Knicks rated lower was due to their “luckiness” with opponent 3-point percentage. Are certain scoring statistics a reflection of a team’s underlying talent/skill, or are they simply random noise that will fluctuate game-by-game and cannot predict future performance? If we find that certain metrics are stable for teams and/or players throughout a season, we can incorporate scoring regression to make better predictions out of sample.
Exploring stability of box score statistics
Single game performances tend to be noisy, especially when looking at individual statistics. Using the mean values for each box score statistic for each individual game, we will bucket each team and box score statistic into 5 game rolling windows and find the correlation across teams during this season
These results are fairly intuitive. While the number of points scored and the free throw is less fluky, shooting percentages all over the court fluctuate wildly from game to game. It also makes sense that the free throw has better stability than the live shooting percentages considering a shot taken at a deal ball with no defense will rely on skill rather than luck.
Exploring stability of ShotQuality metrics
ShotQuality expectation variables were created to derive process-based insights on the quality of possessions. This takes into account the individual player’s shot making ability when computing, so these metrics should have better stability and robustness to the noise of a lucky game, right? Let’s repeat the analysis using the SQ expectations.
Wow! The game to game correlation between the ShotQuality metrics is far superior to that of ordinary box score statistics. Putting the correlations on the same axis should further hammer this point home.
We now know that generally, box score statistics are extremely fluky and unstable metrics for teams, especially for shooting percentages. Given this information, how can we take this into account for prediction? If we are going to predict future game outcomes, it might be smart to create a weighted ‘game grade’ model measured by its RSQ to future points scored
(ie, out of sample point differential).
Descriptive vs Predictive
In this analysis, a statistic’s ability to describe the final score of a game is measured by its correlation to the same game’s point differential. A game grade’s ability to predict future performance is measured by its RSQ to future point differential (aka out of sample point differential).
The out of sample RSQ is measured across three rolling windows: 1 game, 4 games, and 8 games. The windows are equally sized (not rolling, expanding, or dynamically moving windows) and constrained to games from this season. For example, a team approaching game number 5 has 4 preceding games (1, 2, 3 and 4) that are used to make a prediction about the team’s next 4 games (games 5, 6, 7, and 8). Those games (5, 6, 7, and 8) are used to predict the next 4 games (9, 10, 11 and 12), and so on:
In general, these metrics by themselves (game grades, weighting metrics and more are a topic of discussion for another time) are not able to predict out of sample very well, and are not better than the point differential. It is intuitive that for each observed box score metric, it is better at describing the same game, meanwhile the ShotQuality metrics are much more accurate at predicting future games. This holds true in every case of this analysis besides one instance of predicted free throw percentage, which is something to investigate in the future.
Wrapping up
The main objective of this post was to look at which scoring metrics we can expect to perform similarly from night to night, and which ones are ultimately random. Understanding which metrics random (and filtering them out from the ‘game score’) and which are stable allows us to better rank the performance of teams on a given night.
We also looked at how well each of these metrics could predict the point differential of a future game, which is where the ShotQuality metrics outperformed the regular box score statistics. This goes to show that ShotQuality metrics account for game to game randomness much better than box score statistics and should be used to assess team performance going forward. In the future, we can investigate how ShotQuality metrics can be used to predict the outcomes of future games.