Posted by Neil Paine on August 31, 2009
They've been having a spirited debate about retrodiction over at APBRmetrics recently, and apparently we're all going to engage in some sort of challenge over whose pet metric could "retrodict" a team's 2009 performance the best. This, of course, should be nothing new to faithful readers of the BBR blog, because we retrodicted 2008-09 several months ago. But my question is, does accurate retrodiction across the entire league really "prove" anything? For the vast majority of teams, any cheeky wee monkey is going to be able to predict performance effectively based on data from the past few years because A) roster turnover is rarely drastic enough to the point that a team's stars are no longer with the team anymore, and B) even in the case of roster shakeups, coaches almost always employ new players in the same role they had been playing in for their previous team. Since we know a lot of basketball productivity is role-specific, there's really not a lot of point in boasting that past production in your metric predicted future production when a player plays exactly the same role in each situation.
To me, it seems like the more telling cases would be ones where there was a lot of roster shakeup, where players were asked to switch roles. The mythical Team of Fred Hoibergs (the "Ames Mayors"?) would be the prototype, right? I mean, you can easily predict the productivity of a player like Hoiberg in Year Y based on Y-1, Y-2, etc. if he's being used the same way in each of the seasons (low-usage 3-point specialist)... But good luck using Wins Produced, PER, Adjusted +/-, or the like to predict his production if in Year Y he's asked to be, say, the primary scorer on the team.
Now, this kind of dramatic change doesn't happen often. Like I said, players are almost always used in comfortable roles, ones similar to those they've been asked to fill in the past. But the real progress is to be made in situations of shakeup, of turmoil, where players are asked to do something they aren't familiar with. To that end (and this is only scratching the surface, I realize, but we'll probably touch on this often in the future), I thought that instead of retrodicting every team, maybe we should only focus on teams with obvious personnel changes, where the unpredictability is going to be highest because of changing roles and team dynamics. So here's a list of the teams (since 1965) with the lowest "Continuity Scores" (continuity being the % of team minutes being filled by players who were on the roster the season before):
It seems to me that retrodicting these types of teams would be more informative than the typical squad, because your metric is going to have to rise or fall on its ability to anticipate how a player will react to a different role on the team. We're always talking about "Holy Grails" when it comes to player ratings, and figuring out a way to effectively retrodict situations like this would go a long way toward developing the best metric for building future teams.