How do you rate a prediction? This is a surprisingly tricky question.
Let's say you were required the guess the actual weekly score. Now, imagine that every DEF that week scored exactly 2 touchdowns (it was a crazy week, lots of picks and fumbles after they switched from leather/rubber/polyurethane to teflon this week. All of your scores would be off this week by 12 points in standard scoring, but the actual rank order might be very close.
What if you're only required to submit a predicted order for the week. Now imagine that the results were heavily tiered. Your #1 through #5 predictions all got 18, 19, 20, 21, and 22 points, respectively, while the remaining teams all grouped around 5 points. In this case, your predicted order for the top 5 teams is exactly wrong, but you got that those 5 would be in the top tier that week. So you were accurate, but not if you just look at the order.
FantasyPros uses a unique scoring algorithm (see here) that I've tried to replicate in my code. It's been validated by experts and I like it well enough. It gives higher scores for tiered situations like above and is as good as anything I've come up independently. It works as follows:
Imagine that you predicted the weekly order above. At the end of the week you have the actual order above. Is this good? Your model couldn't even predict which team placed 1st for the week. Bro, do you even fit?
What FantasyPros does is it takes your ranking and assigns the real score for the #1 team to who you said was going to place first and so on. Thought BUF would be in 4th? Then whatever the 4th place ranked team scored (here, 6 pts) that's how much you predicted BUF would get this week.
Now that you have the actual and predicted scores for each team, take the absolute value of the difference. FantasyPros then uses the top 15 DEF to rate their pros. I can take the sum of the differences to quickly see which model is better or do something like a t-test to show that the averages are actually significantly different. For example, according to my accuracy I beat the average of the top 20 FantasyPros experts in weeks 8 and 9, but not by a statistically significant amount. But the mean difference between actual and prediction was better in my model, and at the end of the day isn't that what really matters?
No. Crushing my idiot brother and his idiot friends, seeing them driven before me, and hearing the lamentations of their women (and men) is what is best. But second to that is an accurate data analysis.