Data scientist, physicist, and fantasy football champion

Week 10 K - Evaluation

As good as I am at all of this I can sometimes get a little excited to click “publish” and I don’t always double check my code. For example, I missed taking the absolute value in my accuracy scoring for weeks. That could have been disastrous, but fortunately it had little effect overall.

I say this because I just noticed that in my week 10 kicker prediciton post I used the wrong models for C and D because I just wasn’t looking closely enough. Here I changed it to the correct models, and it didn’t have large effect on the models, but I’m still a little frustrated. I’ll be more careful in the future (about this one particular thing).

In Week 10 I tested four of my own models. Every model used the same 5 factors (once I did it right):

  • The team’s predicted score
  • The opponent’s predicted score
  • Which team is the home team (i.e., which stadium)
  • Who the opponent is
  • Whether the player’s team is home or away

There were slight differences in the models:

  • Model A was a survival model (Weibull regression) using 2016 data only
  • Model B was a survival model (Weibull regression) using 2015 and 2016 data
  • Model C was a linear model (normal regression) using 2016 data only
  • Model D was a linear model (normal regression) using 2015 and 2016 data

In the table below I also included the actual rank and scores for the kickers this week and the average rankings of the top 20 most accurate (kicker accuracy) from FantasyPros. It changes a little through the week but I collected it Sunday morning.

Rank

Model.A

Model.B

Model.C

Model.D

FP.20

Actual.Rank

Actual.Score

1

Nick Novak

Cairo Santos

Nick Novak

Dustin Hopkins

Justin Tucker

Dustin Hopkins

17

2

Phil Dawson

Dustin Hopkins

Mike Nugent

Stephen Gostkowski

Stephen Gostkowski

Ryan Succop

14

3

Mike Nugent

Nick Folk

Dustin Hopkins

Nick Folk

Mason Crosby

Brandon McManus

14

4

Chandler Catanzaro

Dan Bailey

Jason Myers

Nick Novak

Chandler Catanzaro

Steven Hauschka

14

5

Dustin Hopkins

Greg Zuerlein

Chandler Catanzaro

Josh Brown

Matt Bryant

Dan Bailey

14

6

Jason Myers

Nick Novak

Josh Lambo

Justin Tucker

Dan Bailey

Cairo Santos

13

7

Josh Lambo

Josh Brown

Phil Dawson

Chris Boswell

Josh Lambo

Chandler Catanzaro

12

8

Caleb Sturgis

Stephen Gostkowski

Caleb Sturgis

Connor Barth

Brandon McManus

Caleb Sturgis

11

9

Steven Hauschka

Chris Boswell

Andrew Franks

Mike Nugent

Cairo Santos

Phil Dawson

11

10

Dan Bailey

Graham Gano

Josh Brown

Josh Lambo

Caleb Sturgis

Greg Zuerlein

10

11

Josh Brown

Steven Hauschka

Nick Folk

Cairo Santos

Stephen Hauschka

Jason Myers

10

12

Chris Boswell

Connor Barth

Brandon McManus

Chandler Catanzaro

Will Lutz

Matt Bryant

10

13

Brandon McManus

Mike Nugent

Chris Boswell

Dan Bailey

Graham Gano

Justin Tucker

9

14

Nick Folk

Justin Tucker

Dan Bailey

Blair Walsh

Robbie Gould

Mike Nugent

9

15

Mason Crosby

Josh Lambo

Matt Bryant

Greg Zuerlein

Chris Boswell

Nick Novak

9

It doesn’t really look like any of these models really nailed it, but let’s see what happens when we use my version of FantasyPros’ accuracy algorithm. Remember how the accuracy algorithm works? In Model A I said that Nick Novak would be the top player, which means that I predicted that he would get 17 points which is what the real #1 player (Dustin Hopkins) got this week. That means that my error is the difference between his actual score (9 points) and my predicted score. Do this for every player, take the sum and you have the model accuracy. You can also divide it by the number of players evaluated (15) to get the mean difference between my prediction and reality, but we’ll see in a second that those numbers are depressing

There are also a few other points of caution. You should check out the Week 10 DEF evaluation post to see all of these. It feels a little silly to repeat them all here.

Model.A

Model.B

Model.C

Model.D

FP.20

46

40

46

50

44

They all did about the same, but Model B was a little better than the rest. If I look at the distribution of differences for each player and run a t-test then none of these are statistically significantly different from each other. In other weeks Models C and D were the best and I guess in this week it was Model B. Still though, I’m proud that I’m somewhat close the FantasyPros experts predictions.

I can’t shake the feeling that I should be leaning toward models A and C which both use this year’s data. Things change between years: quarterbacks, coaches, offensive and defensive lines. Each of these things should affect a player from year to year, but this week the model that did best was the one that was slightly closer the the historical values (meaning from both 2015 and 2016). OK fine, so the means are close, but what if there are just more big outliers in Models A and C that pull their accuracy down? Maybe the distributions of differences change between the models, and we’d see that A and C actually skew lower:

Nope. Hmmm, weeelllllll, maybe a little.

The thick line in the middle of the box is the median while the top and bottom edges are the 25th and 75th percentile. What I wanted to see to justify my love of Models A and C is those boxes shifted a little lower with one or two big outliers up high (or the whiskers extending very high). The whiskers are indeed a little higher for those models and the outlier in Model A probably pulls its accuracy down a bit. Still, the median lines are all the same for each model and I can’t justify strongly preferring one model over the other unless that line was a bit lower.

The median player prediction was off by 3 points in my models and 2 points in FantasyPros. That’s a single extra (or one fewer) field goal, so it’s probably understandable. I’ll keep working on it.

Week 11 DEF Predictions

Week 10 DEF - Evaluation