Build an NFL win probability model (Super Bowl Sunday Edition)

Photo by David Eulitt, Getty Images

Use Python and sklearn to model NFL game outcomes and build a pre-game win probability model. Super Bowl prediction at the end of the post!

If you have any questions about the code here, feel free to reach out to me on Twitter or on Reddit.

If you have any questions about the code here, feel free to reach out to me on Twitter or on Reddit.

Build an NFL Game Outcome and Win Probability Model in Python

This post was originally written for Open Source Football here, and has been adapted to the Fantasy Football Data Pros blog to share with my readers.

In this post we are going to cover predicting NFL game outcomes and pre-game win probability using a logistic regression model in Python. And since it's almost Super Bowl Sunday, at the end of the post we will be using the model to come up with a super bowl prediction!

Previous posts on Open Source Football have covered engineering EPA to maximize it's predictive value, and this post will partly build upon those written by Jack Lichtenstien and John Goldberg.

The goal of this post will be to provide you with an introduction to modeling in Python and a baseline model to work from. Python's de-facto machine learning library, sklearn, is built upon a streamlined API which allows ML code to be iterated upon easily. To switch to a more complex model wouldn't take much tweaking of the code I'll provide here, as every supervised algorithm is implemented via sklearn in more or less the same fashion.

As with any machine learning task, the bulk of the work will be in the data munging, cleaning, and feature extraction/engineering process.

For our features, we will be using exponentially weighted rolling EPA. For those who don't know what EPA is - it stands for estimated points added, and is a statistic that quantifies how many points a particular play added to a team's score. It's based on a lot of things - field position, yards gained, down and distance, etc. and is generally a good way to quantify and measure offensive and defensive efficiency/performance. A 30 yard play on 3rd and 13, for example, might have 3.2 estimated points added.

We will be taking average EPA per play for each team (split in to rushing, passing, and then further split in to defense, offense) for each week and then calculating a moving average (we will also lag the data back one period, so we only use data up to and not including a particular game). Moreover, instead of calculating a simple moving average, we will be calculating a exponential moving average, which will weigh more recent games more heavily.

The window size for the rolling average will be 10 for all teams before week 10 of the season. This means that prior to week 10, some prior season data will be used. If we're past week 10, the entire, and only the entire, season will be included in the calculation of rolling EPA. This dynamic window size idea was Jack Lichtenstien's, and his post on Open Source Football on the topic is linked above. His post showed that using a dynamic window size was slightly more predictive than using a static 10 game window.

EPA will be split in to defense and offense for both teams, and then further split in to passing and rushing. This means in total, we'll have 8 features:

1. Home team passing offense EPA/play

2. Home team passing defense EPA/play

3. Home team rushing offense EPA/play

4. Home team rushing defense EPA/play

5. Away team passing offense EPA/play

6. Away team passing defense EPA/play

7. Away team rushing offense EPA/play

8. Away team rushing defense EPA/play

The target will be a home team win.

Each of these features will be lagged one period, and then an exponential moving average will be calculated.

We're going to be using Logistic Regression as our model. Logistic Regression is used to model the probability of a binary outcome. The probability we are attempting to model here is the probability a home team wins given the features we've laid out above. We'll see that our LogisticRegression object has a predict_proba method which shows us the predicted probability of a 1 (home team win) or 0 (away team win). This means the model can be used as a pre-game win probability model as well.

We'll be training the model with data from 1999 - 2019, and leaving 2020 out so we can analyze it further at the end of the post.

To start, let's install the nflfastpy module, a Python package I manage which allows us easy access to nflfastR data.

Next, we import some stuff we'll need for this notebook and also set the base styling for our matplotlib visualizations.

The code block below will pull nflfastR data from the [nflfastR-data](https://github.com/guga31bb/nflfastR-data) repository and concatenate the seperate, yearly DataFrames in to a single DataFrame we'll call data. This code block will take anywhere from 2-5 minutes.

The code below is going to calculate a rolling EPA with a static window and a dynamic window. I've included both, although we'll only be using the rolling EPA with the dynamic window in our model.

We can plot EPA for the Green Bay Packers alongside our moving averages. We can see that the static window EMA and dynamic window EMA are quite similar, with slight divergences towards season ends.

Now that we have our features compiled, we can begin to merge in game result data to come up with our target variable.

We'll set some variables to isolate our target and features. Here, we are only including the dynamic window columns. I've found that the dynamic window does produce a slightly better model score. You're welcome to use the EWMA features with the static window size.

Here, we finally train and test our model. In sklearn, each supervised algorithm is implemented in the same fashion, First, you bring in your class (which we did earlier in the code). You then instantiate the class, providing model hyperparameters at instantiation. Here, since we're just doing Logistic Regression, we have none (although sklearn allows us to provide C as a hyperpameter - which controls regularization). Then, you call the fit method which trains your model. For evaluating model accuracy, you have a couple options. Notice here we did not split our model in to train and test sets, as we'll be using 10-fold cross validation to train and test instead using the cross_val_scores function we brought in earlier.

We see our week model has about a 63.5% accuracy score. Not terrible, considering if you head over to nflpickwatch.com, you'll see that the best experts tend to cluster around 68%.

We also found the negative log loss. This is important because the model could have value as a win probability model as opposed to a straight pick-em model.

The model can definitely be improved. Some possible improvements and things I'd like to explore in future iterations of this model:

1. Using John Goldberg's idea of opponent-adjusted EPA

2. Weighing EPA by win probability

3. Engineering other features beyond EPA, including special teams performance

4. Switch to a more complex model + hyperparameter tuning

Something fun we can do now is see how the model would have predicted this past season. Notice we took out 2020 data from our training data, and so now we are free to see how the model would have done in 2020 (since none of the 2020 data was used for training).

These are the 10 games this season the model was most confident about. No surprises here that the KC-NYJ game was the most lopsided game this season.

We can also view how our model would have done this season by week. There doesn't seem to be a clear trend here. You would expect that the model would get better as the season went on, but the data doesn't make that clear here.

The model would have done the best in week 2 of the 2020 season, with a 87.5% accuracy score!

We can also view how our model would have done in the playoffs by filtering out weeks prior to week 17.

Of course, an NFL game prediction post a week out from the super bowl isn't complete without a super bowl prediction.

Final Thoughts

The goal of this post was to provide you with a baseline model to work from in Python that can be easily iterated upon.

The model was came out to be around 63.5% accurate, and can definitely be improved on. The model may also have more value as a pre-game win probability model.

In future posts, I would like to incorporate EPA adjusted for opponent and also look at how WP affects the predictability of EPA, as it's an interesting idea and I think there could be some value there. Also, trying out other features besides EPA could certainly help. Turnover rates, time of possession, average number of plays ran, average starting field position, special teams performance, QB specific play, etc, could all prove useful. We saw in the visualization of feature importance that passing EPA per play had the most predictive power for a home team win and away team win. Exploring QB specific features could help improve the model. Of course, that would require roster data for each week.

As always - thank you for reading!