Grading Nate Silver and 538

It is no secret that I am a huge fan of fivethirtyeight.com and its creator, Nate Silver. Since being created in the span of less than a year, the site has become one of the most popular political blogs in the country, and it has made Silver a minor celebrity. For evidence of this, look no further than his plethora of appearances on tv news shows and a glowing New York Times profile of him today with, get this, four separate pictures of Silver. You know you have hit it big when the Times does an article on you with four pictures.

Needless to say, I think the attention is very well-deserved. For those of you unfamiliar with 538, I encourage you to do three things right away: go to www.fivethirtyeight.com, promptly bookmark it, and make it one of your daily stops.

Basically, Silver, a genius and baseball statistician extraordinaire takes polling, demographic, and other data and uses it with a statistical model he has created to predict the outcomes of political races. He takes all of the polling done for a particular state and feeds it into his model with assigned weights, and has generated projections for all the states all year, updating them daily. He used his skill to tremendous effect in correctly pin-pointing many of the presidential primary results this year.

Besides providing fascinating insights into what all of the available polling and political numbers tell us, I like 538 for another reason. Despite never being a political operative or pundit, I find Silver's actual writing and free-wielding analysis to be some of the sharpest stuff out there. Period. In my humble opinion, he is one of the best pure political thinkers around, much better than individuals who have been covering politics for their lives. In other words, it is not just about the cool color-coded charts, maps and numbers for me -- Silver has smart stuff to say as well.

Anyway, through his model, Silver correctly predicted the overall outcome in 49 of the 50 states in the general, getting only Indiana wrong. For this, he has been getting an enormous amount of praise from respected political watchers and others.

However, I do not agree with this line of adulatin, and I think that there is a better way to go about assessing 538. Frankly, I do not think it is all that hard to accurately predict the base outcome of a statewide election. For example, I correctly predicted the outcome in 47 states, missing only Indiana, Montana and North Dakota (though, to be a bit fair, while I stand by them completely, the latter two were my upset specials, kind of gimmicky). Does that make me a genius? No way. It was not terribly hard to look at over the states and say who was going to win them: around 40 (if not more) were pretty much done-deals by the end, with most of those decided the minute the general election season started. This is not to say that getting 49, or 45, or 47 right is not impressive; it is, and Silver did a great job. I just would like to evaluate his work a little more precisely so we can properly decide how good Silver's model was for election 2008.

Consequently, what I am going to do in this post is very simple. I am going to take our very simple methodology for grading the swing state pollsters and apply it to the final state projections Nate Silver calculated for all 50 states (which are still up on his frontpage on the right side) and see how close he was, on average, to the actual outcomes.

Just as a refresher, what we are going to do is determine how close both halves of Silver's state projections were to the actual outcome. For example, if Silver's final projection for Texas was 55-42 McCain, and the final result was 57-41 in favor of McCain, Silver would get a score of 3 for the state because his number for McCain was two points off and his number for Obama one point off the end results. The score provided represents on average, how close one was to the two candidates' final percentages each, combined. Therefore, the lower the score, the closer you are to being perfect. At the end, I will average all 50 scores 538 got so we can get a sharper picture of Silver's accuracy than just by saying he got 49 state outcomes correct.

(I recognize the subtle irony here in rating 538, since Silver's model relies entirely on polling data to fashion projections and he also rates the pollsters himself based on their own accuracy.)

Let's get to it, and if you would just like read my rating(s) for 538 and my analysis, just scroll past the state-by-state numbers.


Alabama (Actual result: McCain 60.4, Obama 38.8)
538 projection: McCain 60.8, Obama 38.5
T2L Score: 0.7

Alaska (Actual result: McCain 61.5, Obama 36.2)
538 projection: McCain 56.4, Obama 42.2
T2L Score: 11.1

Arizona (Actual result: McCain 53.8, Obama 45)
538 projection: McCain 51.3, Obama 46.4
T2L Score: 3.9

Arkansas (Actual result: McCain 58.8, Obama 38.8)
538 projection: McCain 55, Obama 44
T2L Score: 9.0

California (Actual result: Obama 61, McCain 37.2)
538 projection: Obama 59.2, McCain 38.6
T2L Score: 3.2

Colorado (Actual result: Obama 53.5, McCain 44.9)
538 projection: Obama 52.7, McCain 46.1
T2L Score: 2.0

Connecticut (Actual result: Obama 60.6, McCain 38.3)
538 projection: Obama 58.8, McCain 39.1
T2L Score: 2.6

Delaware (Actual result: Obama 61.9, McCain 37)
538 projection: Obama 61.5, McCain 37.4
T2L Score: 0.8

Florida (Actual result: Obama 50.9, McCain 48.4)
538 projection: Obama 50.1, McCain 48.4
T2L Score: 0.8

Georgia (Actual result: McCain 52.3, Obama 47)
538 projection: McCain 51.2, Obama 47.5
T2L Score: 1.6

Hawaii (Actual result: Obama 71.8, McCain 26.6)
538 projection: Obama 62.4, McCain 34.9
T2L Score: 17.7

Idaho (Actual result: McCain 61.5, Obama 36.1)
538 projection: McCain 59, Obama 38.4
T2L Score: 4.8

Illinois (Actual result: Obama 61.7, McCain 37)
538 projection: Obama 60.2, McCain 38.4
T2L Score: 2.9

Indiana (Actual result: Obama 49.9, McCain 49)
539 projection: McCain 50, Obama 48.4
T2L Score: 2.5

Iowa (Actual result: Obama 54, McCain 44.7)
538 projection: Obama 54.6, McCain 42.9
T2L Score: 2.4

Kansas (Actual result: McCain 56.8 Obama 41.4)
538 projection: McCain 55.8 Obama 42.2
T2L Score: 1.8

Kentucky (Actual result: McCain 57.5 Obama 41.1)
538 projection: McCain 56 Obama 42.4
T2L Score: 2.8

Louisiana (Actual result: McCain 58.6 Obama 39.9)
538 projection: McCain 54.2 Obama 44.3
T2L Score: 8.8

Maine (Actual result: Obama 57.6 McCain 40.5)
538 projection: Obama 56.1 McCain 41.6
T2L Score: 2.6

Maryland (Actual result: Obama 61.9 McCain 36.8)
538 projection: Obama 61.1 McCain 38.1
T2L Score: 2.1

Massachusetts (Actual result: Obama 62 McCain 36.2)
538 projection: Obama 59 McCain 39.1
T2L Score: 5.9

Michigan (Actual result: Obama 57.4 McCain 40.9
538 projection: Obama 55.2 McCain 42.6
T2L Score: 3.9

Minnesota (Actual result: Obama 54.2 McCain 44)
538 projection: Obama 54.1 McCain 44
T2L Score: 0.1

Mississippi (Actual result: McCain 56.4 Obama 42.8)
538 projection: McCain 54.7 Obama 43.4
T2L Score: 2.3

Missouri (Actual result: McCain 49.4, Obama 49.3)
538 projection: McCain 49.4, Obama 49.2
T2L Score: 0.1

Montana (Actual result: McCain 49.7, Obama 47.2)
538 projection: McCain 50.4 Obama 47.7
T2L Score: 1.2

Nebraska (Actual result: McCain 57, Obama 41.5)
538 projection: McCain 56.9, Obama 42.5
T2L Score: 1.1

Nevada (Actual result: Obama 55.1, McCain 42.7)
538 projection: Obama 51.7, McCain 46.9
T2L Score: 7.6

New Hampshire (Actual result: Obama 54.3, McCain 44.8)
538 projection: Obama 53.8, McCain 44
T2L Score: 1.3

New Jersey (Actual result: Obama 56.8, McCain 42.1)
538 projection: Obama 56.1, McCain 41.5
T2L Score: 1.3

New Mexico (Actual result: Obama 56.7, McCain 42)
538 projection: Obama 54.6, McCain 44.9
T2L Score: 5.0

New York (Actual result: Obama 62.2, McCain 36.7)
538 projection: Obama 62.1 McCain 36.7
T2L Score: 0.1

North Carolina (Actual result: Obama 49.9, McCain 49.5)
538 projection: Obama 50.1, McCain 49.1
T2L Score: 0.6

North Dakota (Actual result: McCain 53.3, Obama 44.7)
538 projection: McCain 50 Obama 47.3
T2L Score: 5.9

Ohio (Actual result: Obama 51.2, McCain 47.2)
538 projection: Obama 50.8, McCain 47.4
T2L Score: 0.6

Oklahoma (Actual result: McCain 65.6, Obama 34.4)
538 projection: McCain 62.7, Obama 37.1
T2L Score: 5.6

Oregon (Actual result: Obama 57.1, McCain 40.8)
538 projection: Obama 56.5, McCain 41.8
T2L Score: 1.6

Pennsylvania (Actual result: Obama 54.7, McCain 44.3)
538 projection: Obama 53.2, McCain 45.1
T2L Score: 2.3

Rhode Island (Actual result: Obama 63.1, McCain 35.3)
538 projection: Obama 60.8, McCain 36.7
T2L Score: 3.7

South Carolina (Actual result: McCain 53.8, Obama 44.9)
538 projection: McCain 54.4, Obama 44.7
T2L Score: 0.8

South Dakota (Actual result: McCain 53.2, Obama 44.7)
538 projection: McCain 53.3, Obama 44.6
T2L Score: 0.2

Tennessee (Actual result: McCain 56.9, Obama 41.8)
538 projection: McCain 55.9, Obama 43
T2L Score: 2.2

Texas (Actual result: McCain 55.5, Obama 43.8)
538 projection: McCain 55.3, Obama 44.5
T2L Score: 0.9

Utah (Actual result: McCain 62.9, Obama 34.2)
538 projection: McCain 61.3, Obama 35.5
T2L Score: 2.9

Vermont (Actual result: Obama 66.8, McCain 31.6)
538 projection: Obama 61.1, McCain 36.5
T2L Score: 10.6

Virginia (Actual result: Obama 52.7, McCain 46.4)
538 projection: Obama 52.2, McCain 46.5
T2L Score: 0.6

Washington (Actual result: Obama 57.5, McCain 40.7)
538 projection: Obama 56.4, McCain 42
T2L Score: 2.4

West Virginia (Actual result: McCain 55.7, Obama 42.6)
538 projection: McCain 53.9, Obama 44.1
T2L Score: 3.3

Wisconsin (Actual result: Obama 56.3, McCain 42.4)
538 projection: Obama 54.6, McCain 43.1
T2L Score: 2.4

Wyoming (Actual result: McCain 65.2, Obama 32.7)
538 projection: McCain 61.9, Obama 36.6
T2L Score: 7.2

All Scores: AL (0.7), AK (11.1), AZ (3.9), AR (9), CA (3.2), CO (2), CT (2.6), DE (0.8), FL (0.8), GA (1.6), HI (17.7), ID (4.8), IL (2.9), IN (2.5), IA (2.4), KS (1.8), KY (2.8), LA (8.8), ME (2.6), MD (2.1), MA (5.9), MI (3.9), MN (0.1), MS (2.3), MO (0.1), MT (1.2), NE (1.1), NV (7.6), NH (1.3), NJ (1.3), NM (5), NY (0.1), NC (0.6), ND (5.9), OH (0.6), OK (5.6), OR (1.6), PA (2.3), RI (3.7), SC (0.8), SD (0.2), TN (2.2), TX (0.9), UT (2.9), VT (10.6), VA (0.6), WA (2.4), WV (3.3), WI (2.4), WY (7.2)

Nate Silver's T2L State Average: 3.36

While we did not tabulate a 50-state pollster average to compare this number to, this is pretty damn good. 538's number for all 50 states is less than three and a half. This is superior than just about all the best pollsters just over the swing states (and thus not impacted by bad scores in blow-out states that are harder to poll).

What if we take out Silver's projections for Hawaii and Vermont, easily the two biggest blowout states for Obama and not to mention, the two biggest blowouts in the whole union?

538's T2L score (subtracting Hawaii and Vermont): 2.91.

That is just incredible. His projection average for the other 48 states comes out to be below three points. Forget predicting the overall win-loss outcome for 49 out of 50. What this means in context is that Silver's projections, on average, were able to come within 2.9 points of the two candidates' final percentages each, then combined. This is much more impressive than just calling a state. Silver did an amazing job in this, his first full year of trying to project results. Even looking at all 50 states, his score of 3.36 is absolutely great.

Let's try to get some better context here by averaging out Silver's score for the 17 swing states we used to rank the pollsters. These states were: Arizona, Colorado, Florida, Georgia, Indiana, Michigan, Minnesota, Missouri, Montana, Nevada, New Hampshire, New Mexico, North Carolina, North Dakota, Ohio, Pennsylvania, and Virginia. From these we get:

538's swing state score: 2.36

Another superb showing. If I were to put him on my swing state pollster rankings, Silver would sit in first place, slightly above CNN/Time and their firm Opinion Research. For a first-time political prognosticator, this is pretty damn spectacular.

Granted, it is important to remember that Silver derived his own projections just about entirely with the polling data of the firms on my list (as well as many other pollsters). He himself is a not a pollster. That being said, he is not trying to be a pollster, so much as someone who is attemping to dissect complex data for laymen like myself, and to come to projections from that data. Clearly, he knows exactly what he is doing.

Conclusion

To be totally honest, when I set out to go through all of 538's state projections, I did not think his projections would turn to be perfect. As I have said many times, I think 538 is the best political blog out there period, but I wondered just how close he would be. Well, comparing his numbers with my rudimentary and simplistic formula, I see he did better than I could have imagined.

This is not a report card, but if it were, Nate Silver and 538 deserve an A+++. And the scary thing is that I only expect his model to get sharper as he learns from some kinks this cycle and tinkers with his model.

Oh, and you know what his national popular vote projection was? Obama 52.3, McCain 46.2. What was the actual vote? Yup, just about 53-46.

538 deserves every shred of praise it has been getting and more. If I were advising Nate, I would tell him to open a consultant shop right now, as he could make millions.