Review: Nate Silver's 538 Blog and "The Signal and the Noise"

 

By David Pillow

 

Nate Silver’s 538 Blog.

Nate Silver

On the day before the election of Obama and Romney, I predicted amongst my Facebook friends an Obama win.  I justified that prediction based on the work of Nate Silver. At that time, Nate Silver found Obama polling at 50.9% while Romney was getting 48.2% of promised votes. By traditional standards, these estimates fall within a 3 to 5 point margin of error. Some would call that a tie, but Nate Silver was putting the odds of an Obama victory at 92%.   

A day earlier, CNN called the race even. So did Politico, Rasmussen, and Gravis Marketing. That may have seemed to many convincing evidence of a tie.  But then 9 other polls were also released on that Sunday before the election: Pew found a 3 point Obama lead; Rand found a 3.2 Obama lead, YouGov found a 2 point Obama lead, and ABC and NBC each found 1 point Obama leads. Not one of the polls found a Romney lead. The average across all 13 polls gave Obama a 1.1 lead. If Obama and Romney were perfectly tied—much like Bush v. Gore—one would expect to find some polls with ties, some with Obama leading, and some with Romney leading. And the average across all those polls would be very close to a tie.

Averaging across polls is what Nate Silver does. In the sciences and social sciences, the technique is called meta-analysis. It is a quantitative strategy of making sense of numerous scientific studies examining a particular hypothesis. In the past, researchers used to conduct qualitative reviews of the literature when trying to make sense of diverging results. In such reviews, they would declare some of the studies to be good and some to be bad, thereby setting the stage to argue away some of those results so that they might arrive at a tidy, overall conclusion. Meta-analysis emerged as a quantitative strategy of reducing biases in such reviews and has emerged as a powerful tool for summarizing complex research literatures. 

Nate Silver has developed sophisticated models for predicting elections and has been recognized for having among the most accurate predictions in 2008 and 2010.  He “called” the election perfectly in 2012.  Yet it is important to recognize that he didn’t actually “call” the election in the same way that pundits are prone to do.  That is, he didn’t offer a simple, deterministic sounding claim that Obama would win.  Instead, he provided probabilities.  As such, his predictions for every state recognized that there was a chance that his estimates could be wrong—that the estimates were probabilistic approximations of the truth with an inherent degree of error. 

Nate publishes his predictions on a site called:  Fivethirtyeight. 538 is the number of electoral votes that will be cast this election. It takes 270 of those votes to win the Presidential Election, and Nate Silver takes the electoral system quite seriously. He doesn’t just average national polls; he averages the results of all the state polls as well. He then calculates the probability of a win in each state and subsequently combines results across states to make overall predictions of the number of electoral votes each candidate might obtain. Hence, it is theoretically possible for Silver to conclude that a candidate could win lose the popular vote but win the Electoral College—a diverging prediction you’ve never before seen from the pollsters at Gallup. 

Nate argues that we need to distinguish the signal from the noise.  In context of polling, the best indication of the signal is the average of the polls.  Each individual poll that diverges from the average is considered to reflect noise.  Noise in this instance is sampling error—the error that naturally comes about as a result of random sampling process.  That said, each poll, no matter how noisy, gets added into to the estimate.

 

 

Nate Silver’s Book:  The Signal and the Noise.

 

Having followed Nate Silver’s blog with great interest during the election process, I decided to read his book hoping to gain further insights into the statistical modeling that Silver uses.  This book, however, is not anything like a technical report or manual on the science of polling.  In fact, there is (arguably) not even a full chapter devoted to the 538 site, and it certainly doesn’t detail the statistical model Silver uses.  Instead, the book takes us on a number of journeys to explore how individuals often mistake noise for signals to get predictions wrong.  He starts with the financial crisis of 2008 to detail how the rating agencies, banking regulators, mortgage brokers and others made colossal failures of prediction in judging the housing bubble.  Here he shows that there were warnings, but these warning were given insufficient weight.  In the next chapter, he explores worlds of television punditry and political science to show how often “the experts” get their calls wrong, borrowing very heavily from the work of the psychologist, Philip Tetlock. From there, we’re swung into the world of baseball where we learn about the rivals between the “statheads” and “scouts.”  Interestingly, this is one of two areas where Nate Silver made money before the development of the 538 blog.  Nate was a stathead who developed a system for predicting which players in the minors would make it big in the majors and sold the system for a profit.  But before you assume that this is simply a story of how stats beat scouts, don’t.  You would be wrong. Instead, Nate explores the varied contributions of quantitative and qualitative information to provide a much more nuanced perspective.

In his first 3 chapters, Nate explores how various cognitive and self-presentational biases distract us from making accurate predictions.  In the subsequent 3 chapters, he explores how difficult it is to formulate predictive models in dynamic systems, arguing that mistaken assumptions can lead to huge distortions in predictions when the effects are nonlinear.  The topical templates he utilizes here include weather forecasting, earthquakes, and economics.  Silver received a Bachelor’s degree in economics from the University of Chicago, but had I not known his educational background, I could easily have imagined leaving each chapter believing that he had earned advanced degrees in meteorology and geophysics as well.  Not only does he distinguish predictions from forecasts in these chapters, but he makes clear the importance of confidence intervals in ways that I had not quite fathomed before. 

Finally, in his last set of chapters Nate starts driving his readers to leave behind deterministic styles of thinking—trading them in for probabilistic styles of thinking instead.  More specifically, he pushes us to think as Bayesians.  He provides a simple explanation for how Bayesian statistics operate and then illustrates how natural this style of thinking is with a chapter on poker.  Here again, we find that Nate has considerable experience having earned hundreds of thousands of dollars playing Texas Hold’em online.  And if you haven’t found a topic yet to interest you, there is another chapter on computers playing chess and one to be had on controversies over global warming. 

If you liked “Freakonomics,” you are going to love “The Signal and the Noise.”   The topical style is somewhat similar, but there is a key difference:  where Steven Levitt admits to having no overarching theme, Silver has a clear mission and makes no bones about it.  But where you might have imagined that Nate Silver is merely a stats geek with an interest in politics, we find that he moves around arenas of scientific inquiry and philosophy in a very sophisticated manner.  Let me put it this way:  I expected what might be analogous to a Colombo episode—with a math modeling detective investigating one subject in detail; instead, the book was more akin to a James Bond movie—jetting across all parts of the world in an effort to provide a message that might help us stay off the catastrophes of bad predictions. 

This is not to say that there are no negatives to the book.  Some chapters don’t seem to contribute as much to the overall theme as others and drag on a little more than is necessary.  But it is also written so that you can skip the arcane chess moves by Deep Blue and still find the overall book cohesive.  In addition, it is not crystal clear how we should apply a Bayesian style of statistics everywhere to provide for better forecasting.  In this respect, the James Bond nature of the book takes a toll:  its generality leaves us without clear steps for solving specific problems, and eventually the believability of some parts of the story gets tested (see Michael Mann’s take on Silver’s parsing of the global warming debate, http://www.huffingtonpost.com/michael-e-mann/nate-silver-climate-change_b_1909482.html).   

All that said, the overall message that we should remain faithful to the data is one well worth hearing from Silver.  Interestingly, it is a message that I failed to heed when I first posted on Facebook my own map predicting an Obama win.  You see, while I took 95% of my inspiration from Silver in drawing my own electoral map, mine was not precisely the same as Silver’s.  Silver’s map arguably put Florida in the W column for Obama—though only by a fraction of a percentage point.  As I was posting for an audience that included many conservatives, I tried to avoid appearing overconfident and figured that Silver had to miss at least one state.  This is exactly the style of thinking that throws predictions off, however.  Silver’s predictions beat mine because I made an adjustment given self-presentational concerns.

I not only highly recommend the book, but I’m considering making it required reading for my graduate students in psychology.  I hope you’ll give it a read, or at least pick a chapter for exploration.