Global Integrity is a non-partisan organization and does not endorse candidates. However, we can endorse the most innovative metrics and analysis tools of the 2008 campaign cycle, and that nod goes to Nate Silver of fivethirtyeight.com.
Telephone polling in the United States is in significant upheaval. First, mobile communications has reduced the accuracy of landline polling. Second, a major uptick in voter turnout is making a mess of ‘likely voter’ projection models. Throughout the primaries, polling was iffy, with the ‘surprise’ results amplified by journalists eager to transform the bureaucracy of a state-by-state primary process into a gripping narrative of sudden reversals and come-from-behind victories. On top of this, both side of a campaign have an interest in keeping perceptions muddy. Candidates are observed most closely by their own supporters, and the candidates are wary of saying anything that would cause their volunteers to believe the race isn’t very close and still worth fighting for.
The result has been that a great many people have a hard time sorting out how the campaigns are actually doing. Into the fray steps Nate Silver, resident of Chicago who was, until this campaign cycle, best known as a baseball analyst. Silver has a gift for teasing meaning out of quantitative data and pointing gift that towards the questions that matter. But he also blends the hard data with qualitative analysis that builds the credibility of his reporting — Silver explains what numbers mean, and what they don’t mean. He’s also willing to cover the stories that don’t generate numbers with old fashioned reporting — his coverage of the get out the vote ground game(with Sean Quinn and photographer Brett Marty) has been thoughtful and steady, while the premier pundits spent three days parsing the various meanings of “lipstick on a pig”.
Silver himself is quick to point out that he is rooting for the Democrats, but he also maintains that his numbers — aggregated from dozens of published polls and other metrics — have no party loyalties. His record bears this out — when John McCain surged in the polls following the GOP convention, Silver documented the rise and his projections tightened to match. While his projection model currently gives Barack Obama a 96.3 percent chance of winning, he is careful to discuss the McCain victory scenarios in some detail. He’s using a practice we employ in our own work: if you ask specific, verifiable questions, and base the answers on transparent, unpackable evidence, bias and slant don’t have much room to influence the results.
Meanwhile he has become the de facto high commission on polling methodology: if you do shoddy work, he’ll carefully explain why no one should trust your numbers. And given his high profile lately, that tends to get noticed by pollsters, who stake their livelihood on a perception of accuracy.
So what can would-be analysts in Nairobi or Brussels learn from Nate Silver and fiverthirtyeight.com? We’ll draw a lesson from our book, A Users’ Guide To Measuring Corruption, and list the best practices we identified for corruption metrics. They apply nicely here as well:
Know what you want to measure, and find the right tools to do it.
Build understanding incrementally, rather than look for a single, perfect assessment or poll.
Use unpackable data. If you’re working with polls, dig into the demographic breakouts.
Don’t exclude groups that are hard to sample, be they minorities, women, or people with cellphones.
Whenever possible, combine quantitative data with qualitative assessments.
Gravitate toward local assessments.
Embrace the need for multiple assessments. These tools complement each other.
Use metrics responsibly — explain what they mean, and what they don’t mean.
Transparency of methodology is essential. Take the time to explain why you make the decisions you do.
Fivethirtyeight.com has all of these covered, and given Silver’s newfound fame, it seems that good methodology is very much in demand. (You can read more about the best practices in A Users’ Guide here.) Congratulations to Mr. Silver on his success, and we hope the demand for this kind of empiricism in the political process is a sign of things to come.
— Jonathan Werve
