For the last three days, I’ve been in Amman, Jordan, getting schooled in Arab democracy from an international team of researchers from the Arab Reform Initiative (ARI). Technically, I was supposed to be there helping them design research methodology, but it looks like they’re doing just fine on their own.
Monday morning found me in a conference room full of Arab governance researchers, while I was trying to ignore the fact that my interal clock thought it was 2am. I didn’t present anything until the next day, and I was contemplating gradually reducing the volume on my translation headset and ignoring the room altogether. But it turns out the work ARI is doing is really interesting. Foiled!
Building a Better Barometer
The headline was the Arab Democracy Index (download PDF), an international research effort designed by a group of Palestinian academics – notably, Khalil Shikaki and Waleed Ladeweh of the Palestinian Center for Policy Survey and Research. The Index tracks the transition of Arab nations toward democracy and draws data from researchers across the Middle East.
Their Index does a lot of things right. Dr. Khalil makes a real effort to nail down the conceptual boundaries of what they are doing. They do their own polling rather than slapping together an aggregation of whatever’s available. They have a list of 40 indicators that are presented in an unpacked fashion — each indicator can be examined independently of the whole. The research teams are local and deeply knowledgeable.
That’s the good news. There are areas for improvement that are being debated actively, and I won’t rehash them all here — the workshop was built around making improvements to the Index, and the real work will happen after I’ve left. My suggestions to them were:
1) Focus on what matters. Rankings don’t matter in and of themselves; policy reform matters. If an index isn’t doing something useful for changemakers trying to impact policy, it’s generally a waste of time.
2) Simplify. If there’s debate over whether a given concept fits in the basket of things you’re seeking to assess, it’s probably just draining resources.
3) Create deep information. In Global Integrity’s experience, it’s not how many questions you ask (the width) that creates impact. Impact instead comes from the depth of the information being discussed by policy makers, NGOs, and the media.
Numbers are Summary
In social science, numbers are a summary of a set of observations. If your social-issue index only cranks out numerical scores, you might only be looking at the tip of the information iceberg. Better to get below the waterline with narrative commentary, references, details of specific cases, historical context, researcher disputes, and complexity.
Here’s an example: I could tell you that journalists in the United States are safe investigating corruption 99.47% of the time (I invented this number).
Or I could tell you that in 2007 a veteran journalist, Chauncey Bailey, was shot in the chest with a shotgun while walking to his Oakland office; that his killer was hired by a company Bailey was investigating for fraud; and that the shooter told police he was “a good soldier” (sadly, this is true).
Which one will you remember tomorrow? Which one seems like something worth caring about? Deep information drives policy change.
Getting to the Point
I was discussing this idea with the executive director of the Arab Reform Initiative, Bassma Kodmani. She said to me (paraphrasing greatly here): You Americans, you always get right to the point of something. Our people here, they’re political scientists. They like to argue about the details.
I’m not sure if it’s an American thing, or a social-entrepreneurship thing, but lately, I’ve been fixated on measurable impact. Maybe it’s a seven-year itch — it’s been that long since Nathaniel, Marianne, and I worked on the first Global Integrity proof-of-concept. Or maybe it’s seeing the Arab Democracy Index from the outside with a journalist’s skepticism. Maybe it’s that I keep using the same examples when people ask me about our impact.
The bottom line is it’s really easy to focus on the tools — our gorgeous, award-winning social science tools — rather than the messy, awkward societies we’re trying to improve together.
Global Integrity is building new projects, and we’re thinking big. I want to make sure we’re not just doing work that is destined for a dusty bottom drawer. Three days in the Arab world talking politely about largely non-existent democracy and I’m suddenly fiendishly impatient; I want to be hitting hard.
— Jonathan Eyler-Werve
