There’s been plenty of chatter in recent months about a possible post-2015 Millennium Development Goal on governance following the release of the High-Level Panel of Eminent Persons on the Post-2015 Development Agenda. The report called out “good governance and effective institutions” as one of 12 “illustrative” goals that nations might consider when adopting new development targets for the post-2015 era. This is potentially big news for transparency and accountability boosters; enshrining good governance in the post-2015 MDG process could provide significant political cover for continued reform efforts while simultaneously unlocking new resources for the work.
A chief risk to the prospect of a good governance goal is the data problem. How would we measure countries’ progress towards “good governance” in a post-2015 framework? Whose data should be used? Is it accurate and robust enough for such an august task? Is it possible to generate quality data on a global scale, regularly, that could be used for tracking?
These questions are generating anxiety in many quarters. But there’s a good answer at hand. Rather than expend time and resources trying to invent “one governance dataset to rule them all,” we should look to better coordinate and ramp up existing data gathering efforts that together, as a mosaic of information, could help to tell us whether countries are making progress towards better (or good enough) governance. That’s our best shot at answering the post-2015 good governance skeptics.
An Improved, If Muddled, Landscape
When Global Integrity first got into the business of measuring governance (specifically, anti-corruption and integrity systems) in 1999, it was a lonely landscape. You had the Transparency International Corruption Perceptions Index (CPI), some Freedom House rankings, eventually a more refined CPI in the form of the Worldwide Governance Indicators, and a handful of one-off efforts from various scholars and academics. The explosion of new governance data at the turn of the millennium — anti-corruption data from Global Integrity data and TI’s National Integrity Systems studies, fiscal and budget transparency data from the likes of the International Budget partnership and the PEFA assessments, the Doing Business surveys, and sector-focused data such as the Resource Governance Index and Rule of Law Index — changed the scene dramatically. While none of those tools were prefect, they all advanced the state of the art and moved us towards a place where it was possible to generate solid, accurate primary data on governance and transparency. We managed to move beyond pure perceptions or composite data that often relied on poorly constructed business firm surveys as their source material.
Despite that progress, real challenges remain. All of the above tools suffer from one methodological weakness or another (bias in question selection, poorly anchored score choices, lack of inter-coder reliability, etc.). Country coverage remains sporadic, random, and tends to focus on the same 75-100 “usual suspect” countries while ignoring half of the planet. Most NGO data producers favor covering developing and middle-income countries over the OECD countries for mandate reasons. As a community of data producers our pace of research remains too slow, and while the requisite level of effort to generate these data has diminished somewhat, the work remains hard and too expensive.
My Precious (Data)
In response to those challenges, a somewhat stillborn effort began in the late 2000s to explore developing even more ambitious governance indicators that were more universal in their coverage, solved the methodological challenges inherent in the “second generation” datasets, and could be replicated over time. The hope for a “one governance data set to rule them all” became a serious pursuit involving leading scholars and practitioners (as well as aid donors) and gravitated quickly towards “benchmarks” as a silver bullet solution. The idea was that if one could benchmark the expected or ideal performance specific to particular public sector entities — for example, how long it should really take to receive a passport from office X or a construction permit from ministry Y — one could then gather real-life performance data against those benchmarks, the deltas for which would tell us where corruption (and/or inefficiencies) were cropping up. It had a certain empirical sex appeal to it (“finally, we can put a % sign next to corruption levels!”).
In practice, however, this proved exceptionally difficult, incredibly costly, and very time intensive. What we had hoped was a silver bullet turned out to be more of a unicorn; the benchmarks exercises were too balky and costly to be feasible in practice beyond small pilots.
(For those interested in reviewing this history, I highly recommend this seminal paper from Prof. Michael Johnston (Colgate University) for the OECD circa 2009. The aspirations and assumptions of that era seem remarkably off with the benefit of hindsight. For example: “A further advantage of the [Data-and-Benchmarks] strategy is that it is not particularly resource-intensive, and that such resources and effort as it does require will themselves help improve governance and integrity. Gathering data is more a matter of leadership and co-operation than of money; after all, the strategy rests on assembling and disclosing information about activities that an agency is supposed to be doing already.” If only.)
A Better Approach: Coordination and Dashboards
As we look ahead to the post-2015 MDG era, it’s easy to despair that we’ll never have good enough data with which to track country progress on a good governance goal. If empirical benchmarks are too hard, what’s next in the short amount of time remaining?
I strongly believe that a feasible, useful, and cost-effective approach would be to focus on a) scaling up and better coordinating the high-quality second-generation data while b) simultaneously exploring ways in which governments themselves could contribute through self-assessments, existing household surveys, and national statistics offices. The end result could be a sort of “dashboard” displaying the results of a few dozen key governance data sets that when analyzed together could tell a compelling story of whether governance was improving or backsliding in the country.
In practice, this could look like the following. Imagine country X in 2016 when we finally have regular, predictable data being generated from leading third-party data providers which have an established reputation for impartial, solid research. You’d see current Global Integrity and TI data assessing anti-corruption and integrity systems; IBP and PEFA data rating budget and fiscal transparency; Revenue Watch data if country X has significant extractives resources; some good civil liberties and internet freedom assessments from the likes of Freedom House and the Web Foundation; freedom of information ratings from the Carter Center, AccessInfo Europe, and the Center for Law and Democracy; and rule of law data from the World Justice Project. Sitting alongside those rankings and data are complementary data from the government itself: perhaps an MDG-specific governance self-assessment rating, a UN Convention Against Corruption self-assessment, and/or household and business firm surveys speaking to governance concerns (bribery, trust in government) carried out by domestic statistical and research agencies.
This quilt of information should be able to tell us, generally, whether governance is improving or not in country X from year to year. If in Year 2 country X backslides on 12 out of the 14 data sources, that’s pretty clearly not good news. Alternatively, if in Year 3 it improves on 9 out of 14 despite stagnation on 5, one can make a reasonable assumption that, overall and at the national level, things are improving.
Unlike some other post-2015 MDGs, the actual good governance “goal” won’t be as empirically precise as a particular dollar amount per day of income or maternal mortality rates (and as Dani Kaufman rightly points out, let’s not pretend that those more empirical metrics are perfectly precise either). But it could be quite actionable (and action-worthy) to define a good governance goal as striving for improvement on some or all of the dashboard indicators and/or improvements against a trailing global (or regional or income cohort) average. One could get cute and through some simple normalization techniques combine scores into an overall “super” governance score for countries. I could see value in that as long as the underlying scores remained entirely transparent and we could tell the specific story, for each country, of why its “super” score improved or declined year to year.
While this dashboard approach doesn’t solve every potential problem, it has many advantages. These data already exist and are already being generated, but in an ad hoc way. We simply need need to invest in more regular coverage while at the same time de-duplicating datasets where multiple organizations are either asking the same questions or covering the same countries. Second, complementing best-of-breed third-party governance data with government data and/or government self-assessments helps to mitigate the typical government aversion to embracing the good governance agenda and being “judged” by outsiders. Third, this is not terribly expensive, nor is it risky. The sunk costs of developing these datasets have already been borne by others during the past decade; what’s called for here is a simple scaling up of what already works. With established technology platforms like Indaba and FrontlineSMS and newer ones such as Ulula, it’s easer, cheaper, and faster than ever to carry out large-scale data gathering projects of this nature without cutting methodological corners.
At Global Integrity we’re taking some quiet first steps to explore all of this through the development of an idea we’re calling, for now, the Governance Data Alliance. The Alliance has broader aims beyond the post-2015 discussion but is focused on improving coordination between third-party governance data providers. We think there are simple, inexpensive ways of scaling up coverage, de-duplicating efforts, reducing costs, encouraging best practices between data providers, and identifying gaps where new data and indicators should be developed. The original concept note is here, and if you’re interested in joining the ongoing brainstorming discussions please send me a note.
Chime in below in the comments or on Twitter (@integrilicious and @GlobalIntegrity) with thoughts, critiques, and alternatives for how we might be able to measure a post-2015 MDG on good governance.
