Amendment to initial January 1, 2016 policy, taken on February 9, 2017 | March 1, 2017 effective date
Introduction
Global Integrity (GI) research data, including the Global Integrity Report, Local Integrity Initiatives, the State Integrity Investigation, the Campaign Finance Indicators, and the Africa Integrity Indicators, have been published and available for download, and remain accessible to interested users. GI data is produced by expert researchers, contracted for their country level knowledge and expertise, and produced following GI’s published methodology, including a rigorous peer review, as well as internal quality control and consistency reviews. These procedures ensure that the data is both accurate and detailed when published.
Global Integrity stands behind the methodology and resultant data. Expert researchers are trained and provided with detailed coding instructions and research standards. GI staff rigorously assess draft research in light of these, multiple peer reviewers assess the draft research for accuracy and adherence to standards, and GI staff review the final work again to ensure a consistent application of standards and scoring criteria.
However, published data is occasionally challenged as being inaccurate or needing improvement. Internally, GI staff has occasionally identified issues with published data. In rare cases, certain data issues have been highlighted by outside data users. Until October 2015, suggested improvements were assessed on a case by case basis, in very few instances calling for comment, source, and / or scoring alterations. On January 1, 2016 a new policy was set forth that called for no retroactive adjustment of indicators, and the imposition of a two month comment window. This amendment updates this policy, allowing for the retroactive change of data only where substantive, proven factual discrepancies have come to the fore.
Policy summary
GI will continue to focus the bulk of its resources on current and future research rather than investigating potential changes to published data. Therefore, upon publishing any new data, a single two month comment window will be made available for data users to provide feedback and potential corrective information, after which time the published data will remain as is. Within the comment window, where data users provide corrective information that meets the research standards, this will be added to the published research.
Both during the comment window, as well as during the consistency check of each year’s data (in March of each round), prior round data may be changed only where there are substantive, fact-based problems uncovered based on newly provided information.
The goal will be to transparently share the new information provided, while enabling organizational and researcher resources to focus on producing current and future rounds of this important research rather than re-investigating prior rounds.
Prior-round changes
The original policy allowed current round changes to be made by Global Integrity managers on the basis of new evidence uncovered during either their consistency checking of the submitted draft research, or during the two month comment period. However, it did not allow for changes to previous rounds of data. This had led to problems where there was substantive, factual evidence that some prior scores were off based on evidence being missed, or misinterpreted, by the research and reviewer. This amendment now allows for changes in the scoring and comments of prior rounds of research only where there is fact-based, concrete evidence that the score and comment are substantively inaccurate. Such indicators that meet this test will be amended during the consistency and comment period, with published data being updated one month after the closing of the comment period each year. A separate list of all such changes will be kept, and shared upon request.
Scores may be considered for change only if the difference between the published score and the recommended score (based on the new evidence uncovered, or errors in interpreting clear data in former rounds) is 50 or more. Lower differentials, to be sure to respect the initial researcher and reviewer’s interpretation of the facts are kept, will not be amenable to such changes, unless these indicators are the few that have hard divides for all five scoring levels (i.e. the indicators based on % female representation, which have clear cutoff levels for each of the five scoring options).
Exception to changes of less than 50, will be allowed in case there was an error by GI managers and a score originally assigned by the researcher was changed by the managers without justification.
Comment period
Upon the completion of a research round, when GI staff have completed their final standards and consistency checks, the research will be offered for two months in a provisionally final form. GI would publish provisional data, allowing potential users and interested parties to review the provisional data, and will welcome additional evidence or sources should they be necessary. GI will then take a third month to assess the provided material in relation to research standards and scoring criteria, and work with the researcher should any adjustments be deemed necessary. At the end of this three month period, GI will adjust all necessary data in the recently published research, updating it accordingly. This will then be considered the final data and scoring for the research round.
Post-comment period
Once the comment period is complete, no scoring, comments or sources will be adjusted. However, should further areas for improvement be identified, GI will keep detailed notes covering the new information identified along with brief explanatory detail, and this information will be used during the next consistency and comment period for potential further updates. This list may be made available upon request to interested parties.
Tracking
All externally recommended alterations will be tracked by GI in the relevant project database, noting the source and feedback provided, along with an explanation of whether the alteration was deemed necessary, including the rationale. This database will also reflect when the published data was altered (or notated in the post comment period).
Transparency
A list of all changes to already published data (prior round data) will be kept, and shared upon request.