March 22, 2018
Jorge Florez, Research Manager – Global Integrity
Katherine Wikrent, Data Scientist – Open Contracting Partnership
Carey Kluttz, Senior Program Manager – Open Contracting Partnership
Chelsey Lepage, Program Manager – Reboot
New York City is at the forefront of open data at the city level in the US. The city has an open data law, open data portal, an open data team housed in the Mayor’s Office of Data Analytics, and many interesting initiatives to use data to inform local public policy. Two areas with the potential for progress in opening up data are spending and contracts, which is awesome given that NYC spends $14 billion dollars per year through contracts, almost 20% of annual city budget! The Mayor’s Office is committed to ensuring that 30% of the value of these contracts is awarded to Minority and Women-Owned Businesses (M/WBE) by 2021, and $16 billion to M/WBEs by 2025, in a move that aims to improve contracting processes while also contributing to social inclusion and city development.
Given the sheer amount of data and the political commitment towards greater inclusion of M/WBEs, Global Integrity, Open Contracting Partnership and Reboot challenged New Yorkers – government officials, advocates, M/WBE and the general public – to use open data to inform policy and support M/WBEs in taking advantage of this opportunity. To this end, we led an Open Contracting Treasure Hunt during NYC Open Data Week (yes, not just a day!) to spur participants to use open data to better understand the challenges and take advantage of opportunities around greater participation by M/WBEs in city procurement.
The task was daunting! Not only are there many datasets and sources of information, but they are also scattered across different websites from government agencies – a shout out is due for the impressive work of the Mayor’s Offices of Data Analytics and Contract Services, the Department of Small Business Services, as well as Office of the Comptroller! – and designed to meet different goals such as transparency and oversight. There are still data gaps that inhibit certain analyses and difficulties in linking up data sets, which meant participants needed to have cleaned up data presented to them to enable them to make the most of it. Another important part of the context is the diverse activities led by the city to promote the inclusion of M/WBEs, including publishing procurement plans targeting M/WBEs; setting goals for M/WBE participation in procurement by agency and overseeing its implementation in different ways; and conducting capacity building and matchmaking activities.
Treasure hunters digging through the data to find insights and turn them into opportunities for M/WBEs in city procurement
Within this context, our fearless treasure hunters – comprised of data scientists, government representatives, students and civil society activists and advocates with differing levels of familiarity with open data and contracting process – formed teams to plunge into the ocean of NYC open data, aiming to develop practical ideas to inform the work of government and M/WBE. After a short demo of some of the available open contracting data on upcoming procurement opportunities, we guided participants through a data challenge in which they used their skills to either help M/WBEs identify solicitation requests or to inform M/WBE efforts in preparing bids and finding the most promising avenues to grow their businesses.
Our fearless leaders leading the data hunt into how to provide guidance to M/WBEs about preparing procurement bids
What did participants find and what ideas came about?
Participants began the event eager, but concerned about the information they might be able to find and actually use. Once the hunt moved forward they were surprised by the amount of data available, how well it was presented, and the extent to which this data enabled them to get a full picture of city procurement with M/WBEs. The data available covered all: the M/WBEs registered with the city, provided by the NYC Open Data Portal; plans for procuring from M/WBEs by agency, taken from the NYC Mayor’s Office of Contracting Services page; bidding opportunities (with the public and with the private sector!), also provided by the NYC Open Data Portal; and Checkbook NYC data on contracts awarded by agency, amount and vendor.
Once they made sense of the data, ideas on how to use it began to flow! From comparing procurement plans to contracts actually awarded, to exploring bidding opportunities and how these could be reviewed more effectively by M/WBEs, to analyzing what types of opportunities (prime vendor or subvendor) were more likely to be awarded to M/WBEs. Groups were able to move forward with their analyses and uncover interesting insights for government agencies and M/WBEs. For instance, one group placed themselves in the position of an M/WBE selling dairy products and was able to not only see the bidding opportunities that were available in general but to identify a particular subvendor opportunity that might be better suited for their experience and offerings.
Participants also identified challenges related to the potential to use the available data to address the challenges they were working on. During the final discussion and reflection they highlighted that the fact that datasets were designed with different use cases, or goals and purposes, in mind (i.e. to comply with regulations, to make spending transparent, or to enable procurement agencies to find potential vendors) which meant the potential for linking up the data was limited. Three issues dominated the conversation:
- the potential for getting a more robust and actionable picture of procurement with M/WBEs in the city is hindered by the lack of unique identifiers for contracting processes and unique vendors IDs;
- the use of different categories for classifying the contracts by industry or service limited the current potential to use the data to conduct analyses that could more effectively inform policy making, project implementation, and the very efforts of M/WBEs to be more strategic in developing business plans and finding procurement opportunities; and
- the progress made by NYC in opening up information is undeniable, but a greater focus on data governance and interoperability needs to be placed high on the open data agenda in the city.
Unlocking the potential of Open Contracting in NYC
At this event, our goal was to provide participants with a greater appreciation of open contracting data availability and use those data to address real-life problems in NYC. It gave them a quick snapshot of the problem and enabled them to get a better understanding of the challenge while identifying important avenues for dialogue and action to unlock the potential of procurement data. We are convinced that extending NYC open data leadership to cover open contracting will be a quick win for NYC, and provide New Yorkers with a valuable means to improve their lives. We are looking forward to – and stand ready to support – next steps!
