In this blog we share with you about the text mining experiment we conducted at the UNDP Acceleration Lab in Mexico (PNUD in Spanish). The exercise combines Performance Assessment System outcome indicator monitoring, data science, Natural Language Programming and the principles of collective intelligence to collectively assimilate key learnings from people on the front lines of public program implementation.

Where do the texts we mine come from?

How do we know if what the government is doing is being done right? To begin to answer this question, it is important to have transparency and accountability mechanisms such as those of the Mexican government. If the goals set by government programs establish a clear causal relationship between the activities carried out by the program and its purpose, progress is monitored regularly, information is available to anyone and there is continuous reflection on what is preventing the achievement of goals, then we are on the path to continuous improvement.

One of the elements of Results-Based Budgeting involves just that. The persons in charge of operating a budgetary program define specific performance objectives and targets and capture them in matrices and results indicator sheets as a result of the application of the Logical Framework Methodology. They establish a clear causal relationship between the activities they carry out and their purpose, as well as their contribution to a higher purpose that the policy seeks to achieve, and they are the basis for the follow-up and monitoring of program performance in the Performance Evaluation System. All of this generates data that is freely accessible on the Mexican Government's transparency portal.

But there's more. In addition to monitoring the progress of indicators and compliance with goals, people also upload a thought for each indicator into the system once a year. In a brief paragraph, the persons responsible must describe, to the best of their knowledge, the main causes for the difference between the indicator's progress and the target. These are the records that we decided to mine as they contain valuable information about the learning that those on the front line of program implementation are reaching and the challenges they face in implementation. This is information that, due to the difficulty involved in its aggregation and systematization, is difficult to use to identify patterns beyond the individual level of each program.

What are the causes that explain good or bad performance?

This experiment is the first exercise to make sense of and find patterns in the aggregate about what people mention in the texts justifying the differences between progress and goals. What people write has the potential to shed some light on the factors behind the failure of some budget programs to achieve their goals. All the information used to conduct this exercise is contained in the Performance Assessment System's indicator progress database from 2013 to 2019 and contains records of the progress of 97,515 indicators over this period. The justification texts are the result of the work of thousands of people who work in the public service and make up the country's most important repository of knowledge and applied learning on what it means to operate public policies.

The Ministry of Finance and Public Credit sends each year to the agencies a GUIDE FOR REPORTING FINAL PROGRESS AGAINST COMMITTED TARGETS IN PERFORMANCE INDICATORS. The guide describes nine causes predefined by the Performance Assessment Unit that are the starting point for this experiment. Eight specific and a ninth cause that includes situations that by their nature cannot be added.

1. Deficient original programming.
2. Emergencies caused by accidents and/or adverse natural phenomena and/or health risks.
3. Lower demand for goods and services.
4. Delays in the procedures for the budgetary exercise by the Responsible Unit.
5. Non-compliance or delay in budgetary procedures by government agencies other than the RU.
6. Non-compliance or non-conformities of suppliers and contractors, as well as opposition from social groups.
7. Modification of institutional attributions due to regulatory provisions.
8. Non-compliance due to extra-budgetary regulatory situations unrelated to the RU of the goal.
9. Other causes that, due to their nature, cannot be added.

Two text mining algorithms were designed to translate the textual information and to be able to exploit it quantitatively. The first is an algorithm capable of sorting texts according to their linguistic complexity in order to distinguish between those that are easy to read and those that are more sophisticated. The second is an algorithm capable of evaluating the similarity of each of the texts with each of the nine causes predefined by the Performance Assessment Unit.

How to build a collective intelligence?

Collective intelligence is the expanded capacity for learning, decision making and problem solving that societies can achieve when they make appropriate use of technology. It occurs when we manage to create a technological architecture that helps society to assimilate more perspectives and use them to make decisions in a more participatory and pluralistic way in favor of the common good.

Thanks to the analysis proposed by this experiment, it is possible to extract aggregate trends on the reasons why some budget programs do not achieve their goals. All based on the individual learnings of the people who are at the front line of the implementation of the country's policies and without asking people to perform any additional tasks to the monitoring tasks they already perform within the framework of the Performance Assessment System.

This helps to shape the architecture of the Performance Assessment System as a collective intelligence in which the learnings of individual people are aggregated to create something that is more than the sum of its parts. This is the basic premise under which the Acceleration Labs propose the creation of collective intelligence systems in their report https://smartertogether.earth/.

In the next blog, we will tell you the main results of a first analysis exercise of the data obtained in the experiment.

Do you have experience in data science and text mining, are you familiar with the Mexican Government's Performance Assessment System, or would you like to share with us ideas on how technology can be incorporated into the design of improvements in the Results Based Budgeting? Write us, we want to hear from you!