This is the third and last blog in a series about a text mining experiment that we conducted at the UNDP Accelerator Lab in Mexico to collectively assimilate the main learnings of the people who are in the front line of the implementation of public programs. The first can be found here and the second here. In this post, we share how to put into practice the principles of collective intelligence to refine this exercise over time and to get to know the potential it has to become an inexhaustible source of learning for continuous improvement.

This is just a starting point  

Collective intelligence is the expanded capacity for learning, decision-making, and problem-solving that societies can achieve when they use technology appropriately. It occurs when it is possible to create a technological architecture that helps society to assimilate more perspectives and use them to make decisions in a more participatory and plural way, in favor of a common good.

The proposed exercise helps us have a collective reflection on the causes behind some programs not reaching their goals. It is collective because it is informed by the thousands of text records written by people who work on the front lines of the implementation of budget programs. The foregoing, even though each one refers strictly to the program that they must operate.

The starting point is the nine causes contained in the GUIDE developed by the Performance Evaluation Unit. However, they should not be understood as a final list of all the possible causes that explain why the budget programs do not reach the proposed goals. They are just a starting point so that the algorithm has something to compare the texts with and we have a way to group the information in a way that makes sense to the collective.

In fact, after analyzing the results of this first exercise in detail, conclusions can be drawn about the limitations of this first list, as well as ideas for its refinement. The invitation is that cycle after cycle, this list can be refined to make the exercise more and more precise and relevant. This recursive exercise is the central purpose of creating a collective intelligence that allows an orderly dialogue between thousands of people and from which increasingly precise and valuable conclusions can be drawn.

How to refine the exercise for the next cycle?

Let's start with the cause present in 60 and 80 percent of the texts: cause #6 “Non-compliance of vendors and contractors, as well as opposition from social groups”. This, for example, could be mixing two different phenomena. On the one hand, there is opposition from social groups, which could be a reason why some contractors cannot fulfill their commitments, especially if they face stoppages and blockages, but it is not the only conceivable reason why a contractor fails to comply. For example, it could be due to the type of contractors chosen.

Given the high frequency with which this cause is collected by the algorithm, it is worth putting on the table within the Performance Evaluation System if it should be divided into two. This would help to perform a more detailed analysis. On the one hand, to distinguish when non-compliance is due to opposition from social groups and, on the other, when it is related to the type of contractors chosen. Since now, as it is written, it does not allow to discriminate between one or another situation, and the actions aimed at solving each of these two situations are different.

On the other side of the spectrum, the analysis allows us to see that there are causes whose prevalence is so low that it is worth discussing whether it is worth including them in the GUIDE issued by the Performance Evaluation Unit to all dependencies. For example, cause #2 “Emergencies caused by accidents and/or adverse natural phenomena” is consistently present in approximately 4 percent of the texts throughout the analysis period.

It is also worth reflecting on whether this or other causes for which the algorithm collects little evidence of their mention in the database should have a space in the list, without this implying that they should be eliminated only because of their infrequent occurrence. Without question, its monitoring can be strategic, since given the risk profile of natural disasters that exists in the country, this low percentage of mentions could be an indicator of the resilience of public programs. Although to validate this type of conclusion, more evidence is needed in addition to what is presented here.

Where to find new causes?

Finally, it is worth discussing those factors that might explain why things sometimes go wrong but are not contained in the list of causes that feed the algorithm to find matches. The analysis is blind to these factors. If a cause is not on the list, then it is not possible to perform any type of analysis on it.

As we said before, it is only a first starting point. To find what people are talking about but not included in the original list, there is a sample of crucial text records. These are records that, according to the algorithm, do not find similarities with any of the nine predefined causes. Therefore, using the different filters offered by the database, we identified a sample of 2,777 justification texts whose similarity indexes did not reach a value high enough to be classified as "similar" to any of the previously defined causes.

After sorting them using the complexity index, we quickly start to see patterns. We not only found new categories of causes but also evidence of failures to understand the platform's filling instructions and even instances where users respond to justifications with incoherent and meaningless terms. Among the elements that we found to create new categories, elements such as occasions in which people mention that they do not have enough information to measure progress, or external macroeconomic and political factors that affect the fulfillment of their goals stand out.

Do you have experience in text mining, natural language programming, artificial intelligence, or collective intelligence process design? Do you know the Performance Evaluation System or do you want to share with us ideas on how to innovate in this field? Contact us, we want to hear from you!