In our first blog we talked about how we started working with the area of the Ministry of Finance and Public Credit (SHCP) that runs the Performance Evaluation System for Results-Based Budgeting (PES-BRR), to find ways to incorporate learning from those who implement public programs in the system and turn it, little by little, into a more humane system. In this blog, we share with you our first learnings about how the system works, the different actors involved in it and its main challenges.

The subtleties in the definition of effectiveness matter

Effectiveness is the ability to achieve the desired result, to reach a certain goal. The first conversations we had with experts on the subject and our counterparts in the Performance Evaluation Unit of the Ministry of Finance and Public Credit (PEU-SHCP) led us to reflect on the importance of the subtleties with which the effectiveness of a program is defined. In other words, what constitutes a good program - is it its ability to do or is it its ability to learn and adapt? The concept is also related to other dimensions such as effectiveness (doing it all), efficiency (doing it economically), quality (doing it well) and sustainability (doing it and keeping it running).

When a program has been proven to be the best way to solve a challenge, the interest is in making it grow, reach more people, reach them faster, and at a lower cost. If, on the other hand, we are uncertain whether the program as it is designed is the best strategy to solve the problem given the current context, it is the ability to learn that matters most. What is missing, what went wrong, why, how can we adjust or rethink it?

For example, a scholarship program may be an acceptable solution to reduce dropout rates. If we believe that the program, as designed, is already the best possible solution, we effectively limit ourselves to evaluating its ability to deliver grants in an agile and cost-effective manner. But, if we recognize that there may be other alternatives to the design or that conditions have changed, then the definition of effectiveness must be broader. It should also include ways to assess the ability of the people operating it to learn to iterate, adapt the program, improve their processes, and even propose new alternatives.

The importance of finding the balance between accountability and learning.

With the above in mind, we set out to deepen our understanding of the PES, in particular, the tools designed and operated by the PEU-SHCP and its interaction with the Responsible Units (RU). Because we believe that these tools should serve two main purposes. First, programs should be held accountable for achieving their goals and objectives. But also, that there are mechanisms to rescue the learnings of those who face the challenges of its implementation on a daily basis, and that this serves for its continuous improvement.

The general perception of the people we consulted is that the tools and processes operated by the PEU-SHCP favor accountability over the ability to incorporate learning. Perhaps because of the way incentives operate throughout the monitoring processes that the PEU-SHCP requests from the RUs. It is something that will have to be explored and tested.

We also learned that there are informal channels where discussions are created around the challenges of implementation, and that these channels could be formalized and serve to gather lessons learned. One example is the face-to-face BRR training courses provided by the PEU-SHCP to the RUs. We were told that there, the civil servants generate discussions that reach the level of collective catharsis about the challenges they face when implementing their programs. We were strongly recommended to witness what happens in these spaces and so we did. We will share with you what we learned in a later installment of this blog series.

A first view of the actors that make up the system

The next step was to construct a diagram of the areas with whom the RUs interact on a day-to-day basis and the type of information they share with each, both inside and outside their unit. Recall that a RU is any administrative unit in charge of a budgetary program. 

This is the graphic representation of the areas with which the RUs coexist within their units and the issues they deal with each one. The central entity is the RU, on whom the implementation of the program depends, represented here by an orange cube. Its main task is to operate the program, whether it is the delivery of scholarships as in the example used above, the construction of road infrastructure or any other public good and/or service.

RUs must coordinate and negotiate with each of these areas to do their job and stay within the norm. The areas with which they discuss performance are the Planning, Monitoring and Evaluation (PM&E) units. These areas help the RUs to plan their activities, to define the objectives to be achieved and to translate everything into a logical framework that explains in detail how the actions are aligned with the objectives, priorities and strategies of the National Development Plan (NDP).

The second diagram explores the inter-unit communication channels for each of the different topics. The RUs and the areas within the unit remain in blue, but now the areas of other units with which each area must coordinate in each of the topics are shown in purple.

This exercise helped us to understand better that the people in charge of the RUs are immersed in a system of continuous coordination and exchange of information with other areas and units. A system where there is the potential for tensions to arise that may influence the final decision on how performance is understood and reported; and how those responsible for implementing programs interact with the PES-BRR. To find the desired balance between accountability and ability to learn, each piece of the puzzle is key.

Are you a civil servant or do you know how the coordination and exchange of information between these areas operates in reality? What incentives do you think work in favor or against the RUs when they have to report to the PEU-SHCP information on the performance of their programs? You can use this channel to share with us ideas on how to make the PES more human and improve its capacity to incorporate learning that enhances the programs. At the Accelerator Lab we are creating the world's largest and most agile learning network on sustainable development challenges.