The global lockdown that began in March 2020 inadvertently led to some giant strides in innovation. Constrained from our usual ways of working but compelled to keep going, the Independent Evaluation Office (IEO) of UNDP needed a solution. The search for a way forward triggered a series of novel approaches, culminating in AIDA.

AIDA - or Artificial Intelligence for Development Analytics – is a powerful set of machine learning and artificial intelligence (AI) algorithms, creating a tool at the cutting edge of technology for development.

Inception – necessity breeds innovation

To overcome the constraints of the new normal, evaluators got creative. At the IEO, new approaches included accessing big data such as georeferenced information systems, using satellite imagery, and conducting interviews and group discussions by phone or on online platforms.

Evaluators also sought new approaches to extracting and analyzing existing information. The IEO has a digital repository called the Evaluation Resource Centre (ERC), that contains more than 6,000 evaluation reports spanning two decades of development work, but the question remained: how do we use it better?

The IEO began conducting rapid evidence assessments on topics relevant for the response to and recovery from the pandemic, including support to health systems, social protection, livelihood restoration and job creation. Using the ERC, the IEO produced a series of papers, named Reflections, on how UNDP responded to previous crises, including Ebola and the Indian Ocean tsunami. This was an important first step, but still the process was manual and time-consuming, and the results incomplete. Traditional search methods didn’t so much dig into the unstructured data as scratch on the surface.

Eager to meet the challenge head on, the IEO took a leap into the future: artificial intelligence. AI held the key to overcoming the obstacles we faced, as a faster and smarter way of processing large amounts of data from the evaluation archives.

Beginnings

AI solutions can perform tasks that normally require human intelligence. Like a human, they need to be taught how to perform the task at hand. In the first phase of the AIDA journey, the IEO developed multiple algorithms that can process, understand and search the vast stores of reports.

What is truly groundbreaking about AIDA is its capacity to intelligently search and make sense of unstructured data from over 6,000 evaluation reports. AIDA can analyze this wealth of information right down to the paragraph level, pulling out precisely what is needed from vast swathes of information in seconds.

AIDA is accessed through a web portal, behind which lies a complex architecture. A critical factor in this architecture is the interaction between humans and machines, which is how AIDA learns. When a user types in a query, that query becomes the point of meaning around which the expanse of unstructured data in the IEO’s archives crystalizes. The scope of the evidence base in our archives expands simply by using AIDA to search within them.

Initially, our talented team of (human) taggers coded a sample of 500 reports to train the algorithm, which rapidly progressed to incorporate all the reports in the archives. The human team checked and corrected AIDA’s outputs, and the feedback supported the machine’s learning. The feedback loop is a perpetual process, and continues to support AIDA’s supervised machine learning capabilities every time someone uses the tool and provides feedback by clicking thumbs up or down to each result.

What difference does AIDA make?

AIDA has the potential to improve the way we design, implement, and evaluate development programmes. Before AIDA, evaluators would search the ERC by title, like a library catalogue. This was slow, and inevitably resulted in important nuggets of information being overlooked in an archive as vast as the ERC. Where a traditional search skims the surface, AIDA dives in. Moreover, AIDA understands that it is searching within a context: the machine doesn’t look for particular words, but rather for the context in which those words have meaning. Using AIDA, results are not just faster, they’re more relevant.

The results of a search in AIDA - for example, the relationship between climate change and human health - can be filtered and sorted by country, region, year or connection to the Sustainable Development Goals, and exported to a spreadsheet for further analysis.

Because the ERC contains evaluation reports, the information stored in them has already been validated because of the way an evaluation report is researched and compiled. It is a consultative process involving a range of stakeholders and resources, which means the quality of the source information AIDA mines is better triangulated than an average self-produced project report.

AI cannot replace the creativity, understanding and divergent thought processes that humans possess, but it can enhance users’ analytical skills to make a more significant contribution to achieving development objectives, up to and including the Sustainable Development Goals. In the next phase, AIDA will go beyond human capacity in its ability to generate new knowledge based on evaluative evidence in combination with external sources of information as development indicators.

AIDA was developed by the IEO with the support of UNDP’s Information and Technology Management team, the United Nations International Computing Centre and Amazon Web Services.