Locations7. Measure What Matters
This standard is linked to the following Principles for Digital Development: Be Data-Driven.
Every project needs a clear, upfront definition of success and failure. This means attaching numbers to measurable outcomes. You can then implement changes to move a project toward success.
The ideal scenario is that these numbers are automatically gathered from your product with an analytics platform.
Examples are how many daily/weekly/monthly active users you have and how many ‘outcomes’ are achieved where value is created for the person using the product. Think about the cost-per-outcome. For a telemedicine product, you would want to know the cost per virtual consultation. This is the number of virtual consultations divided by the total costs of launching and maintaining the platform.
Stay away from vanity metrics. These metrics make the project look good but do not shape future strategies. Pageviews and downloads are examples of vanity metrics — you can measure and track them, but they are not success metrics.
Think of the steps required to give users value from the platform. Track the number of people that reach each step and how many drop off vs. how many continue to the next step. You can use these metrics to fix usability issues in your product.
Measuring product usage allows you to correct bias and blind spots. There is often a big difference between what users will say to you vs. what they do on the product.
Measuring the performance of a digital solution is an integral part of improving it. Measuring continuous improvement requires a mindset shift. Rather than only thinking about long-term programme outcomes, you will need to gather short-term, immediate insights using analytics and qualitative insights. These insights will guide your decision-making. This can also help when working with and managing vendors and implementers.
Technology solutions are intended to drive development outcomes. Keep your programme outcomes front of mind as you contemplate, guide the development of, and implement digital solutions.
Do:
- Track the number of users, their gender, and level of engagement, whenever possible
- Install analytics (i.e., Google Analytics)If you have a web or mobile component,
- Feed data into a PowerBI Dashboard automatically
- Collect data on product crashes/outages (i.e. Crashanalytics)
- Define your success metrics before launch
- Check your metrics and course correct your product/service regularly
- Use data to make informed design and business decisions
- Establish feedback loops using qualitative insights developed through ongoing testing and user feedback, as well as web and programme analytics
- Track metrics from disaggregated data (while respecting privacy) to understand where the digital divide might be showing up (i.e, the product works well for male users but no female users are completing the journey)
Don't:
- Build your own data/metrics platform from scratch
- Leave monitoring and evaluation until the end of the development cycle, instead think about it from the beginning
- Only measure vanity metrics or tech performance, but focus on impact as well
Tools
- UNDP Canvas #7 Measure What Matters - Canvas
- SIMLab’s Framework for Monitoring and Evaluating Inclusive Technologies in Social Change Projects, SIMLab.
UN Resources
- UNDP Evaluation Guidelines
- Using Data Visualisation
- Data Futures Platform
- A World That Counts: Mobilizing the Data Revolution for Sustainable Development: “Basic Data Principles,” United Nations.
To Read
- Top 10 Product Metrics to Track
- Top 10 Product Analytics Tools
- Data driven design in Agile development: A thorough introduction to data-driven design that shows you how to work with data at different stages of development
- Using Data to Improve Your Service: Read the UK Government guide to using data to improve your service
Case Studies
- TCV: An Innovation Measurement Tool to Improve Public Service Delivery Bangladesh's civil servants and development practitioners had difficulty measuring innovation in the public service and could not justify or measure the effectiveness of simplified, digitalized pilot innovations for citizens. See how they turned to time, cost, and number of visits (TCV) to measure the impact from the citizen’s point of view.