Lessons Learned in Indicator Standardization
March 6, 2023
By Janine Barden-O’Fallon and Bridgit Adamou, Data for Impact (D4I)
Health interventions work best when they are based on evidence. However, it can be challenging to make evidence-based decisions on the benefits of implementing one intervention over another when the results are not comparable. In the global health context, lack of consensus and inconsistent definitions of key indicators can make it impossible to compare results between programs, approaches, countries, and funding organizations. The inability to directly compare outcomes can lead to poor decision making on a large scale, resulting in incorrect assessments and potential investment in less effective programming. Only when programs align their data systems and metrics can they generate information that can be compared and combined to improve programmatic knowledge.
Indicator standardization helps ensure that the same data are tracked across locations and programs. Sometimes standardized indicators exist but are not consistently used. For example, median age at first sex is well-defined and is the recommended indicator to measure age at first sex. However, competing indicators are sometimes used instead. These include “mean age at first sex,” “average age of sexual initiation among youth ages 14-19,” or “age at first intercourse by characteristics of youth.” The slightly different wording, definitions, and calculations of these indicators yield different information and limit comparability.
Sometimes standardized indicators do not exist. Contraceptive method availability is a common indicator of method choice. However, guidance varies on the number of contraceptive methods that should be available and whether specific categories of methods should be available. D4I examined the impact of using various measures of method availability with data from Bangladesh, the Democratic Republic of Congo (DRC), and Haiti. Learn more here.
Sometimes standardized indicators are modified in ways that make them unusable outside of a program-specific context. When an implementing program selects indicators for its monitoring and evaluation (M&E) system, a delicate balance is required. While it is important for a program to include customized indicators to capture data and outcomes specific to their goals, maintaining the definitional integrity of standardized indicators is critical. Over-customized indicators can be inconsistent across programs, not comparable, and limit the utility of data for M&E.
Which modifications are acceptable depends on the indicator, a solid understanding of what information it is intended to capture, and how it should be collected. Modifications of time boundaries (e.g., the last 6 months versus the last 12 months) may be acceptable, whereas modifications on age boundaries (e.g., 15–19 years versus 18–24 years) may not be. Modifications that affect the data source, such as using information from a routine health information system instead of a survey, are often problematic as they change the composition and representativeness of the population surveyed. When modifying the data source is unavoidable, the change should be detailed in reports that use these data.
Indicator reference resources are living documents that are updated as research expands our understanding of a population’s nuances and health needs. D4I contributes to harmonizing and standardizing indicators by updating M&E indicator tools, such as the Family Planning/Reproductive Health Indicators Database developed under our former project, MEASURE Evaluation. We collaborate with funders, implementers, and Monitoring, Evaluation, and Learning professionals to make sure indicators in the database are aligned with researchers’ findings. The figure below provides examples of D4I’s work in this area.
Couple-Years of Protection
D4I supported the update of the Couple-years of protection (CYP) indicator to reflect new conversion factors to estimate contraceptive method use. The update also included new guidance on data sources for the indicator to ensure comparability.
Age at First Marriage
D4I added new gender considerations to the indicator reference sheet for the Age at first marriage indicator based on 2022 work of the Child, Early, and Forced Marriage and Unions and Sexuality Working Group.
Family Planning Financing
D4I has helped partners identify high-quality indicators for family planning financing for global measurement. These indicators will soon be available in the Health System Strengthening section of the database.
Achieving and maintaining standardization is difficult and requires ongoing effort. With these lessons learned, D4I will continue to collaborate alongside international, national, and local organizations, agencies, initiatives, and M&E leaders to update and align our guidance documents on key indicators. The challenge is immense as M&E expertise is diffuse and there is no single source for indicator resources. But with our focus on data quality, standardization, and research findings, D4I continues to be a leader in indicator guidance, supporting the work being done to improve health outcomes around the globe.
Dr. Janine Barden-O’Fallon, Senior Technical Advisor for Population and Reproductive Health for D4I, has worked in the field of monitoring, evaluation, and research in family planning and reproductive health for more than 20 years. She specializes in the collection of primary quantitative and qualitative data, developing M&E tools and methodologies, and providing technical assistance to global initiatives, government agencies, and organizations on the collection, analysis, and use of information for program planning and decision making.
Bridgit Adamou, Senior Family Planning Technical Advisor for D4I, has over 20 years of work experience in international development. Her technical areas of expertise include M&E of family planning, tuberculosis, scaling up best practices, community-based HIV programs, and gender integration. She is an expert in capacity building and training, management of small grants, indicator development, stakeholder engagement, and development of M&E plans.