Since April 2014, Tostan has been collaborating with the Bill and Melinda Gates Foundation and ITAD to revise our monitoring and evaluation (M&E) system to improve decision making for program implementation and strategic planning. In March of this year, Tostan was invited to “Building an Evaluation Framework for Integrated Development Approaches: A Brainstorming and Conceptualization Workshop,” hosted by FHI 360 to share our experiences in the development of the new system. FHI 360 and Tostan are receiving similar grants focused on the monitoring and evaluation of integrated development (ID) programs from the Bill and Melinda Gates Foundation. While Tostan’s grant is focused on establishing a system to measure and evaluate an ID program, FHI 360 has a grant to create a general framework to evaluate an ID program. A framework is something that can be used as a guide to effectively design an evaluation: a tool to better understand the results, or outcomes, that are happening in a community.
As defined by FHI 360, an integrated development program “is the deliberate approach to connect the design, delivery and evaluation of programs across disciplines and sectors to produce an amplified, lasting impact on people’s lives…For the purposes of evaluation, we are focused on program-level approaches.” Tostan’s Community Empowerment Program (CEP) fits within this ID model because of its holistic nature. This workshop was an opportunity for Tostan, represented by MERL Officer Elizabeth Larson, to have an elaborate exchange with experts in ID program evaluation and connect with organizations with similar models. For example, Nuru International, an NGO based in Ethiopia and Kenya, has also successfully evaluated its ID program.
During the workshop, Elizabeth explained Tostan’s approach to holistic development, describing the integral connections between government, education, health (santé in French), environment, and economic development—collectively referred to as the GESEE indicators. Next, Elizabeth described the Tostan approach within communities, discussing the CEP’s non-formal education classes. Additionally, she introduced Tostan’s Theory of Change, which demonstrates how communities evolve over the course of the three-year CEP from having potential with low levels of self-efficacy, to empowered communities with high levels of self-efficacy and collective action.
Elizabeth then presented our current M&E system, providing an overview of our Results Framework, which focuses on the GESEE indicators, and each of the tools Tostan uses to collect data. For example, results from the Generational Change in Three Years (GC3Y) Project’s midterm evaluation in Guinea show that governance had improved since the baseline study and program implementation, as measured by public decision-making based on sex.
Tostan is also known for our approach to measuring social norms, which addresses normative and empirical expectations, the prevalence of a practice, and personal attitudes towards a practice.
Elizabeth discussed the importance of Tostan’s mixed methods approach—using both qualitative and quantitative data collection—and how it enables Tostan to gain an understanding of underlying motivations for the visible change in communities. The power of a mixed methods approach is demonstrated when reviewing the quantitative and qualitative data on normative expectations surrounding female genital cutting (FGC). In the midterm evaluation for the GC3Y Project in Mali, we used qualitative data in conjunction with quantitative data, like the following example, to get a global picture of the situation surrounding social norms.
Finally, Elizabeth described how Tostan uses data for strategic decision-making and action at the community level. For example, data is given to the Programs department to inform them of the CEP’s performance. Likewise, evaluation results are shared with the Communications department to develop infographics to distribute to the Tostan community—including the communities in which Tostan works, so that participants can use the data to advocate for further change on the ground. Lastly, Tostan is working to incorporate participatory action research methods at the community level within its M&E system. This will enable Tostan to understand changes occurring that it could not measure within its established M&E system.
After Elizabeth’s presentation, the participants brainstormed a number of key questions that evaluators should answer through ID program evaluation, such as:
- What are the minimum standards that can be captured (by the implementers) to understand the amplified impact of integration? What are the things that you can leave out?
- What are people’s assumptions on the value of integration?
- Does ID lead to less effort? Does ID lead to more happiness?
- What are the unintended consequences of using an integrated approach?
The workshop concluded with a discussion of what the participants would be looking for in an ID framework by responding to the following question, “What should a practical ID evaluation consist of, in terms of both content and format?” The following represents some of the key responses:
- Should be more of decision making resource rather than a how-to manual, this will make it applicable to a wider variety of potential users
- Use diverse and detailed case studies
- Make it as easy to use as possible by providing a short introduction linking to more detailed explanations
- Provide definitions for all the jargon since it can differ between sectors
In the end, exchanging ideas and lessons learned with our peers who believe in and successfully implement ID programs, we became one step closer to fine-tuning our own framework and M&E system.
With contributions from Elizabeth Larson, MERL Officer
