Just like many of you, I ended the year sifting through donation appeals flooding my inbox. Organizations made very cogent arguments about helping refugees, ameliorating hunger and empowering women in war-torn countries. My heart got the message, but these appeals failed to convince me to take action on behalf of these organizations. I needed more proof that these initiatives were actually making a difference.
That’s why I was thrilled to learn about ImpactMatters, an organization launched by Yale economist Dean Karlan that conducts Impact Audits. Impact audits will guide nonprofits to create evidence of impact and use it in its organizational decision-making. It will also guide donors to give to nonprofits that can actually demonstrate that they are making a difference. The key question tackled by Impact Audits: Does the organization use and produce appropriate evidence of impact? FXB is excited about this initiative since measuring impact is at the heart of our work.
At FXB, we are constantly investing resources to determine how effectively our programs work, what makes them work and how we can improve them for the extreme poor. How do we do this?
We Collect Data
FXB monitors its programs on an on-going basis and evaluates progress semi-annually. At the outset, FXB staff conducts a vulnerability assessment on potential households. Once the participants are selected, FXB administers a household questionnaire. We administer a similar questionnaire at the end of the FXBVillage program as well. To ensure our model is creating long-lasting results, we also follow up with participants one year and two years after the program ends. In addition to carrying out long-term data reporting, FXB staff also record progress from the coaching and training sessions in order to monitors participants’ progress on an ongoing basis.
We Analyze Data
FXB’s country staff is constantly analyzing data as challenges become evident. And on the ground, FXBVillage staff are always adjusting their support strategies to better respond to specific household needs. Since the FXBVillage model rests on the premise that only an integrated and holistic program can alleviate poverty, FXB’s Monitoring and Evaluation manager conducts multidimensional poverty analysis on a regular basis. In 2007, Oxford University researchers, Sabina Alkire and James Foster, developed a multidimensional poverty methodology that uses a counting approach to identifying ‘who is poor’ by considering the range of deprivations they suffer. The resulting measure aggregates information to reflect societal poverty in a way that is robust, can be broken down by regions and groups and, most importantly, can be broken down by dimension and indicator to show ‘how people are poor.’ By focusing on not just the “who” and the “what” but also on the “how” and the “why,” we strive to come up with solutions that are practical and resonate with our participants. What our analysis demonstrates is that after the implementation of our FXBVillage program, the vast majority of selected families are no longer multidimensionally poor.
We Encourage Others to Evaluate Us
• In 2007, an external evaluation led by the Human Sciences Research Council of the FXBVillage program in Rwanda has demonstrated remarkable results: children enroll, remain and advance in school at higher rates than their peers. Furthermore, 86% of the participating families were earning sufficient income to live above the poverty line four years after the program’s end.
• From January 2012 to January 2014, three external evaluations were conducted of an FXBVillage in Burundi, one, two and three years after graduation. The results were very positive, showing that 98.2% of children were going to school regularly and 93.7% of households were eating two or more times per day three years after graduation from the FXBVillage program.
• In 2014, the Oxford Poverty and Human Development Initiative (OPHI) compared data on FXB households in Uganda to neighboring households in terms of wellbeing using their multidimensional poverty index. OPHI’s evaluation found that former FXB beneficiaries are multi-dimensionally less poor than their peers, even three years after they graduated from our program.
In the fall of 2015, FXB partnered with PBG Healthcare Consulting, a cross-disciplinary, graduate student-run organization at the University of Pennsylvania that helped FXB evaluate potential ethically and empirically sound approaches to addressing the control group challenge. FXB was also selected as a client organization for degree candidates from the Harvard Kennedy School Master in Public Policy. These candidates completed a Policy Analysis Exercise, focusing on identifying opportunities and challenges of replicating the FXB model through cross-country data analysis.
By vigilantly analyzing our strengths and weaknesses, FXB strives to maximize our impact and empower as many individuals as possible. We also love learning from other people and organizations. What are your most innovative monitoring and evaluation systems and tools? Leave us a comment below or tweet us at @FXBusa!
– Karina Weinstein, FXB USA Program Director