Assuming Sustainability + the SDG criteria

[Jindra Cekan, PhD]

Folks, following on from Rituu's blog about the EES conference, here's mine on sustainability and the SDGs, not gender per se but maybe interesting. Waving!

Assuming Sustainability and Impact is Dangerous to Development (+ OECD/ DAC evaluation criteria)

We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impactsuch as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.

INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions”.  They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact.”  Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer.  Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable.” 

How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.

https://i0.wp.com/valuingvoices.com/wp-content/uploads/2018/10/IMG_8075.jpg?resize=207%2C136 207w, https://i0.wp.com/valuingvoices.com/wp-content/uploads/2018/10/IMG_... 260w, https://i0.wp.com/valuingvoices.com/wp-content/uploads/2018/10/IMG_... 430w, https://i2.wp.com/valuingvoices.com/wp-content/uploads/2018/10/IMG_... 640w" sizes="(max-width: 300px) 100vw, 300px" />

 

 

 

 

 

 

Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).

There are remarkable assumptions, such as this USAID Uganda CDCS Country Transition Planwhich looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit. Truly, we can plan to exit, but only when data bears out our sustained impact, notwhen the money or political will runs out.

As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters.”  In fact, most ‘impact evaluations’ actually test effectiveness rather than l... Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.

if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!

https://i0.wp.com/valuingvoices.com/wp-content/uploads/2018/10/AGDEN-DAC-subCriteria.jpg?resize=768%2C576 768w, https://i2.wp.com/valuingvoices.com/wp-content/uploads/2018/10/AGDE... 1024w, https://i0.wp.com/valuingvoices.com/wp-content/uploads/2018/10/AGDE... 2000w" sizes="(max-width: 300px) 100vw, 300px" />

We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants.  We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership.” Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms.”  It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…

http://valuingvoices.com/assuming-sustainability-and-impact-is-dang...