Evaluation Conclave Kathmandu - Reflections

[Ratna Mangala Sudarshan]

Attending the Kathmandu Conclave: Feb 26-March 1, 2013

Since I reached on the 26^th afternoon, I missed Katherine's opening keynote, (but have since viewed this on the Conclave youtube and although its not the same, I can see why it was so much appreciated!). What I did attend on the 26^th/ was the workshop on Appreciative Enquiry, also on 27th. (Ganapati Ojha and Ram Chandra Lamichchane, also Tessie Tzavaras Catsambas). This approach as I understood it seeks to bring stakeholders together to agree upon a final goal, and then work together to find pathways to get to that goal; and therefore not the usual starting point of stating and understanding a problem, uderstanding its causes etc. There is something very appealing about this and I can see how it could be a very good way of mobilising or facilitating discussions within a community.

It might be revealing to apply AI to an organisation. Say, at ISST? The way it might proceed would be to first put together a core group including a few trustees, members of staff (from research, admin, community centre team), funders (?); and then ask them to interview each other, forming pairs, to understand strengths: What has been the/ your best time (at ISST/ about ISST)? What was your role in that good time? What or who gave support? And then move on to further reflection: Where would we like to see ISST five years from now? How do we get there? What can each person do to contribute? I guess in answering the question of where one wants to be, one would also end up discussing any 'problems' in the present, but I certainly would agree that it is more constructive to build on strengths and dreams. Perhaps we should seriously consider getting someone to facilitate such an exercise.

On the 27^th, after Robert Chambers' keynote, who pointed out that although this is a time of very rapid change we need to ask ourselves whether change is equally rapid for the very poor; we had our panel on Feminist Evaluation with Priya, Rajib, Shraddha; good questions, although Q&A should have been better handled (and mea culpa there..). Some points that were raised by the audience and could be developed more in future work, included noting the response of the implementing agency to suggestions, and finding a way to capture whether any change happened as a result of the evaluation findings; the implications for peer based planning of the finding in Rajib's study that only a small percentage of people pass on information to others; and it was good to see so much interest in the tools or key elements that could help make every evaluation 'feminist' – so confirming the relevance of the Engendering Policy project.

Following this, I attended the UN Women session on Gender responsive evaluation with Shiva Kumar, Rebecca Miller, Yamini Atmavilas and Shreyasi, and the de-construction of programmes to examine the degree of responsiveness. The AI workshop continued on the 27^th.

The talk by Howard White at the end of the day on 27^th on 'Why a theory of change matters for rigorous impact evaluation' was persuasive. Specially interesting was his point about the difference the denominator makes and which is the correct denominator to use. This can be debated. ISST's first evaluation for DD, an NGO, in which we calculated the youth activists that DD works with as a percentage of the population in that age group in the village, had led to quite a lot of discussion since DD had not calculated this number before; we pointed out that since less than 10 % of the adolescents in the sampled villages were actively engaged with DD, they might be better advised to work more intensively in the same places rather than spread out (which was their preference). Similarly DFID wanted the Mahila Samakhya work to be reflected in aggregates such as overall literacy rate – here we argued that MS works with a particular group of women and it is within this group that change should be measured.

I also attended a part of Patricia Rogers' workshop on theory of change on the 1^st, where she distinguished between the causal theory and the action theory that links an intervention to the outcome. I found this useful because it allows for some variety within the same theory of change and stops the latter from becoming too rigid a framework. One can use this to show that even when we believe a causal mechanism to be the same across different areas, the strategies might need to be very differentiated. So in the case of the Uttarakhand Mahila Parishad, the formation of 'whole village women's groups' could be seen as the intervention, which succeeds in improvement in village forests as one outcome. Here one could argue that the causal mechanism in all villages is similar, that the WVWGs stimulate collective action that is mutually beneficial to all residents, but that the way in which this action is stimulated varies, with each village group developing locale-specific strategies.(?)

The 28^th started with Michael Quinn Patton's keynote, summarising the 10 characteristics of the field of evaluation today and also including what he himself called a 'rant' against the use of RCTs. (John Floretta had the opportunity to respond to this on the 1^st morning!).

Attended the workshop on Growth, equity, resilience (A.K.Shiva Kumar and Michael Bamberger). What I took away from this is that there is a new found enthusiasm for measuring 'resilience' or the ability to recover from shocks (whether natural disaster or financial crisis) although the development of suitable indicators to measure resilience is just starting. Truly the world goes round and round. Start with close-knit communities, set in motion forces of modernisation and change that will break down these bonds, experience disaster and realise resilience requires connectedness, go back to valuing 'community'.....And at the same time, there is discomfort with the community that is present. What we want is connectedness and community with gender equality, without the patriarchal frame, without caste/ religious divisions, and so a new kind of community; and such a new community must develop organically if it is to be sustainable. This takes me back to the discussions with Mike Jackson on cognitive dissonance and community learning. CL – a process whereby the people of a given community collectively and on their own initiative address a variety of problems that confront them. Idealistic, but there are examples, maybe its possible more widely. A facilitator for a CL exercise needs to 'have experienced cognitive dissonance personally and have removed it in some measure....having identified and critiqued all his/ her own assumptions.' Else there is danger of 'imposing a pre-determined outcome'.

The CEPA panel on 'Fit for purpose?' (Azra Jafferjee, Udan Fernando) on the 28^th afternoon was a case study of CEPA and its evolution from a focus on evaluation, to gathering evidence, to policy influence. Very nicely done I thought.

***

What do we mean in saying that evaluation is a 'field'? Evaluation draws on different disciplines, is implemented in so many different ways. It is a kind of toolbox - where different disciplines contribute different tools – and the evaluator brings in her own palette? There is a lot of evidence to show that evaluations done with a gender lens do bring in new insights. We could illustrate the difference that is made within an AI approach when gender/ equity is made more explicit; or any other approach. With some approaches the gender lens is easily introduced, adding or making explicit another dimension; with others though it shifts the macro frame of the evaluation.

[Finally – many thanks Shraddha for the excellent planning!]