On My Homework After Attending Evaluation Conclave 2013 at Kathmandu

[Krinna Shah]

I am glad that I had attended Opening Ceremony, keynote speech ‘Opportunities and Challenges for Participation in Evaluation’, ‘Participatory Approaches to Evaluating Communication for Development’, ‘ Innovative Directions  for Evaluation of Development, ‘The Cost of Inaction’ and ‘Learning and Participating the Evaluation Mantra in South Asia’.  The panels I had attended are: ‘It Takes Two to Tango: Translating Research into Policy’, ‘Feminist Evaluation’, ‘Fit for Purpose? Evaluations, Evidence and Policy Influence’ and ‘Systematic Reviews in International Development’.   I had attended two workshops: ‘Real World Evaluation’ and ‘Impact Evaluation: Theory and Practice’. I had attended Design Clinic on Planning, Managing and Reporting Evaluation. Mentioning titles of each session may indicate what I would have learned. It also means how much I need to read and learn! Scary!   

Keynote speech by Katherine Hay at the opening ceremony was powerful to reach my mind and heart at once. I will revisit her speech and presentation once they are uploaded at the website of the Evaluation Conclave  2013. I have clicked few of her slides having apt title and that will be a good reminder to use visuals and also visual data (as example on evaluation on use of helmet shared by Michael Quinn Patton in video conference) in my presentations.

Are things changing faster or slower for poor people? Are we increasingly trapped in metro cities? Now these questions are tough to look at. By the time I pay attention to my feelings on these questions he further said Evaluation is light years behind then what is happening in the field.  That gives me a jerk. As in, while being part of child rights sector we are too busy coping with need of implementing various projects and trying to establish best practises we rarely pay attention to evaluation.  There is hardly any thinking, resource and skills on evaluation in this sector (in my very personal opinion) at least at project level. This unfolded in following sessions where I had participated. The evaluations do happen for large scale programmes and the lessons are not shared at the field level, I feel. I am open to change my opinion if someone can guide me to the relevant resources.

Do give me a feedback, as this is my first blog and I would like to make corrections as soon as possible, if anyone suggest so. Thanks!

Evaluation is an intervention when done in participatory way for people who are involved.  I would like to ensure this in the mid-term evaluation which will happen soon in the project Strengthening Existing Systems for Prevention of Child Marriage which I look after. I think it is useful to have such very basic concepts in mind while undertaking any activities. It gives a right direction to the work.  There was a reference of measuring empowerment. I will explore more on this in coming one year. There was a question posed “Have the indicators changed?” The common sense of deciding indicators seems to be looked into. That session ended with two liberating basics: ASK THEM! THEY CAN DO IT! Yes, you have guessed it right! This is what I recall from Robert Chambers’ key note speech. The outline of his speech along with sources which was circulated becomes my homework! His speech revived my faith in work per se.

Change is not liner. It is non-systematic. Change is different for different people. This is so relevant with the project we are implementing. The project is about preventing child marriage and it is about social change by strengthening system.  In sort duration it is a challenge to measure any change, but if we can understand the very nature of change we will be able to make some sense of our interventions. Later on during informal discussion after the workshop on Impact Evaluation, I realised that we need to look into Theory of Change. We are ultimately working towards improving the quality of life of children who would otherwise be pushed into marriage which is socially accepted practice. In fact some of them are being married irrespective of interventions done at grass root level by the project team and/or government officials. Priya Nanda’s presentation was about feminist evaluation applied on measuring how women have experienced change in their personal and professional life in the context of workplace programme for women’s empowerment. I am   thinking to capture the change the girls are experiencing in the prevention of child marriage project. Is there any group anywhere in the world using children’s lens in evaluation like feminist lens in evaluation?  I would love to know and learn. Someone has offered me to join during my filed trip to demonstrate participatory methods of evaluation with children she has been using! I wish I could utilise her availability in the project. Someone else said that what he makes sure is to take a participants’ list from any such event. He elaborated by adding that  what all can happen after the event have more direct meaning to work than what all happen at the event! I do agree with this as I am trying to take forward some of the discussions initiated at the Conclave. One of them is communication for social change.

During workshop Engendering Policy through Evaluation, 6-8 Feb.2013, among 3 questions I had posed to Ranjani K Murthy, I had asked, “Is any of evaluation she did had positive results?”  I got teh positive answer. In my limited knowledge on evaluations in child rights sector, I had a feeling that at times evaluations are used to maintain status quo and not to modify any strategy.  The reports of evaluations are used for proving and not improving. With this background I attended the panel: Fit for Purpose? Evaluation, Evidence and Policy Influence.  The only presentation of the panel by Azra Jafferjee (CEPA) was about sharing experiences. I would like to learn about concepts and theories but grass root experiences are more useful in our day-to-day work of fieldwork or advocacy. I thoroughly enjoyed the presentation. This was by then second panel were I was taking photographs on request. I found that clicking photographs made me pay more attention on which moment to capture while sitting and listening.  The panellist had shared most important lesson to take home i.e. Evidence does not speak for itself and evaluators have a responsibility to make a difference in the lives of poor people. It was discussed that evidence is not enough as there are many evidence which can influence the policy but we should work towards providing support to policy makers on how to use evidence. The Chair, Udan Fernando referred the role of the organisation (CEPA) as loud speaker and the antenna at the same time as a group of evaluators. This means constantly sharing the evidence and also be on look out of opportunity of how to use the evidence to influence policy.  Based on their many evaluations, their lessons learnt are:  be rigorous, focussed, engaged, practical, build alliances, be modest and understood; and finally be on it!  Though I am not an evaluator, yet their work seems very close to implementation I understand. I recommend that if possible please do listen the session to find three stories of influencing policy.  I will be unfair if I don’t mention the difficulty the panellist had with collar mike and other mikes as well. Just like Azra’s panel, I had enjoyed the workshop by Jim Rugh. He is a good trainer/teacher and I enjoyed like a school student.

The presentation on ‘ A Systematic Review of Slum Upgrading, Effects on Socio-Economic Outcomes’ by   Ruhi Saith  (JHU) was interesting. This could be found as a blog on the link below.  

http://evidentlycochrane.wordpress.com/2013/02/06/guest-blog-slum-u...

She shared the details of their work, including the studies they reviewed. Interestingly Howard White from 3ie had made it amply clear that a systematic review is not a literature review.  His introductory presentation was a good start, but as it was for the first time I was listening on the subject I could not grasp all of it.  During Ruhi’s presentation I was able to understand the process. I tried to connect such review for the prevention of child marriage project. Another homework! The question which lead to this review is  ...  valuable resources continue to be invested in slum upgrading strategies, how can these resources be invested in the most effective and efficient way? How can policy makers and implementers know what interventions will work to improve the health and wellbeing of those living in slum areas?  (See the blog mentioned above).  By then it was clear to me that the project I had mentioned above should be seen in the light of the evaluations discussed at various places of the Conclave and the given resources for evaluation. I am sure if I had that much technical knowledge on evaluation I would have made more sense of Hugh Waddington’s presentation. Nevertheless only by knowing what I don’t know, what is essential to know and what would be added advantage if I know; I will be able to be more meaningful at work. One point of discussion in this panel was a review of studies included unpublished evaluations as well, as they might not be about successful outcome. This made me think that when I look for what works and what does not, I should not be limited to look for success stories/ best practice but also look for negative outcomes to develop holistic understanding on interventions.

I had opportunity to observe Robert Chambers more than once as a participant! I have shared his picture at the link below as evidence for learning by observation and as a reminder to self.

I end my little longer than 1500 word report on what I have learned from the evaluation conclave and how it will help me in my work by thanking a lot to ISST’s tem for facilitating my participation and all the members of the team of Feminist Evaluation whose presence had enhanced my learning at the Conclave.  Guest blog: Slum upgrading – improving health and wellbeing?EVALUATION FOR DEVELOPMENT, 26TH FEBRUARY -1ST MARCH, 2013