Evaluation for the World We Want! My takes from the Canadian Evaluation Society Conference, 2015

[Ranjani K.Murthy]

The theme of the 2015 Canadian Evaluation Society (CES) Conference "Evaluation of the World We Want" appealed to my mind and heart and I hence decided to submit a paper and apply for bursary. I was fortunate to get part bursary from the CES and part from the Indian Social Studies Trust which enabled me to participate in the Conference. 

The presentations at the opening plenary were thought provoking.  They drew attention to the need for locating evaluations in the context of a development paradigm that focused on equity, human rights, sustainable development and climate change adaptation. It was emphasized that such evaluations would need to look at, and promote, gender and diversity as well. 

If evaluations are to make such a difference, they need to strengthen “evaluation power” (capacities and authority) of agencies involved in evaluations as well as enhance ”power of evaluation” to speak the truth to power holders.  An indicator of the latter was whether and how had budgets/expenditures shifted to incorporate the recommendations. 

Several methods of capturing shifts in power relations were discussed in the CES.  One was using a ‘SenseMaker’ (http://www.sensemaker-suite.com/) to record key shifts in power relations indicated in responses of adolescent girls and mothers to questions on the lives of adolescent girls (in general, not each individual in particular).  Thus if 100 adolescent girls were interviewed, it was recorded as to how much percentage reported that the girls were able to continue in school after onset of puberty, avoid child marriages, etc.  This method could be  used with adolescent boys  or any marginalized group as well to understand shifts in gender and social dynamics.

Another method was to asked women and men from low income groups what had changed in the last five years (for better and worse) and why, and what the government should do to move towards the world women and men from marginalized groups wanted.  This method, like the ‘SenseMaker’,  helped understand the shifts in power and reasons across sectors (e.g health, education) and institutions  (e.g. family, community, market, state etc.); and contrasted with the sectoral  and input/process/output focus of  government evaluations. A power-point on this follows!

A challenging evaluation method for assessing rule of law which is vivid in  my memory is use of human rights monitors to track what happens to a law suit once it is filed (and the person who filed the same), recording demands of bribe and hurdles to justice.  This method, while posing risks for the human rights monitor, has drawn attention to procedural and institutional loopholes as well vagaries of humans.  Such monitoring has enhanced access to justice, but may need to be repeated periodically.  For assessing the rule of law at the national level,  the World Council  of Justice has evolved the Rule of Law index  which is used to rate countries (see http://worldjusticeproject.org/rule-of-law-index.) This index focuses on state accountability, and could incorporate indicators of adherence to rule of law by families, community institutions and work place organisations.  

Evaluation of partnerships is more difficult, and participants in one panel pointed to the need for assessment of membership, frequency of meetings, level of participation, capacity enhancement of members, production of value addition tools,  collective action for change and contribution to changes in practice or policy/legal change . While partnerships may involve multiple partners, whether individual members develop horizontal relationship with another based on shared relationship could be another indicator.   

 The need for quantification in addition to qualitative evaluation was called for by several speakers.  Some spoke in the northern context, pointing that evaluations should point to areas which could be ‘cut back’ with least damage given the budget deficit. On the other hand a majority were concerned that evaluation should also point to social return on investment.  Interventions should be assessed for their social returns, as well as compared with social returns on alternative forms of investment using the same money.  The difference between social return and cost-benefit analysis was debated.         

 At a conceptual level, the UN agencies touched on the elements of gender-mainstreaming.  They themselves called upon the need for greater clarity between terms such as gender-responsive, gender-sensitive, gender-aware and gender-equitable mainstreaming. 

Thus Evaluation of the World We Want is complex.  Evaluation has to both speak to power and alter power structures, resource distribution and institutional norms. What this means may from context to context.

Yes, this was a Conference I immensely enjoyed. The hospitality and 'buddy' system (pairing me with a Canadian participant- Nancy on the photo) and the charm of Montreal were added bonuses.   

 Ranjani.K.Murthy, 2015

[e-mail]

[Rukmini Panda]

Dear Ranjinidi

Thanks a lot for sharing this thought provoking piece.

Regards,

Rukmini

[Rituu B Nanda]

Thanks Ranjani. We can't attend all the conferences but can learn from each other who attend. I particularly enjoyed learning about evaluating partnerships through you. I used to coordinate a large sex worker network so of interest to me.

Wish you had also got some cool breeze from Canada too...very hot here in Delhi:-)