Stakeholder Participation in Evaluation

[Ranjani K.Murthy]

My name is Ranjani.K.Murthy. I work as an Independent Researcher. I made a presentation on participation of stakeholders in evaluation recently in a Pune workshop. The presentation is attached hereStakeholderparticipationinevaluation.pptx

This presentation examines why stakeholder’s participation in evaluation is important; what is the present status of participation of stakeholders in evaluation in India; what are the challenges in getting the participation of stakeholders; and what are the ways forward for participation in evaluation. Gender intensified and gender specific constraints to women’s participation in evaluation are analysed.

Stakeholders in evaluation include marginalised communities and women amongst them, implementing organisations, evaluation team, and donor organisations. Their participation can be a principle by itself, can be a tool for enhancing the efficiency of the evaluation, and can be a tool for strengthening effectiveness of the evaluation. It is the first and last which should be important rationale for the participation of stakeholders in evaluation.

The present status of evaluations in India, with exceptions, is rooted in power hierarchy between evaluation team and marginalised communities, between evaluation team and implementing organisations, between evaluation team members based on gender, caste, race etc., and between donors and evaluation team. 

Some of the challenges to marginalised communities’ participation in evaluation include that the terms of reference is written and in a language which is not really know to them, the timing of the evaluation in the year, the time of the evaluation during the day and presence of caste and gender hierarchies.  Most of the general barriers like “English written TOR” have a gender intensified impact affecting women more than men as women’s literacy is lower and women have lesser knowledge of English

The challenges to implementing agencies’ participation in evaluation include the fear that evaluation is often linked to funding,  (at times) domination by evaluation team, the fact that they have to manage multiple donors, and field level staff’s world load and their lack of knowledge of English.  Majority of the field level staff are women, with few women being at the leadership levels. Similarly there are challenges to participation at the evaluation team and donor levels.   

Looking at ways forward the presentation would like to share one experience from India where the evaluation was commissioned by the implementing agency, and was seen as a learning process.  The terms of reference was evolved by the staff and the gender expert. Participatory and gender aware methods were identified and staff were trained in these methods. In addition interview schedules were used which were evolved collectively and in local language. Each staff did the impact assessment in another staff’s area and a senior staff consolidated the findings.  The focus of the evaluation was to assess gender and poverty impact, and methods such as gender division of labour mapping,  gender based access and control over resources mapping, body mapping, women’s access to institution mapping and happiness index were used.  The information which emerged was triangulated with information from focus group discussions with groups and data from government service providers like schools, nutrition centers, health facilities etc.  A before after comparison (and reasons for difference mapping) and member and non-member comparison (and reasons for difference mapping) was made to ascertain causality of change.  Such a process was owned more by the implementing agency and to some extent by the marginalised communities/women themselves.    

I seek your inputs to the following questions:

i) Who are the stakeholders in evaluation?

ii) What are the constraints in participation of marginalized communities, and women and sexual minorities amongst them in evaluations?

iii) What are the constraints faced by implementing agencies in participation in evaluation? In particular by women staff?

iv) Would you have any experience to share on overcoming the constraints faced by marginalized women and sexual minorities and women staff in evaluation?

[Rituu B Nanda]

First response from Linkedin

Abraham wondimu •stake holder participation is very crucial specially in multidisciplinary development project I evaluate many projects in this regards I learn a lot of things.

(Thanks to Abraham from Ethiopia for sharing)

[Ranjani K.Murthy]

Rituu B Nanda said:

First response from Linkedin

Abraham wondimu •stake holder participation is very crucial specially in multidisciplinary development project I evaluate many projects in this regards I learn a lot of things.

(Thanks to Abraham from Ethiopia for sharing)

Ranjani: Abraham, how do you involve women and sexual minorities in evaluation.  In what aspect of evaluation do you involve them and how?  Thanks

[Shankar Talwar]

I used "stakeholder analysis" approach to assess natural resource utilization (land, water and forest) in a mountain region in Asia. Stakeholder analysis is an excellent tool to see the varying perspective of different stakeholders.

[John Packiaraj]

Stakeholder analysis helps implementing partners think carefully about who is important to your project and their level of interest or influence. From this analysis, we can decide how they should be involved in various stages of the project cycle. A comprehensive stakeholder analysis encourages participation, involvement and good relations among people, groups, and institutions who were affected by this project. The analysis reveals those stakeholders who were powerful and may have had undue influence. It also helps to identify those stakeholders important to the success of project (women seed sellers, seed traders, farmers, etc.).

A stakeholder analysis will help ensure that the data gathered by the M&E system meet the information needs of the project stakeholders. When designing your project, you may have done a stakeholder analysis to identify who should inform the planning process. If that is the case, it can be reviewed and updated to ensure that you understand stakeholder information needs.

[Sanghamitra Dhar]

Fantastic topic!Thanks for starting it . . . I would love to respond from my personal experience in M&E of a tsunami rehab. prog & some other prog.,but for time constraints. Although I'll try to reply to all the four questions raised by you tonight. 

[Ranjani K.Murthy]

Shankar Talwar said:

I used "stakeholder analysis" approach to assess natural resource utilization (land, water and forest) in a mountain region in Asia. Stakeholder analysis is an excellent tool to see the varying perspective of different stakeholders.

Ranjani: How have you done stakeholder analysis in evaluation? Who are the stakeholders in evaluation?

[Ranjani K.Murthy]

Sanghamitra Dhar said:

Fantastic topic!Thanks for starting it . . . I would love to respond from my personal experience in M&E of a tsunami rehab. prog & some other prog.,but for time constraints. Although I'll try to reply to all the four questions raised by you tonight. 

Ranjani: Awaiting your response. Great!

[Rituu B Nanda]

Another response from Linkedin

 

Charles Aondoaseer Hemba, Nigeria

I agree entirely with Wondimu that stakeholders' participation is very critical in evaluations which can be seen as a means of verifying if the project has had the desired impact on its beneficiaries. It will be pertinent at this point to point out that evaluation succeeds project implementation in the project cycle, therefore a basis will have to be set for an evaluation right from the beginning of project. If this basis is participatory i.e. involves the beneficiaries/stakeholders, evaluation of such a project is usually easier against situations where the projects are just dropped in the stakeholder area. When it is participatory, the stakeholders themselves being in the picture from the beginning will be able to bring out the changes they have experienced as a result of the project. If it is done in a participatory manner, all the intricacies of gender, caste, illiteracy, hierarchies etc would have been taken care of at that level as well as easing the evaluation methodology to be used. This will also inform the composition of the evaluation team when the project is ready for an evaluation.

[Sanghamitra Dhar]

Mr Prasad, it was interesting reading your response and I look forward to more from you on the other questions as you promise :)

 

Kalpathy V Eswara Prasad said:

My name is Eswar and I am a process and institutional development consultant with some interesting evaluation engagements.At the moment I am to-ing and fro-ing from interior Dhalai district and Agartala. Ranjani, you raise some interesting questions.I am responding purely on the basis of my expereince..for whatever it is worth. i) Who are stakeholders in evaluation? This to me depends on the kind of project or program that is being evaluated.stakeholders have some 'stake' in the project and that their support is essential for the success of any project.I find it useful to make it clear right at the beginning of any such venture with client systems that I practice only stakeholder-based evaluation.I have had encounters with client systems where they do not share the ToR with stakeholders! So, as someone who is practicing evaluation seriously,I find it authentic to talk about from the contracting stage. I also find it useful at the start of any evaluation exercise to si with the evaluand or the organization or project or etc. and spend time to do a serious identification of whom they consider as their stakeholders. This is not a listing but much more than that.Who are their stakeholders and what kind of engagement they have /had with the project.how important politically they are to the project,etc. Finally, it does seem useful to identify out of these who the 'Primary' stakeholders are, In the last few years,I have found it also very useful to explore with the project organization, who their 'intended' stakeholders are(See Patton).WHo are those most likely to show interest in the project effort in the future? I often stumble upon interesting responses. Just to give an example a watershead project may have a whole lot of stakeholders that is different from a project implementing Forest Rights.So depending on the nature of the project one needs to slog to identify who the stakeholders are ii) What constraints....? This is a huge asking as many have to pay a price to participate in any evaluation....in the form of opportunity cost. In one particular instance,I do remember that most of the stakeholders, apart from the implementing organization, were daily wage labour who would return late night from work.In such instances, My colleagues and I would conduct meetings with them (with prior I permission) in the night in public spaces such as an open playground as many were women .

I have also found it extremely rewarding to make evaluation as participative as possible.The biggest constraint for the marginalized community seems emerging from their inability to come to terms with the fact that they are more intelligent and articulate than the evaluators! So a variety of methods help-apart from focus group discussions, separate meetings with women in every context, etc.The most rewarding experiences have been for me with tribal women who have expressed themselves through imagery.

I have gone a bit too long and I stop here.I will take up the other questions shortly.

Best

K.V.Eswara Prasad

[Sanghamitra Dhar]

Ms Murthy, interesting ppt & this brief note!

Agreeing with Mr Prasad, I would add that who a 'stakeholder' is in an evaluation varies according to what the project is about, what is it addressing, for instance a 'livelihood project on pisciculture' & a 'livelihood project on pickle-making & other such eatables' in a tsunami rehabilitation programme encouraged both the genders to engage into their respective choice of work which was designed gender-sensitively. In the former project primarily the men of the community were engaged although it was found that most often engagement of the womenfolk is also encouraged and practiced but when we got into the process of identifying the stakeholders with the community (Participatory approach, ofcourse!), it were the men as forerunners & quite agreeably so! although the women identified themselves as secondary or tertiary stakeholder, as per there involvement and vice-versa was the case in the women-centered project. However, in a programme addressing gender- discrimination the identification of stakeholders was complex as it has varied, multi-layered 'stakes' and dimensions.

As far as sharing the ToR with the community is concerned I believe in the inception stage of a project when a baseline is being carried out & project is being formalized (I had the good fortune to associate with such a programme), the implementing agency which are serious about their operations in the field do have a detailed discussion about what ought to be the indicators of impact-assessment for the project cycle - midterm, endterm & so on. Its actually a methodological stand - if its participative, facilitative nature then that top-down, 'we-will-tell-you-whats-good-for-you' doesn't come into play.

Another factor as a development practitioner, I have realised and imbibed in my work at all times and humbly request everyone to keep in mind is that we should not be overambitious while conducting an evaluation and reason our arguments logically according to the appropriateness of the project.

BTW Ms Murthy, language is really not a matter (translators are there) in field settings if the facilitator is skilled and amicable, if it weren't so I won't have had such enriching experiences in different parts of India, as practically I couldn't have learnt all the languages where I went for the evaluations ranging from Kerala/TN/MP/Orissa/Assam etc. 

Constraints about discussions on sensitive issues (caste, religion, politics, etc) really should be entrusted on people who have good interpersonal skills, for instance when I was studying a state road project in Darrang district in Assam, we (I & my male colleague) encountered a strangely volatile atmosphere that we were able to pacify only due to our 'people-skills' we joked later! [they later shared was out of their distrust on 'experts' who were not doing anything constructive about there situation, we were probably the third team of evaluators!]

Also I find everyone using the terms project and programme in synonymity however I'm using it as distinct.

Hope I have been able to share on all the queries you raised and look forward to more stimulating discussions on the same. Although I have tried hard to cut down on my wordcount but sorry it still looks too lengthy! :)

Sanghamitra Dhar said:

Fantastic topic!Thanks for starting it . . . I would love to respond from my personal experience in M&E of a tsunami rehab. prog & some other prog.,but for time constraints. Although I'll try to reply to all the four questions raised by you tonight. 

Ranjani: Awaiting your response. Great!

[Ranjani K.Murthy]

Sanghamitra-Thanks for your useful comments. Ranjani

Sanghamitra Dhar said:

Ms Murthy, interesting ppt & this brief note!

Agreeing with Mr Prasad, I would add that who a 'stakeholder' is in an evaluation varies according to what the project is about, what is it addressing, for instance a 'livelihood project on pisciculture' & a 'livelihood project on pickle-making & other such eatables' in a tsunami rehabilitation programme encouraged both the genders to engage into their respective choice of work which was designed gender-sensitively. In the former project primarily the men of the community were engaged although it was found that most often engagement of the womenfolk is also encouraged and practiced but when we got into the process of identifying the stakeholders with the community (Participatory approach, ofcourse!), it were the men as forerunners & quite agreeably so! although the women identified themselves as secondary or tertiary stakeholder, as per there involvement and vice-versa was the case in the women-centered project. However, in a programme addressing gender- discrimination the identification of stakeholders was complex as it has varied, multi-layered 'stakes' and dimensions.

As far as sharing the ToR with the community is concerned I believe in the inception stage of a project when a baseline is being carried out & project is being formalized (I had the good fortune to associate with such a programme), the implementing agency which are serious about their operations in the field do have a detailed discussion about what ought to be the indicators of impact-assessment for the project cycle - midterm, endterm & so on. Its actually a methodological stand - if its participative, facilitative nature then that top-down, 'we-will-tell-you-whats-good-for-you' doesn't come into play.

Another factor as a development practitioner, I have realised and imbibed in my work at all times and humbly request everyone to keep in mind is that we should not be overambitious while conducting an evaluation and reason our arguments logically according to the appropriateness of the project.

BTW Ms Murthy, language is really not a matter (translators are there) in field settings if the facilitator is skilled and amicable, if it weren't so I won't have had such enriching experiences in different parts of India, as practically I couldn't have learnt all the languages where I went for the evaluations ranging from Kerala/TN/MP/Orissa/Assam etc. 

Constraints about discussions on sensitive issues (caste, religion, politics, etc) really should be entrusted on people who have good interpersonal skills, for instance when I was studying a state road project in Darrang district in Assam, we (I & my male colleague) encountered a strangely volatile atmosphere that we were able to pacify only due to our 'people-skills' we joked later! [they later shared was out of their distrust on 'experts' who were not doing anything constructive about there situation, we were probably the third team of evaluators!]

Also I find everyone using the terms project and programme in synonymity however I'm using it as distinct.

Hope I have been able to share on all the queries you raised and look forward to more stimulating discussions on the same. Although I have tried hard to cut down on my wordcount but sorry it still looks too lengthy! :)

Sanghamitra Dhar said:

Fantastic topic!Thanks for starting it . . . I would love to respond from my personal experience in M&E of a tsunami rehab. prog & some other prog.,but for time constraints. Although I'll try to reply to all the four questions raised by you tonight. 

Ranjani: Awaiting your response. Great!

[Ranjani K.Murthy]

Easwar, your response was very interesting. I am particularly interested in the use of imagery and participatory methods in evaluations. Do share these when you get the time. 

Sanghamitra Dhar said:

Mr Prasad, it was interesting reading your response and I look forward to more from you on the other questions as you promise :)

 

Kalpathy V Eswara Prasad said:

My name is Eswar and I am a process and institutional development consultant with some interesting evaluation engagements.At the moment I am to-ing and fro-ing from interior Dhalai district and Agartala. Ranjani, you raise some interesting questions.I am responding purely on the basis of my expereince..for whatever it is worth. i) Who are stakeholders in evaluation? This to me depends on the kind of project or program that is being evaluated.stakeholders have some 'stake' in the project and that their support is essential for the success of any project.I find it useful to make it clear right at the beginning of any such venture with client systems that I practice only stakeholder-based evaluation.I have had encounters with client systems where they do not share the ToR with stakeholders! So, as someone who is practicing evaluation seriously,I find it authentic to talk about from the contracting stage. I also find it useful at the start of any evaluation exercise to si with the evaluand or the organization or project or etc. and spend time to do a serious identification of whom they consider as their stakeholders. This is not a listing but much more than that.Who are their stakeholders and what kind of engagement they have /had with the project.how important politically they are to the project,etc. Finally, it does seem useful to identify out of these who the 'Primary' stakeholders are, In the last few years,I have found it also very useful to explore with the project organization, who their 'intended' stakeholders are(See Patton).WHo are those most likely to show interest in the project effort in the future? I often stumble upon interesting responses. Just to give an example a watershead project may have a whole lot of stakeholders that is different from a project implementing Forest Rights.So depending on the nature of the project one needs to slog to identify who the stakeholders are ii) What constraints....? This is a huge asking as many have to pay a price to participate in any evaluation....in the form of opportunity cost. In one particular instance,I do remember that most of the stakeholders, apart from the implementing organization, were daily wage labour who would return late night from work.In such instances, My colleagues and I would conduct meetings with them (with prior I permission) in the night in public spaces such as an open playground as many were women .

I have also found it extremely rewarding to make evaluation as participative as possible.The biggest constraint for the marginalized community seems emerging from their inability to come to terms with the fact that they are more intelligent and articulate than the evaluators! So a variety of methods help-apart from focus group discussions, separate meetings with women in every context, etc.The most rewarding experiences have been for me with tribal women who have expressed themselves through imagery.

I have gone a bit too long and I stop here.I will take up the other questions shortly.

Best

K.V.Eswara Prasad