Stakeholder Participation in Evaluation

[Ranjani K.Murthy]

My name is Ranjani.K.Murthy. I work as an Independent Researcher. I made a presentation on participation of stakeholders in evaluation recently in a Pune workshop. The presentation is attached hereStakeholderparticipationinevaluation.pptx

This presentation examines why stakeholder’s participation in evaluation is important; what is the present status of participation of stakeholders in evaluation in India; what are the challenges in getting the participation of stakeholders; and what are the ways forward for participation in evaluation. Gender intensified and gender specific constraints to women’s participation in evaluation are analysed.

Stakeholders in evaluation include marginalised communities and women amongst them, implementing organisations, evaluation team, and donor organisations. Their participation can be a principle by itself, can be a tool for enhancing the efficiency of the evaluation, and can be a tool for strengthening effectiveness of the evaluation. It is the first and last which should be important rationale for the participation of stakeholders in evaluation.

The present status of evaluations in India, with exceptions, is rooted in power hierarchy between evaluation team and marginalised communities, between evaluation team and implementing organisations, between evaluation team members based on gender, caste, race etc., and between donors and evaluation team. 

Some of the challenges to marginalised communities’ participation in evaluation include that the terms of reference is written and in a language which is not really know to them, the timing of the evaluation in the year, the time of the evaluation during the day and presence of caste and gender hierarchies.  Most of the general barriers like “English written TOR” have a gender intensified impact affecting women more than men as women’s literacy is lower and women have lesser knowledge of English

The challenges to implementing agencies’ participation in evaluation include the fear that evaluation is often linked to funding,  (at times) domination by evaluation team, the fact that they have to manage multiple donors, and field level staff’s world load and their lack of knowledge of English.  Majority of the field level staff are women, with few women being at the leadership levels. Similarly there are challenges to participation at the evaluation team and donor levels.   

Looking at ways forward the presentation would like to share one experience from India where the evaluation was commissioned by the implementing agency, and was seen as a learning process.  The terms of reference was evolved by the staff and the gender expert. Participatory and gender aware methods were identified and staff were trained in these methods. In addition interview schedules were used which were evolved collectively and in local language. Each staff did the impact assessment in another staff’s area and a senior staff consolidated the findings.  The focus of the evaluation was to assess gender and poverty impact, and methods such as gender division of labour mapping,  gender based access and control over resources mapping, body mapping, women’s access to institution mapping and happiness index were used.  The information which emerged was triangulated with information from focus group discussions with groups and data from government service providers like schools, nutrition centers, health facilities etc.  A before after comparison (and reasons for difference mapping) and member and non-member comparison (and reasons for difference mapping) was made to ascertain causality of change.  Such a process was owned more by the implementing agency and to some extent by the marginalised communities/women themselves.    

I seek your inputs to the following questions:

i) Who are the stakeholders in evaluation?

ii) What are the constraints in participation of marginalized communities, and women and sexual minorities amongst them in evaluations?

iii) What are the constraints faced by implementing agencies in participation in evaluation? In particular by women staff?

iv) Would you have any experience to share on overcoming the constraints faced by marginalized women and sexual minorities and women staff in evaluation?

[Ranjani K.Murthy]

Thanks Charles for observing that participatory evaluation can only happen when the project is itself participatory. In total agreement. Ranjani   

Rituu B Nanda said:

Another response from Linkedin

 

Charles Aondoaseer Hemba, Nigeria

I agree entirely with Wondimu that stakeholders' participation is very critical in evaluations which can be seen as a means of verifying if the project has had the desired impact on its beneficiaries. It will be pertinent at this point to point out that evaluation succeeds project implementation in the project cycle, therefore a basis will have to be set for an evaluation right from the beginning of project. If this basis is participatory i.e. involves the beneficiaries/stakeholders, evaluation of such a project is usually easier against situations where the projects are just dropped in the stakeholder area. When it is participatory, the stakeholders themselves being in the picture from the beginning will be able to bring out the changes they have experienced as a result of the project. If it is done in a participatory manner, all the intricacies of gender, caste, illiteracy, hierarchies etc would have been taken care of at that level as well as easing the evaluation methodology to be used. This will also inform the composition of the evaluation team when the project is ready for an evaluation.

[Shankar Talwar]

This was done in the year 1996-97 while I was working as a post doc at the Ecological Anthropology department of university of Georgia, Athens, USA. This was a part of a course on Human dimensions of natural resources. We typically analyzed all stakeholders using forest, waterbodies like river and lakes, and cultivated land downstream. I do not have any soft copies of the document. I will try and write as and when it is possible.

Ranjani K.Murthy said:

Shankar Talwar said:

I used "stakeholder analysis" approach to assess natural resource utilization (land, water and forest) in a mountain region in Asia. Stakeholder analysis is an excellent tool to see the varying perspective of different stakeholders.

Ranjani: How have you done stakeholder analysis in evaluation? Who are the stakeholders in evaluation?

[Rituu B Nanda]

Third response from Linkedin

Erick Ngosia from Kenya. Thanks Eric!

Stakeholder participation is important in any project if effective results have to be achieved. I work on an accountability project that plays an oversight role on the Constituency Development Fund (CDF) and LATF ( Local Authority Transfer Fund). Here, the CDF refers to funds the government gives each constituency in our country for development and the Local Authority transfer Fund refers to funds given by the local government to help the society within the municipalities. It is however from this that we came up with stakeholders who include citizens from the municipalities covered by the local authorities and citizens from the given constituencies where this funds are channeled to. Stakeholders have to be informed of the monitoring approach and methods to be used and also to have the capacity to carry out some activities on their own. Hence to me stakeholders form a very critical phase within our organization.

[Rituu B Nanda]

Second response from Erick

Erick Ngosia •Ranjani K Murthy, we have 222 constituencies, from each of this constituency, we organized a forum whereby, the constituency members selected a man and a woman to represent them in the Constituency Monitoring Committee that works with the projects in every constituency and also a man and a woman from every local authority. So, for every constituency, there is a man and woman on the ground. We involve this people, together with the projects implementers and other stakeholders in the monitoring process. They visit every project that is being funded by CDF in each of the constituency and record their findings on an activity form that we use on analyzing the results. We again accompany them with the quantity surveyor and building and construction specialist for every constituency to help in finding out if the materials used and the money put in the project is worth

[Rituu B Nanda]

Linkedin response

Dr. Kamran • In any evaluation process ,beneficiary /stakholder involement should be of first priority .This is one of the tangible indicator that can show you the sucess or failure of a project ,if suppose you have set a very good indicators other than stakeholder /beneficiary involement for evaluation of your project ,i dont think so you are able to show the relevency of your project in the impact analysis 

[BIHONEGN TEFERI AYNALEM]

Hi Ranjani!

Marginalized community groups has to be participated not only during evaluation but also on planning. If your project has not participated these groups during planning period, you have to participate them during evaluation even. But if they were not participated during planning and implementation they don't have that much awareness about the evaluated issue, so that they could not contribute much as expected.   

Bihonegn T.

[Ranjani K.Murthy]

BIHONEGN TEFERI AYNALEM said:

Hi Ranjani!

Marginalized community groups has to be participated not only during evaluation but also on planning. If your project has not participated these groups during planning period, you have to participate them during evaluation even. But if they were not participated during planning and implementation they don't have that much awareness about the evaluated issue, so that they could not contribute much as expected.   

Bihonegn T.

[Rituu B Nanda]

Hi Ranjaniji,

I was part of a participatory evaluation project , PAR for two years in India and Cambodia which was funded by IDRC. Here are some of my learnings:

1. Projects which are participatory in nature, the engagement of stakeholders is easier and more likely
2. Earlier the engagement of stakeholders, more is the degree of participation and greater ownership
3. A challenge we faced was that the project team considered evaluation was an external thing and they had no role in it except provide data. We tried a problem based approach in India and strength-based approach in Cambodia for the baseline. During the course of the project we realised that strength-based approach greatly helped us in encouraging stakeholder participation. We used community life competence approach (www.communitylifecompetence.org). Thereafter, we changed to strength-based approach in India as well.

All the best for your presentation in Kathmandu!

 

 

[Rituu B Nanda]

Linkedin  response

Best,
Michael" data-li-editable="false" data-li-edit-sec-left="900" data-li-time="" />

Michael Moore •Dear all, thanks for this. A project design process that is participative for the get go, perhaps using a community planning process (Future Search) that involves communities in setting project objectives for learning, action and transfer of learning to the context strengthens the project design and ownership. However does this necessarily strengthen the reliability and objectivity of the evaluation of impact?
Best,
Michael

[Rituu B Nanda]

Two more responses from Linkedin

 

Hasina RAKOTOZAFIARISOA •First, it is necessary that each stakeholder approves of the vision, after that they align to share responsibilities and their respective contributions. These preparatory steps ensures ACCEPTANCE. During and after implementation evaluation will be participatory manner. However, a specialist should supervise them to ensure quality: choice of indicators, form and presentation ... The evaluation reports demonstrate their efficiency through objectives and especially compared to their contributions. One of the added values ​​of this assessment is the identification of good practice, to formalize its scaling and replication in other localities.

 

Susan Ajambo •Very relevant and interesting topic! I want to comment on the issue of participation especially for the marginalized. Many evaluation projects indicate that the communities in which they work are stakeholders on paper yet they do not involve them the way it ought to be. This is due to challenges arising from either the finders or the mangers of the project on one hand and the marginalized communities on the other hand. First, meaningful participation is very expensive and requires a lot of time yet projects are usually time bound. Secondly. the professional orientation of the designers of the project also matters: if they are from a field of study that considers participation to be important, they surely include it in the design and viceversa. Another challenge comes from the implementors who tend to worship the design documents and follow them to the dot regardless of where the project is heading. Such, cannot think of working within the design to involve the stakeholders. Participation is crucial but quite challenging!

[Rituu B Nanda]

http://www.linkedin.com/miniprofile?vieweeID=228131447&context=...">

Follow Xenia

Xenia Hidalgo-Panameno, MCP-ARCH • The participants sometimes don´t believe in the effectiveness of the programs, that´s why they don´t want to participate in evaluation. But as an incentive it is a method to awake the willingness to participate, plus that their working time could be interrupted, therefore it is needed to pay them

[Rituu B Nanda]

stanley capela

• Right now in NYS there is an emphasis in moving developmental disabilities services to a managed care model. One area that they are focusing on is the use of Council on Quality Leadership Personal Outcome Measures. This process requires interviewing individuals with developmental disabilities regarding 21 outcomes where is the individual achieving outcome and does agency provide supoort. It is a very good approach but resource intense since the interviews take 2 to 3 hours and the involvement of stakeholders who can range from high to low functioning. Reading your comments, I came away that the issues are similar in identifying who is the stakeholder and how do you engage them in the evaluation process.

For additional info go to the Council on Quality Leadership website. Here are the personal outcome measures: Basically you are trying to determine is the individual achieving the outcome and is the program providing support to the individual to achieve the outcome.
CQL Personal Outcome Measures

People are connected to natural supports
People have intimate relationships
People are safe
People have the best possible health
People exercise rights
People are treated fairly
People are free from abuse and neglect
People experience continuity and security
People decide when to share personal information
People choose where and with whom they live
People choose where they work
People use their environments
People live in integrated environments
People interact with members of the community
People perform different social roles
People choose services
People choose personal goals
People realize goals
People participate in the community
People have friends
People are respected

I forgot to mention the CQL POMS are used world wide. The organization actually has an individual that has gone to different parts of the world to train people in places like Canada, New Zealand etc.