What should be the Global Evaluation Agenda 2016-2020?

[Rashmi Agrawal]

A roundtable consultation was organized on September 26, 2014 by Institute of Applied Manpower Research (IAMR), New Delhi, India in collaboration with national Planning Commission, Government of India to deliberate on the challenge as to how the global evaluation community can contribute to ensuring that evaluations play a key role in planning and implementation of policies and programmes for attaining future sustainable development goals at national, regional and international levels.  The international community has identified four pressing issues in this connection and discussions/ brainstorming sessions are going on to address these important areas. 

The roundtable also focused upon the four issues under question.  The event witnessed representation of various national, international organizations such as UNICEF, UN Women, ISST, CMS, Planning Commission, 3i.e., etc.  The detailed list of participants is annexed.  There were rich deliberations and the summary of the outcomes is mentioned below:

Director-General, IAMR welcomed all the participants and stated that evaluations serve as tools to redesign developmental interventions. While the importance of evaluations is recognized all around, the results of evaluations are often not taken seriously.  He also pointed out that there is a need to develop a culture for evaluation and evidence based policy making at various levels.

Issue 1: What are the most important strategies to ensure governments and parliaments improve policy making and implementation, by demanding and using equity focused and gender responsive evaluation in decision making.

The discussions suggested:

• Need for formulation of a national evaluation policy emphasizing gender based and equity focused evaluations with awareness generation mechanism among legislators and policy makers on importance of gender mainstreaming in evaluation. The evaluation policy should inter alia include the specific gender sensitive indicators to be taken into account while designing interventions and evaluations. The policy should be flexible and able to adapt itself with varying situations
• The policy should be simple but focused upon robust design which looks at both short term and long term impacts.  A high powered committee should be set up at various levels for vetting project designs and evaluation designs to ensure gender sensitivity.
• The policy should emphasize ethics in evaluation and may provide for a High Powered Committee for evaluation designs with representatives from various stakeholders or Thematic Resource Groups
• The national policy should be followed up by operational guidelines for evaluation that incorporate equity focused and gender responsive evaluation criteria. These criteria should be based on five basic aspects of evaluation that is efficiency, effectiveness, relevance, impact and sustainability.
• International agencies to evolve a code of conduct of evaluations that emphasizes gender and equity and influence local policy makers to work in conformity with such a goal. This can take the form of a ‘Convention’ or ‘Recommendation’ that can exert pressure on national governments.
• Post evaluation action taken report by implementing agencies on the basis of the evaluations pointers should be made mandatory and shared with the policy planners, evaluation community as well as the public.
• It is also important to integrate evaluations in the planning process itself.

Issue 2: What are the most important strategies to ensure that Civil Society Organizations in general and Voluntary Organizations for Professional Evaluation in particular, have stronger institutional capacities to contribute to equity-focused and gender-responsive national evaluation systems?

• Evaluation policy should stress upon mapping, establishing and promoting capacity building institutions and evaluators’ competence.
• Encourage formation of national evaluation societies and their interaction with regional and professional evaluation organizations for knowledge and product sharing.
• Promote e-learning on the theory and applications of gender responsive and equity focused evaluations.
• Involvement of academia is extremely important and encouragement to universities to start special courses on monitoring and evaluation is necessary.
• Equity and equality relate to change of behaviours and mind sets. To incorporate these aspects in evaluations, it is necessary to evolve new techniques and methodology that could measure behaviours in more authentic manner. The new techniques can be tried and change in behaviours need to be mapped. Such success stories can be replicated
• Evaluation is also confronted with changing developmental issues like rights, sustainability, environmental issues and so on which involve complex methodological issues. There should be an appropriate mix of qualitative and qualitative methods to deal with such issues. The capacity building programmes should therefore need to be continuously modified to impart such changing scenario.

Issue 3: What are the three most important strategies to ensure that individual evaluators have the capability to produce good quality, context-relevant, equity-focused and gender-responsive evaluations?

• Evolve a set of national competencies in tune with international frameworks established by IDEAS or adapt such internationally established competencies frameworks. There should be a accreditation and certification system to ensure availability of  competent professional evaluators
• Strengthen government and private agencies involved in building capacity on evaluations to build a growing cadre of professional evaluators who are sound in focusing in social equity and gender equality.
• Evaluators should upgrade their knowledge with latest tools and techniques through re-training programmes, workshops or knowledge sharing at various forums. An e-Library should be developed to serve as a knowledge and resource centre accessible to all professional evaluators to facilitate this process of knowledge acquisition. E- library should be enriched by contributions from all professional evaluators and experiences of various stakeholders. Some volunteers can form a core group for this purpose.
• There are some resources on gender and equity focused evaluations like gender sensitive indicators developed by The World bank. There is a international community of practice on gender and evaluations which is trying bring together experiences and resources on gender. Such resources should be accessed/ publicized and countries can adapt them as per their local needs.

Issue 4: How to ensure that enabling environment, institutional capacities and individual capabilities will mutually reinforce each other? And how to ensure that very diverse multi-stakeholders work in partnership based on their own value added and comparative advantages?

• Mutual reinforcement of enabling environment and institutional and individual capacities is the crucial element.  In a way, the presence of these three elements itself may catalyse their working together.  Stakeholders need to be convinced that their individual/ group interests also get promoted in the short or long term by working for the causes of social equity and human rights.
• Strategy to work together has to evolve and for this purpose there is a need to have a common language for better understanding.
• Evaluation instruments/protocols/findings may be placed in public domain, with proper and prominent acknowledgement of source of support, so that individuals and institutions feel that it is an opportunity to being recognized by those who refer to the document (i.e. marketing of their skills/professional expertise).

• Participatory approach towards evaluations has strongly been emphasized during deliberations. Community participation including women should be a part of evaluations from the very start

• The need for a collaborative effort of creating repository/tool kits/webinars stories on evaluations conducted in India and abroad is needed and cases should be pointed out when evaluations made a significant measurable impact of programs.
• A gender specialist can be involved in evaluations to provide gender focus.

Some other Issues that emerged during deliberations (Indian context):

 At national level evaluations serve in helping redesign programs. Evaluations may also be used as assessments. Many flagship programs of the government in India with a major share of public spending are functioning ad-hoc. A template for in-house study has been evolved. There is a need to deal with issues of efficiency in allocation. Program Evaluation Office in national Planning Commission has included flexibility in guidelines, flexi-funds for states’ developmental interventions. For using such funds there is a   need for states involvement in evaluation at grass-roots level, modification of programs and plans and making them better instead of universally applicable programs in the country.

Planning is exercise in resource allocation and evaluation gives information and statistical evidences so that implementation is done in right manner, process lacuna are identified and corrective measures taken. There is a need for building comprehensive chain of evidences, information to fill in critical gaps to be utilized for improvements in future implementation, evaluation to build strategies and learning, sustaining these activities is important, methodological changes, utilization of in-house evaluation, innovation, professionalization and improved accountability. It was further pointed out that evaluation findings are political in nature. There is a need for an overarching evaluation process and though some states have evaluation units - there should be close linkages between state and central evaluation offices.

Country is doing a lot for gender mainstreaming but still it is a daunting challenge. Adding value to evaluation is another issue that needs attention for its better utilisation. Integration of various aspects such as professional evaluators’ engagement with CSOs, ensuring first principles of evaluation (if we don’t count properly, we can’t measure properly), gender-responsive thinking,  rights based approach and a need to work together (strategy for engaging with policy makers) are extremely important. Professionals need to work together with gender specialists.

Governments often think about audit as evaluation - the distinction between the two needs to be made clear, terms like equity and gender responsiveness need to be simplified and universalized. It was pointed out that audit is management of resources and is distinct from performance management, sustainability and evaluation (process, impact).

The purpose of evaluations also depends upon questions being asked, theory of change and causal connections/impact evaluation/answer more questions/how results achieved and why, differential impacts, intended and unintended impacts. It was pointed out that there is a necessity to sensitize government efforts for right action within government and evaluations can aid this process.

Within NGOs the findings of evaluations and ownership of programs by marginalized populations has complex dimensions. The involvement of beneficiaries in evaluation can increase enthusiasm, community participation, FGDs, stakeholders, enable transformation, and create an environment for people to come forward and share. Creation of   measurable indicators and outcomes, and ensure more focus on behavioural change tools should be paid attention. Participants shared through examples that evaluations where stakeholders are involved from the beginning the likelihood of use of evaluations is higher. For instance when evaluations are conducted by Central government, engaging not only the state officers but also district officers is critical as they know what is happening in the field. It was also noted that when communities are involved in the interventions and evaluations, the quality of implementation improves as communities take lead in the programmes affecting them.

 There is a need to ensure equity focused evaluations, understand reasons behind evaluation, develop protocols on emerging issues, code of standardizing ethics, evaluation of programs, long term and short term impact, and gender responsiveness with efforts in capacity building.

 There is a lot of anxiety and fear around evaluations because external evaluators are seen as experts who come from outside to judge the intervention. if evaluation  is a ‘self-improvement and learning exercise’ those affected by the evaluation will be more involved in the evaluation process and take greater ownership of evaluation findings.

Participants recognized that programmes have differential impact on women and other marginalized community members because there are differences in vulnerabilities of different population groups. Therefore gender and equity-lens are critical both in the programmes as well as evaluations.

All agreed that while sensitisation around gender and equity focused evaluations is important one also needs to change how evaluations are perceived. More over use of jargons in evaluation can be a stumbling block in understanding of evaluation concepts and engagement of key stakeholders in evaluations which adversely affects the use of evaluation findings.

National Agenda

• There should be a push for a national plan/policy for evaluation.
• There should be a resource mapping with regard to capacity development in evaluation.
• Evaluations are to be mainstreamed in the national planning with focus on social inclusion and gender equality.
• A repository how the behavior change takes place and what has worked should be developed in the form of successful case studies.
• Events should be organized on evaluations by involving university community.

          India is planning to work upon on some of the issues mentioned above during a period of one year.

List of Participants

 

1. Dr. Pawan Agarwal,  Planning Commission
2. Dr. P. K. Anand,  Planning Commission
3. Ms. Ratna A. jena, Planning Commission
4. Alok Shrivastava, CMS
5. Rituu B. Nanda, Institute of Social Study Trust
6. B. Mishra, UNICEF
7. Ms.Misaki Akaska Ueda UNICEF
8. Ms.Beryl Leach, 3ie
9. Ms. Suhela Khan, UN Women
10. Nabesh Bohidar, Care India
11. Dr. Rashmi Agrawal, IAMR
12. Dr. G. P. Joshi, IAMR
13. Dr. Shachi Joshi, IAMR
14. Dr. Sanchita Bhattacharya, IAMR
15. Neha Kumra, IAMR
16. Dr. Ruby Dhar, IAMR
17. Dr. P.C. Parida, IAMR

[Helen Bash-Taqi]

Many thanks for sharing the report

[Asela Kalugampitiya]

Dear Rashmi,

Many thanks for sharing this important info. Great that you led this meeting. The global consultation for the evaluation agenda is available online if anyone wants to contribute to that. Please find the link. https://www.linkedin.com/groups/EvalPartners-4292632