E-discussion: Evaluating SDGs with an equity-focused & gender responsive lens (no one left behind)

[Rituu B Nanda]

The purpose of the consultations (18th Jan-18th Feb 2016)

Following the approval of the 17 Sustainable Development Goals (SDGs) by the UN General Assembly and by the international development community in 2015, EvalPartners (including EvalGender+) and United Nations Evaluation Group (UNEG)’s members have begun to form working groups to strengthen monitoring and evaluation systems to assess these different goals.  The purpose of the present consultations, organized by EvalGender+, UNEG and the Independent Evaluation Office (IEO) of UN Women, is to provide guidance to strengthen M&E systems to assess all SDGs with an equity-focused and gender-responsive lens, in addition to Goal 5 (Achieve gender equality and empower all women and girls) and Goal 10 (Reduce inequality within and among countries).  The above mentioned agencies and network, in collaboration with other strategic stakeholders, are planning to prepare a guidance note that will assist both actors directly involved in social equity and gender equality, as well as all actors involved with the evaluation of the SDGs, in ensuring that social equity and gender equality are adequately addressed in all of the SDG evaluations. It is intended to produce a first version of this Guidance note by June 2016.  The SDG strategy is to work through, and to help strengthen existing M&E systems at the national and local levels and consequently the focus of the present consultations is on indicators and approaches that can be implemented through existing M&E structures – many of which may have limited experience and resources to address social equity and gender equality issues.

In addition to their importance as stand-alone sustainable development goals, both of these are cross-cutting themes that must be integrated into the assessment of all of the other goals.  For example, the achievement of Goal 2 (end hunger), Goal 3 (ensure healthy lives and promote well-being for all at all ages), Goal 7 (ensure access to affordable, reliable, sustainable modern energy) and Goal 13 (take urgent action to combat climate change) – to mention only four, all have important gender dimensions that affect the achievement of these goals.  Similarly, there are social equity dimensions to all goals. To read more on SDGs visit the link "Transforming our world:  The 2030 agenda for sustainable development.https://sustainabledevelopment.un.org/post2015/transformingourworld

The consultations identify four important themes that must be addressed in the M&E systems to assess SDGs with an equity-focused and gender-equality lens, in addition to goals 5 and 10:

• Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.
• Theme 2:  Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”
• Theme 3:  Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities
• Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

Each of these themes has different implications in different countries, regions, sectors and according to the type of organization.  Consequently we invite you to share your experiences and perspectives to assist the EvalGender+, UNEG, UN Women IEO teams in ensuring that the Guidance Note will reflect the diversity of experiences and perspectives in different countries, regions and types of organization.

 

Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.

In recent years a number of “new metrics” have evolved which can potentially widen the range of indicators and measures available for the monitoring and evaluation of development results from equity and gender responsive perspectives. These include:

• Data that can now be collected through mobile phones, tablets, internet, GPS mapping and other new information technologies
• Big Data collected from satellites and drones, remote sensors, analysis of twitter and social media, mobile phone records, digital electronic transfers including purchase of mobile-phone air time and ATM withdrawals and crowdsourcing
• Participatory consultations (e.g. Most Significant Change, Outcome Harvesting, PRA)
• Concept mapping
• Mixed methods evaluations and
• Feminist research methods (e.g. oral history, feminist ethnography and content analysis, power relations, social justice and empowerment approaches)

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose)as they relate to equity-focused and gender responsive evaluation.

• In your experience what are some of the limitations of current data collection methods and the kinds of indicators they produce?
• What are the most difficult issues to measure with respect to social equity? and with respect to gender equality?
• What are the new challenges for assessing sustainable social equity and gender equality?
• What have proved some of the most effective methods?
• In addition to those mentioned above, what other new metrics are you familiar with?
• Which of the new metrics show the greatest promise?

Theme 2: Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”

As SDGs are interconnected, national policies and programme to implement them will be complex. As programs grow in size and scope, the number of partners and stakeholders and in terms of the kinds of social and behavioral changes they seek to produce, they become more complex – both in terms of how they are designed and implemented, but also in terms of how they must be evaluated.  Complexity is defined in terms of: (a) the nature of the programme, (b) the number of partners and stakeholders and the patterns of interaction among them (including the level of consensus or disagreement among them on the goals of the programs), (c) the number of external (contextual) factors that influence how the programme is implemented and its outcomes and (d) the complexity of the causal chains through which outcomes are to be achieved.  A number of additional factors are particularly important for the evaluation of social equity and gender equality, including: (i) social and cultural constraints and pressures, (ii) the power relationships and social definition of gender relations and social equity, (iii) multiple influences on processes of behavioral change, (iv) the role of social media, and (v) the long, non-linear causal chains through which changes are produced.

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity-focused and gender responsive evaluation.

• Which dimensions of complexity are most important in your work on social equity and gender equality?
• How does complexity affect our understanding of the effectiveness of different interventions on the production of changes in social equity and gender equality?
• What methods and approaches have you found most effective for understanding the outcomes of complex programs on social equity and gender equality?
• The processes of change are long, involving many actors and contextual factors.  Also the processes are not linear as advances on one front often involve set-backs on others.  What kinds of evaluation strategies have you found most effective in these complex scenarios?
• What are the special challenges for understanding the impacts of different interventions on the most vulnerable populations?  What evaluation methods are most effective for studying these very sensitive processes of change?

Theme 3: Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities

 

The SDGs pose challenges for national evaluation systems as the SDGs require the involvement of a broader range of stakeholders, a broadening of the range of indicators to be measured and the methodological and organizational problems required to assess sustainability which requires collecting data over a much longer period of time.  Many programmes are intended to produce benefits that continue over five or even ten years and the evaluation must (ideally) continue over all of this period.  So instead of conventional evaluations that often only cover the 3-5 years of project implementation, the SDG evaluations may be required to continue for twice as long.  The application of a social equity and gender equality lens will often present additional challenges for national evaluation systems, including the fact that the evaluation of gender outcomes and impacts is often the weakest part of many national evaluation systemsand the methodologies for evaluating social equity are also not well developed in many countries (or in the evaluation literature in general).  Given resource constraints of many evaluation agencies, it will often not be possible to consider specialized evaluations that focus exclusively on equity and gender, and it will be necessary to adapt standard M&E methodologies to address these issues.  It will be important to consider the extent to which some of the multi-shareholder partnerships can bring in agencies with expertise in these areas and with additional resources that may permit the selective application of gender and equity focused data collection and analysis methodologies

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what will be the main challenges that national evaluation systems will face when evaluating social equity and gender equality?
• In the countries with which you are familiar, how well established are the methodologies for evaluating these two areas.
• Which kinds of organization have the most experience in the evaluation of these two areas?  Are these organizations already part of the national evaluation systems?  If not, what will be required to ensure their active involvement?
• What kinds of evaluation capacity development will be required to strengthen the capacity of the national evaluation systems to address these issues?
• What are the example of successful partnership in your country or in your area of work to strengthen M&E systems in general, and equity-focused and gender-responsive systems in particular?
• What are the opportunities and challenges for such partnerships?

Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

 

Experience from all regions and sectors shows that one of the biggest challenges facing evaluation systems is the very low rate of utilization of evaluations.  In many cases evaluation findings do not reach many of the key organizations and groups (including community and women’s organizations), in other cases they are not presented in a form which is easily accessible to some groups, particularly the most vulnerable.  Even when evaluations are reviewed, action is often not taken on many of the recommendations.  These challenges are likely to be even more serious for social equity and gender equality as these themes are less familiar to many organizations and the mechanisms to review and action are often less developed.

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what are the factors affecting the demand for and use of evaluation?
• Are there additional factors affecting the demand and utilization of social equity and gender equality evaluations?
• How could the demand and utilization of these evaluations be increased?
• What types of evaluation presentations would you propose to make evaluation more accessible to stakeholders?

 

 

 

[Isha Wedasinghe Miranda]

No one left behind. 

Missing Goal in Goal 3? GOAL 3 : Good Health and Well being.

What is missing  Goal is : Mother Child wellbeing and Child Plus

Good health and well being calculates Just under two  indicators on Mother's Mortality, (childbirth related deaths  etc) under Goal 3.1 and 3.2 .

What is missing in the  indicators as well as should have a  indicator is  

1.Mother Mortality deaths (in other forms of deaths, Ex. HIV AIDS, etc.) which has included as a overall indicators of 3.3 on wards. 

2. Nutritional level 

3. Child abuses  from 5 years to 14 years. 

4. Well being in refugees and IDP camps etc. 

My proposal to the team lets have separate goal for as above Mother Child wellbeing and Child Plus. 

if not this issues may not given  prominent place in the  world . 

[Francis Mwaijande]

Theme 3: Towards equity-focused and gender responsive national evaluation systems- Multistakeholder partnerships to strengthen national evaluation systems.

1. The main challenge I foresee in Tanzania, is the weak National Evaluation System that is not supported by policy. The M&E is not well developed although it appears in the Organogram. At the National level there are structures but down the stream of Local Government Authorities (LGAs) and villages the system is not operational. The MKUKUTA for example is well developed to monitor the SDGs, however downstream tracking and reporting is weak. Capacity building of the structures and knowledge in the LGAs and villages shall be highly needed for SDGs.

2. Equity focused gender responsive evaluation is relatively new. I am not sure if the traditional evaluation tools and mechanisms can be applicable. It needs also a capacity building to all stakeholders to build a common understanding on the concept, indicators, and tools of analysis. In many ways, possibly it only the UN agencies M&E that have the knowledge and capacity on Equity focused and gender responsive.

[Basundhara Bhattarai]

Theme 3: Towards equity-focused and gender responsive national evaluation systems- Multistakeholder partnerships to strengthen national evaluation systems. 

In my experience of over 25 years in Nepal and elsewhere, making national evaluation system more equitable and gender responsive has always been a challenge. For more than a decade, gender inequity and social exclusion are already high in state agenda including new constitution making, overall development plans and all sectoral development plans. For instance, when I take the example of disaster recovery response strategies and planning processes, which I am currently working on, it seems so weak in putting  gender responsive evaluation system in place. In most of the cases, women's number is counted as gender responsive and this is not enough.  Even with a highly gender responsive policy document, gender is almost no where in the implementation processes to be followed. And there is no such (monitoring?) mechanisms that cross-checks discrepancies between policies/programs and actual implementation. Institutionalization of the concept is far behind. 

[Rituu B Nanda]

Also Basundhara, what about transgenders in the context of disaster? I know someone who conducted a study on this some years ago in Nepal. I am very happy that we are getting an opportunity to learn from you. Many thanks!

[Grace Okonji]

 

 

Contribution to theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development.

This is a very useful topic. From my experience of working for an international organization in the past, there are some reasons that I have reflected upon that may contribute to the factors affecting the demand for and use of evaluation.

(i)                  Evaluation is often a requirement, for example, that projects with a budget of more than USD100,000.00 must be evaluated. Over and above this, all donors required evaluation of the projects that benefited from their funds.  While this requirement is good, it ended up being a “tick the box” as completed evaluations followed by a management response. That was almost the end of it.  Perhaps there is need to ‘evaluate’ the management responses that were carried out.

(ii)                In relation to (i) there is no strong convergence between users of evaluation and ownership of the evaluation. An evaluation rarely serves the needs of one stakeholder. Nevertheless, most evaluation reports are written with just one stakeholder in mind such as the donor.  In many aspects the expectation is that Governments need to own the evaluation process but in reality the Government sits on the other side as the one being evaluated and in most cases defending itself.  I do not recall Government strongly disseminating recommendations of evaluations. Contrary to this, they do take seriously the recommendations of audit.

(iii)               The demand for and use of evaluations is closely determined by “when is the best time to disseminate?” In my opinion there needs to be close linkage between evaluation recommendations, planning and monitoring processes. The M&E teams/Programme teams need to be always upfront in synthesizing lessons from evaluations from many other countries that can inform the next relevant programme plans. In many planning documents that I ever became party to, it was also a ‘tick the box’ affair if you have taken into account past evaluations in the process of developing the programme. It would be strategic for each M&E team/programme team to bring to the table evaluation findings from their own organization and other organisations on how the findings and recommendations will inform their programme design.

(iv)              One factor affecting the demand and utilization of social equity and gender equality evaluations is the limited dissemination of evaluation findings and lessons. Just last year I completed an excellent equity focused and gender responsive evaluation for an organization in Malawi, a year later I have not seen it posted in the relevant websites for that organization. To the best of my knowledge many countries can learn from lessons learned if such reports are made accessible.

(v)                There seems to be failure to have a dissemination and marketing strategy that is appropriate to specific stakeholders. To a large extent this relates to “who is charged with the responsibility to disseminate.” Many organisations have communications teams,   but I did not have experience of these teams being charged with responsibility for disseminating evaluation reports. On the other hand evaluation users are not aware of excellent websites where evaluation findings are posted, such as the UN Women, gender equality evaluation portal; the UNDP evaluation resource Centre etc. Of course these websites are only appropriate to limited users who can access internet so appropriate media for dissemination is critical. In addition, allocation of resources for evaluation seems to end with production of report so limited resources are available for dissemination.

(vi)              How the demand and utilization of these evaluations be increased: In addition to taking into account the weak points listed above, there is need for meaningful stakeholder engagement throughout the evaluation process so as to increase ownership, help determine the intended use of the evaluation findings, review interim findings and contribute to recommendations. Experience has shown that people are more inclined to support initiatives that they participated in. Someone would want to at least scan the final report as they played some part in its development.

(vii)             What types of evaluation presentations would you propose to make evaluation more accessible to stakeholders: Presentation is difficult to determine because each organization comes with its format. In my opinion, the converging point that might be useful is an action-oriented report that relates to each of the stakeholders and suits the interests of a particular target audience. For example, women would be more interested in how the distance to the water-point changed their lives while men in a pastoral community would be more interested on how their livestock benefited and the government will be interested on what additional policies to put in place.  

 

 

 

 

[Rituu B Nanda]

Its great to learn from you Grace especially on dissemination of evaluation reports. I also like your point about stakeholders owning the evaluation.

1.Would you have an experience of a gender and equity focused evaluation you conducted where stakeholders owned the evaluation?

2. How were you able to convince the government of Malawi (your response number iv) for gender and equity focused evaluation?

Looking forward to hearing from you. Thank you

[Florence Etta]

Theme 3: Towards equity-focused and gender responsive national evaluation systems- Multistakeholder partnerships to strengthen national evaluation systems. 

Thanks for all the great contributions so far very illuminating!

I particularly like the conversation around indicators. Thanks Isha. 

I would love to ask if contributors on this list could kindly share the current indicators that have been concluded especially for SDGs 5 & 10 as the basis for more useful and perhaps targeted conversations. 

I wonder if it would also be useful  for each person to indicate the question for which they are providing an answer/s if at all possible?

Please forgive me for asking I think it might be useful for organising my own thinking & learning that is why.  

[Isha Wedasinghe Miranda]

My Dear Florence 

I agreed with you. we can help Rituu on this too. at the end of the day we have to concentrate on question around the  indicators which is line with themes . 

Dear Rituu

Please think about it how we can plugin all these  information  according to the themes and indicators.

 

[Rituu B Nanda]

  To read more on SDGs and indicators visit the link "Transforming our world:  The 2030 agenda for sustainable development. https://sustainabledevelopment.un.org/post2015/transformingourworld

[Rituu B Nanda]

Dear Grace,

I have learned so much from you. I was thinking what about the communities? We collect data but do we go back and share the findings with them? What is your experience? Asante Sana

[Florence Etta]

My dear Rituu, It seems to me that much as we want this discussion to be free flowing and perhaps non constricting, you or whoever it might be may need to give us - the reading and contributing public - some minimal guidelines as to how we might proceed to make it easier to contribute as well as follow each and every contribution.

The guidelines might for instance be in respect of the following;

Length; how long can each post be taking into consideration the typical expectations of discussion list netiquette?

Subject line: Might it be useful for each post for example to identify which theme 1,2,3, or 4 that the post is about and which question within that theme it is answering?

These are just a few thoughts.

Do forgive me for these suggestions if they are not useful for whatever reason please simply ignore them. 

[Rituu B Nanda]

Dear Preeti,

you can answer any or all themes which you would like to. All four themes have been posted so that members can post on whatever they have experience or inputs on. I very much look forward to your response.

Thanks

Rituu