E-discussion: Evaluating SDGs with an equity-focused & gender responsive lens (no one left behind)

[Rituu B Nanda]

The purpose of the consultations (18th Jan-18th Feb 2016)

Following the approval of the 17 Sustainable Development Goals (SDGs) by the UN General Assembly and by the international development community in 2015, EvalPartners (including EvalGender+) and United Nations Evaluation Group (UNEG)’s members have begun to form working groups to strengthen monitoring and evaluation systems to assess these different goals.  The purpose of the present consultations, organized by EvalGender+, UNEG and the Independent Evaluation Office (IEO) of UN Women, is to provide guidance to strengthen M&E systems to assess all SDGs with an equity-focused and gender-responsive lens, in addition to Goal 5 (Achieve gender equality and empower all women and girls) and Goal 10 (Reduce inequality within and among countries).  The above mentioned agencies and network, in collaboration with other strategic stakeholders, are planning to prepare a guidance note that will assist both actors directly involved in social equity and gender equality, as well as all actors involved with the evaluation of the SDGs, in ensuring that social equity and gender equality are adequately addressed in all of the SDG evaluations. It is intended to produce a first version of this Guidance note by June 2016.  The SDG strategy is to work through, and to help strengthen existing M&E systems at the national and local levels and consequently the focus of the present consultations is on indicators and approaches that can be implemented through existing M&E structures – many of which may have limited experience and resources to address social equity and gender equality issues.

In addition to their importance as stand-alone sustainable development goals, both of these are cross-cutting themes that must be integrated into the assessment of all of the other goals.  For example, the achievement of Goal 2 (end hunger), Goal 3 (ensure healthy lives and promote well-being for all at all ages), Goal 7 (ensure access to affordable, reliable, sustainable modern energy) and Goal 13 (take urgent action to combat climate change) – to mention only four, all have important gender dimensions that affect the achievement of these goals.  Similarly, there are social equity dimensions to all goals. To read more on SDGs visit the link "Transforming our world:  The 2030 agenda for sustainable development.https://sustainabledevelopment.un.org/post2015/transformingourworld

The consultations identify four important themes that must be addressed in the M&E systems to assess SDGs with an equity-focused and gender-equality lens, in addition to goals 5 and 10:

• Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.
• Theme 2:  Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”
• Theme 3:  Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities
• Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

Each of these themes has different implications in different countries, regions, sectors and according to the type of organization.  Consequently we invite you to share your experiences and perspectives to assist the EvalGender+, UNEG, UN Women IEO teams in ensuring that the Guidance Note will reflect the diversity of experiences and perspectives in different countries, regions and types of organization.

 

Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.

In recent years a number of “new metrics” have evolved which can potentially widen the range of indicators and measures available for the monitoring and evaluation of development results from equity and gender responsive perspectives. These include:

• Data that can now be collected through mobile phones, tablets, internet, GPS mapping and other new information technologies
• Big Data collected from satellites and drones, remote sensors, analysis of twitter and social media, mobile phone records, digital electronic transfers including purchase of mobile-phone air time and ATM withdrawals and crowdsourcing
• Participatory consultations (e.g. Most Significant Change, Outcome Harvesting, PRA)
• Concept mapping
• Mixed methods evaluations and
• Feminist research methods (e.g. oral history, feminist ethnography and content analysis, power relations, social justice and empowerment approaches)

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose)as they relate to equity-focused and gender responsive evaluation.

• In your experience what are some of the limitations of current data collection methods and the kinds of indicators they produce?
• What are the most difficult issues to measure with respect to social equity? and with respect to gender equality?
• What are the new challenges for assessing sustainable social equity and gender equality?
• What have proved some of the most effective methods?
• In addition to those mentioned above, what other new metrics are you familiar with?
• Which of the new metrics show the greatest promise?

Theme 2: Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”

As SDGs are interconnected, national policies and programme to implement them will be complex. As programs grow in size and scope, the number of partners and stakeholders and in terms of the kinds of social and behavioral changes they seek to produce, they become more complex – both in terms of how they are designed and implemented, but also in terms of how they must be evaluated.  Complexity is defined in terms of: (a) the nature of the programme, (b) the number of partners and stakeholders and the patterns of interaction among them (including the level of consensus or disagreement among them on the goals of the programs), (c) the number of external (contextual) factors that influence how the programme is implemented and its outcomes and (d) the complexity of the causal chains through which outcomes are to be achieved.  A number of additional factors are particularly important for the evaluation of social equity and gender equality, including: (i) social and cultural constraints and pressures, (ii) the power relationships and social definition of gender relations and social equity, (iii) multiple influences on processes of behavioral change, (iv) the role of social media, and (v) the long, non-linear causal chains through which changes are produced.

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity-focused and gender responsive evaluation.

• Which dimensions of complexity are most important in your work on social equity and gender equality?
• How does complexity affect our understanding of the effectiveness of different interventions on the production of changes in social equity and gender equality?
• What methods and approaches have you found most effective for understanding the outcomes of complex programs on social equity and gender equality?
• The processes of change are long, involving many actors and contextual factors.  Also the processes are not linear as advances on one front often involve set-backs on others.  What kinds of evaluation strategies have you found most effective in these complex scenarios?
• What are the special challenges for understanding the impacts of different interventions on the most vulnerable populations?  What evaluation methods are most effective for studying these very sensitive processes of change?

Theme 3: Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities

 

The SDGs pose challenges for national evaluation systems as the SDGs require the involvement of a broader range of stakeholders, a broadening of the range of indicators to be measured and the methodological and organizational problems required to assess sustainability which requires collecting data over a much longer period of time.  Many programmes are intended to produce benefits that continue over five or even ten years and the evaluation must (ideally) continue over all of this period.  So instead of conventional evaluations that often only cover the 3-5 years of project implementation, the SDG evaluations may be required to continue for twice as long.  The application of a social equity and gender equality lens will often present additional challenges for national evaluation systems, including the fact that the evaluation of gender outcomes and impacts is often the weakest part of many national evaluation systemsand the methodologies for evaluating social equity are also not well developed in many countries (or in the evaluation literature in general).  Given resource constraints of many evaluation agencies, it will often not be possible to consider specialized evaluations that focus exclusively on equity and gender, and it will be necessary to adapt standard M&E methodologies to address these issues.  It will be important to consider the extent to which some of the multi-shareholder partnerships can bring in agencies with expertise in these areas and with additional resources that may permit the selective application of gender and equity focused data collection and analysis methodologies

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what will be the main challenges that national evaluation systems will face when evaluating social equity and gender equality?
• In the countries with which you are familiar, how well established are the methodologies for evaluating these two areas.
• Which kinds of organization have the most experience in the evaluation of these two areas?  Are these organizations already part of the national evaluation systems?  If not, what will be required to ensure their active involvement?
• What kinds of evaluation capacity development will be required to strengthen the capacity of the national evaluation systems to address these issues?
• What are the example of successful partnership in your country or in your area of work to strengthen M&E systems in general, and equity-focused and gender-responsive systems in particular?
• What are the opportunities and challenges for such partnerships?

Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

 

Experience from all regions and sectors shows that one of the biggest challenges facing evaluation systems is the very low rate of utilization of evaluations.  In many cases evaluation findings do not reach many of the key organizations and groups (including community and women’s organizations), in other cases they are not presented in a form which is easily accessible to some groups, particularly the most vulnerable.  Even when evaluations are reviewed, action is often not taken on many of the recommendations.  These challenges are likely to be even more serious for social equity and gender equality as these themes are less familiar to many organizations and the mechanisms to review and action are often less developed.

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what are the factors affecting the demand for and use of evaluation?
• Are there additional factors affecting the demand and utilization of social equity and gender equality evaluations?
• How could the demand and utilization of these evaluations be increased?
• What types of evaluation presentations would you propose to make evaluation more accessible to stakeholders?

 

 

 

[Tomasa Ravines]

My country Peru, has important equity challenges. Despite it is a middle income country, there are important gaps: between the rural and urban population, because of the ethnicity origin, age, gender, language and others. The Gender Inequality Index in 2013 was 0.433 (INEI Peru: 2013).  We have a lot of work to do in favor of women rights for example there are important problems: illiteracy, low level of women participation in the economically active population, low level of women in political participation, high percentage of violence based gender, poor conditions to facilitate the exercise of sexual and reproductive health rights, etc. Peru did not achieve the DMG 5: the rate of maternal mortality was reduced to 40% but not to 75% which was planned (PAHO: May 2014). The government management of gender public policies has a lot of problems two of them are the lack of evaluation capacities and its low effectiveness for public expenses. 2016 is an electoral year so if we move forward or move back depends on which list will be elected and the pressure of civil society, we will see what happens.

[Liisa Horelli]

Dear all,

I just read all responses to the very interesting discussion on the Multi-stakeholder E-Consultation on the evaluation of SDGs from the equity focused and gender responsive lens. Here are some comments:

Theme 1. The relevance of new metrics is of course very important as the eternal problem with metrics is that there seem to be always too many indicators which in the end means that you cannot "see the wood from the trees". Therefore, the core question is the basic design of the evaluation and the chosen criteria that the indicators shall then measure. As somebody pointed out, it is the task of the change theory to point out some of the core criteria. Do we have enough relevant change theories in this type of evaluation? The problem is also that sex/gender is only to a certain extent a variable that can be measured in quantitative terms. In many cases it is a construct or a process: doing gender + (intersectionality) in context that has to be studied and researched ending in qualitative results concerning patterns of behavour, organizational structures, gender contracts etc. However, I do think that ICTs can help measuring, like clouds, social media, mobile phone data etc. but so far ICTs don't work in many  contexts.

Theme 2: We are living in an increasingly complex world and even everyday life is complex. However, there are different types of complexity: simple, disorganized and organized. Simple problems can be solved through appropriate action and its evalution (such as building a park or playground). Disorganised complexity, for ex. different kind of service systems in a city that overlap and make it difficult to access relevant services, can be solved by partial reduction through statistics and by applying diff. perspectives which can then be evaluated. However, organized complexity (for ex. administratively opposing decisions and implementation leading to chaos) cannot really be solved, only studied through diff. methods like action research, negotiations with m & e.

Anyway, I have found useful the application of Critical Systems Heuristics (CSH) by Reynolds & Williams (2012) Systems thinking and equity-focused evaluations.

Critical Systems Heuristics (CSH) comprehend ´reality` as comprising interrelationships whose boundaries  are interpreted/judged from multiple perspectives.Reality is affected by sources of influence that are: motivation/values; power/resources; knowledge; legitimacy/victims that are measured through 12 CSH questions.

Theme 3: I must admit that I cannot understand the subtitle of this theme, as the National evaluation system comprises the different practices of m & e in the country,  "guided"  by its institutions, eval. culture, perhaps national policy, as well as active individuals, organisations and enterprises. So partnerships is just one part of it. As for the kinky questions in this theme, there are no easy answers for them since there seem to be only few appropriate methods, lacking evaluation capacity and willingness, too little systematic training and scant resources for this type of endeavor.

Theme 4: No easy answers for this one. More capacity building for both producers and users of evaluation. Evaluations should be able to answer the questions more quickly, in a simple language, and the reports should not be too long.  Clear vision of what we want to achieve with gerder focused evaluation is most important, as stated in one of the comments.

[Isha Wedasinghe Miranda]

Dear Liisa 

This is very good analysis . 

[Rituu B Nanda]

Hi Liisa,

Thanks for to taking out time to provide comprehensive response

Theme 1: In my experience too ICTs have limitations in terms of gender and equity focused evaluations. Even in surveys we need good facilitation skills. For instance I am currently working with bonded labourers. A survey question like are you bonded labour yielded negative results in a site. For the baseline we are doing participatory statistics rather than regular survey. Something new!

[Rukmini Panda]

Thanks Lisa for her comprehensive response and thanks Rituu for highlighting the facilittaion skill. Prior to having gender equity focused evaluation system, there is dearth need of creating nationally a pool of people who eventually may multiply to create a wide pool to undertake evaluations. My experience is also same as Rituu's. While undertaking a study on reproductive health issues of migrant workers, asking them directly what are reproductive issues yielded discouraging results. When we think to ask some qualitative question on how women's agencies have been promoted with the interventions. It is really a very big problem to make the facilitators understand the same way you think on women's agencies.

Very happy to be part of this discussion.

Regards,

Rukmini

[Rabab Saab]

It is my pleasure to be part of this group discussing the SDGs.  It is important that we have proper metrics that reflect the values inherent in these goals.  I would like to work on developing some of these metrics along with other evaluators.

[Chi Bemieh Fule]

Theme 4.

In my opinion, the low demand for evaluation in Cameroon is basically due to its misconception. Many politicians mistake it for Auditing to which they may be liable in case they fall short of managing resources efficiently. Sometimes, it is simply due to negligence - in my context most politicians are of the advanced age group (60 years and above) and try to minimise their work load as much as they can, hence lack of objectivity. Consequently, the few results-oriented  programs you find around are typically those with external funding.

[Awuor Ponge]

Dear Chi Bemieh,

I seriously agree with you on this. This is not unique to Cameroun. Everywhere, it is the politicians who are charged with policy formulation that are the greatest enemies of evaluation as they, as you rightly put it, "mistake it for auditing..." This is because they fall short in their oversight role at the implementation of policy.

Now the greatest question we should be asking ourselves is, How do we demystify Evaluation for the political elite? How do we make them see that Evaluation in essence is aiming at helping them achieve impact in whatever they set to do and to measure their levels of success? How do they know what programmes work and which do not work? In summary, how would you go about convincing the 'advanced age group politician' of the need for Evaluation? It is only through this that they will see the need for evaluations and demand for the same... and from this, we shall be able to work on our part to supply the necessary to respond to the demand.

Thank you!

[Pablo Rodriguez Bilella]

Dear Rituu and colleagues, 

near the end of this forum, I would like to make a comment related with Theme 2: Evaluation and Complexity.  I´m really glad that this theme has been included in this forum, as I find it quite important for a well thought evaluation of the SDG.  We know that monitoring and evaluation can be relatively straightforward processes within simple projects or programs, and there are well established procedures and methods that can be applied. However, M&E systems faces other difficulties in order to be designed and implemented at the level of complex organizations and interventions.

One risk to overcome is the thinking that complex situations are unpredictable we cannot develop a theory of change. On the contrary, it is quite important to capture our hypotheses of how we think we can get to the change we want. What we need is a “complexity aware” theory of change! The traditional paradigms of formative and summative evaluations may not quite take into account the complex and emergent nature of social change. While it is possible to build a bridge (and then make a summative evaluation), it is not possible to “create" an  equal society, as equal societies are emergent. And here, other innovative approaches and emergent methodologies could find their place:  developmental evaluation, social network analysis,  human systems dynamics, realist evaluation, process tracing, and qualitative comparative analysis. In my understanding, all this approaches share a concern of going beyond measuring impact, they actually trey to get to the “how” and the “why”, helping us to understand what works in context, highlighting the factors that are helping and hindering success. 

For instance, as these approaches are interested (with their different accents) to yield information that indicates how a particular intervention works (i.e., generative mechanism) and the conditions that are needed for a particular mechanism to work (i.e., specification of contexts) , then they are more likely to be useful to policymakers than other types of evaluation that don´t pay attention to these issues. In sum, these evaluation approaches are interested in asking, “what worked?”, “under what conditions?” and “with what approach?”.

Very close to these ideas, is the topic of monitoring. Monitoring has a privileged place in the context of interventions in complex environments. As I once read, monitoring is a fine art, and requires us to pay attention to story, patterns, finely grained events and simple numbers that are used to measure things rather than to be targets.  

Pablo 

-

----------------------------------------------------------------------------------------------------

Pablo Rodríguez Bilella 

¡¡Visite mi blog: Al Borde Del Caos !! Desarrollo, Evaluación y Complejidad... 

Twitter: @txtPablo  

Facebook, LinkedIn & Google +

----------------------------------------------------------------------------------------------------

[Rituu B Nanda]

Dear Pablo,

I have similar experience with Theory of Change. When the situation is complex and constantly changing it is not possible to determine the different pathways. In my experience Participatory Action Research is a very effective way to reflect and act in complex and dynamic environment. I speak from experience as I am currently doing Systemic participatory action research. See this book 

Burns, D. (2014) ‘Systemic Action Research: Changing Systemic Dynamics to support sustainable change’ in  Action Research Journal

http://arj.sagepub.com/content/12/1/3.full.pdf+html

Best regards,

Rituu

[Dunstan Mubangizi]

I am thrilled and excited to be part of this global platform engaged in the discussion around the integration of SDGs into Governments’ M&E mechanisms!

From an M&E perspective the major problem faced by most developing countries like Uganda is that both information management and decision making is focused on the administrative process of expenditures and activities rather than on the poverty outcomes, impacts and goals that are being pursued. Planning, budgeting and incentives are geared towards tracking inputs, activities and, recently, immediate outputs. Recurrent and development expenditures are reviewed separately, rather than for their combined impact in achieving overall goals. Monitoring and evaluation remain overly centred on compliance with government requirements and regulations rather than end-results of policy, program and project efforts and more particularly tracking results/Impact on social equity and gender equality. Civil servants get rewarded, including appointments to manage multibillion government projects and gain more political praises and popularity for doing paperwork well rather than making a difference in people’s lives. Monitoring and evaluation in Uganda are fragmented, with multiple government and donor planning and progress reporting formats. Policy formulation, work planning and budgeting are undertaken as separate exercises at the sector and district levels. With a proliferation of different funding arrangements, officials are burdened with a large volume of reporting but have little systematic information about effectiveness of actual public service delivery.

[Valerie Karr]

My research focuses on tracking inclusive development and donor agency commitments to inclusion. In terms of the above questions, my feedback relates to recommendations regarding transparent and accountable donor funding portfolios. I would recommend 

Recommendation: Improve collection and coding of data on disability inclusion within official development assistance programs. 

Use the SDGs (including targets and indicators) and develop coding and tracking mechanisms for analyzing inclusion within donor portfolios.