E-discussion: Evaluating SDGs with an equity-focused & gender responsive lens (no one left behind)

[Rituu B Nanda]

The purpose of the consultations (18th Jan-18th Feb 2016)

Following the approval of the 17 Sustainable Development Goals (SDGs) by the UN General Assembly and by the international development community in 2015, EvalPartners (including EvalGender+) and United Nations Evaluation Group (UNEG)’s members have begun to form working groups to strengthen monitoring and evaluation systems to assess these different goals.  The purpose of the present consultations, organized by EvalGender+, UNEG and the Independent Evaluation Office (IEO) of UN Women, is to provide guidance to strengthen M&E systems to assess all SDGs with an equity-focused and gender-responsive lens, in addition to Goal 5 (Achieve gender equality and empower all women and girls) and Goal 10 (Reduce inequality within and among countries).  The above mentioned agencies and network, in collaboration with other strategic stakeholders, are planning to prepare a guidance note that will assist both actors directly involved in social equity and gender equality, as well as all actors involved with the evaluation of the SDGs, in ensuring that social equity and gender equality are adequately addressed in all of the SDG evaluations. It is intended to produce a first version of this Guidance note by June 2016.  The SDG strategy is to work through, and to help strengthen existing M&E systems at the national and local levels and consequently the focus of the present consultations is on indicators and approaches that can be implemented through existing M&E structures – many of which may have limited experience and resources to address social equity and gender equality issues.

In addition to their importance as stand-alone sustainable development goals, both of these are cross-cutting themes that must be integrated into the assessment of all of the other goals.  For example, the achievement of Goal 2 (end hunger), Goal 3 (ensure healthy lives and promote well-being for all at all ages), Goal 7 (ensure access to affordable, reliable, sustainable modern energy) and Goal 13 (take urgent action to combat climate change) – to mention only four, all have important gender dimensions that affect the achievement of these goals.  Similarly, there are social equity dimensions to all goals. To read more on SDGs visit the link "Transforming our world:  The 2030 agenda for sustainable development.https://sustainabledevelopment.un.org/post2015/transformingourworld

The consultations identify four important themes that must be addressed in the M&E systems to assess SDGs with an equity-focused and gender-equality lens, in addition to goals 5 and 10:

• Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.
• Theme 2:  Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”
• Theme 3:  Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities
• Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

Each of these themes has different implications in different countries, regions, sectors and according to the type of organization.  Consequently we invite you to share your experiences and perspectives to assist the EvalGender+, UNEG, UN Women IEO teams in ensuring that the Guidance Note will reflect the diversity of experiences and perspectives in different countries, regions and types of organization.

 

Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.

In recent years a number of “new metrics” have evolved which can potentially widen the range of indicators and measures available for the monitoring and evaluation of development results from equity and gender responsive perspectives. These include:

• Data that can now be collected through mobile phones, tablets, internet, GPS mapping and other new information technologies
• Big Data collected from satellites and drones, remote sensors, analysis of twitter and social media, mobile phone records, digital electronic transfers including purchase of mobile-phone air time and ATM withdrawals and crowdsourcing
• Participatory consultations (e.g. Most Significant Change, Outcome Harvesting, PRA)
• Concept mapping
• Mixed methods evaluations and
• Feminist research methods (e.g. oral history, feminist ethnography and content analysis, power relations, social justice and empowerment approaches)

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose)as they relate to equity-focused and gender responsive evaluation.

• In your experience what are some of the limitations of current data collection methods and the kinds of indicators they produce?
• What are the most difficult issues to measure with respect to social equity? and with respect to gender equality?
• What are the new challenges for assessing sustainable social equity and gender equality?
• What have proved some of the most effective methods?
• In addition to those mentioned above, what other new metrics are you familiar with?
• Which of the new metrics show the greatest promise?

Theme 2: Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”

As SDGs are interconnected, national policies and programme to implement them will be complex. As programs grow in size and scope, the number of partners and stakeholders and in terms of the kinds of social and behavioral changes they seek to produce, they become more complex – both in terms of how they are designed and implemented, but also in terms of how they must be evaluated.  Complexity is defined in terms of: (a) the nature of the programme, (b) the number of partners and stakeholders and the patterns of interaction among them (including the level of consensus or disagreement among them on the goals of the programs), (c) the number of external (contextual) factors that influence how the programme is implemented and its outcomes and (d) the complexity of the causal chains through which outcomes are to be achieved.  A number of additional factors are particularly important for the evaluation of social equity and gender equality, including: (i) social and cultural constraints and pressures, (ii) the power relationships and social definition of gender relations and social equity, (iii) multiple influences on processes of behavioral change, (iv) the role of social media, and (v) the long, non-linear causal chains through which changes are produced.

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity-focused and gender responsive evaluation.

• Which dimensions of complexity are most important in your work on social equity and gender equality?
• How does complexity affect our understanding of the effectiveness of different interventions on the production of changes in social equity and gender equality?
• What methods and approaches have you found most effective for understanding the outcomes of complex programs on social equity and gender equality?
• The processes of change are long, involving many actors and contextual factors.  Also the processes are not linear as advances on one front often involve set-backs on others.  What kinds of evaluation strategies have you found most effective in these complex scenarios?
• What are the special challenges for understanding the impacts of different interventions on the most vulnerable populations?  What evaluation methods are most effective for studying these very sensitive processes of change?

Theme 3: Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities

 

The SDGs pose challenges for national evaluation systems as the SDGs require the involvement of a broader range of stakeholders, a broadening of the range of indicators to be measured and the methodological and organizational problems required to assess sustainability which requires collecting data over a much longer period of time.  Many programmes are intended to produce benefits that continue over five or even ten years and the evaluation must (ideally) continue over all of this period.  So instead of conventional evaluations that often only cover the 3-5 years of project implementation, the SDG evaluations may be required to continue for twice as long.  The application of a social equity and gender equality lens will often present additional challenges for national evaluation systems, including the fact that the evaluation of gender outcomes and impacts is often the weakest part of many national evaluation systemsand the methodologies for evaluating social equity are also not well developed in many countries (or in the evaluation literature in general).  Given resource constraints of many evaluation agencies, it will often not be possible to consider specialized evaluations that focus exclusively on equity and gender, and it will be necessary to adapt standard M&E methodologies to address these issues.  It will be important to consider the extent to which some of the multi-shareholder partnerships can bring in agencies with expertise in these areas and with additional resources that may permit the selective application of gender and equity focused data collection and analysis methodologies

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what will be the main challenges that national evaluation systems will face when evaluating social equity and gender equality?
• In the countries with which you are familiar, how well established are the methodologies for evaluating these two areas.
• Which kinds of organization have the most experience in the evaluation of these two areas?  Are these organizations already part of the national evaluation systems?  If not, what will be required to ensure their active involvement?
• What kinds of evaluation capacity development will be required to strengthen the capacity of the national evaluation systems to address these issues?
• What are the example of successful partnership in your country or in your area of work to strengthen M&E systems in general, and equity-focused and gender-responsive systems in particular?
• What are the opportunities and challenges for such partnerships?

Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

 

Experience from all regions and sectors shows that one of the biggest challenges facing evaluation systems is the very low rate of utilization of evaluations.  In many cases evaluation findings do not reach many of the key organizations and groups (including community and women’s organizations), in other cases they are not presented in a form which is easily accessible to some groups, particularly the most vulnerable.  Even when evaluations are reviewed, action is often not taken on many of the recommendations.  These challenges are likely to be even more serious for social equity and gender equality as these themes are less familiar to many organizations and the mechanisms to review and action are often less developed.

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what are the factors affecting the demand for and use of evaluation?
• Are there additional factors affecting the demand and utilization of social equity and gender equality evaluations?
• How could the demand and utilization of these evaluations be increased?
• What types of evaluation presentations would you propose to make evaluation more accessible to stakeholders?

 

 

 

[Rabab Saab]

I am pleased to see the interest in the SDGs and the focus on no person left behind.  I also would like to contribute to this body of work, especially in Theme 1.  Please let me know how those interested are organized so the work can begin.

Rabab Saab

[e-mail]

www.linkedin.com/in/rababsaab

 

[Emmanuel Trepanier]

Hello everyone, 

I agree with Getaneh, simplifying (and streamlining) the indicators and their use for measurement is key if a concerted approach to engendering the evaluation of SDGs is to be developed. There is a huge number of actors and good practices to consider. I look forward to engaging with you online and perhaps in person in New York in March! 

All the best, 

Emmanuel Trépanier

Senior Consultant

Universalia Management Group (Montreal, Canada)

[e-mail] 

[Rituu B Nanda]

I look forward to meeting you Emmanuel. How would you envision streamlining indicators? Indicators set at international level may not be relevant at country level and those set at national level may vary from what is happening at the ground level. I would very much like to learn from your experience. Hope you will add your profile picture soon!

[Rabab Saab]

I would love to be part of the first theme.  I am an M&E specialist with over 17 years of experience.  Developing metrics like indicators and corresponding data collection tools is my specialty.

Here's my information.  I look forward to receive information on how to start.

[e-mail]

[phone]

[Rituu B Nanda]

Dear Rabab,

You can respond here by clicking on the reply button. This is the link. Gender and Evaluation community is hosting the discussion. Let me know if you have any queries.

Warmly,

Rituu

[Yoke Octarina Sudarbo]

I would like to join the second theme. Please, inform me on how to join it. Thank you.

[Edridah]

I would like to join first theme. Thank you

[Liisa Horelli]

I am so sorry that I did not have the chanced to participate in the on-line webinar and now I am a bit confused how to participate in the commenting of these themes. Should we form groups according to the themes or can we just comment the themes in general?

Liisa

[Rituu B Nanda]

Hi Liisa, please comment on different themes here on this wall. We would like to collate the responses at one place. The webinar was kickoff for this discussion and recording would be hopefully be available. Let me know if you have any other query. Warm greetings!

[Isha Wedasinghe Miranda]

No One Left Behind 

Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.

Some Comments on above:

1.1 Explanation

• Qualitative methods involves the collection, analysis, and interpretation of data that are not easily reduced to numbers. These data relate to the various social world and the concepts and behaviors of people within it. In social Integration, peace and conflict, healthcare, education involves complex human interactions as example educational situations demand complex understanding.
• Limited quantifying data will influence the poor quality. Data can interpreted in two commonly known horizontal and vertical interpretation of the information attached to the data. Either way it will limit the outreach due to the time factor in the indicators could play the major role in order have an expected results.

Indicators interpretation i.g. Questions on one of the indicators (example )

• Goal 05. Proposed Indicator 1:

Proportion of ever-partnered women and girls (aged 15-49) subjected to physical and/or sexual violence by a current or former intimate partner, in the last 12 months.  

How well we can correctly get these  data ?

Questions are ?

• Violence after the consent given by the either party during the physical interaction or
• Rape and physical sexual violence against his or hers consents?

 

Concerns and Proof of evidence:

• How and where do you collect these data from? Most cases should be collect it from Legal entities such place of  an institutions ?
• Validity on hidden data’s collection through viable conversations with individuals?

               

1.2 Explanation

Thinking of equity what is that we want to distribute equality?  Few things must take into the considerations

• Data - Quality, reliability, measurability and accountability.
• What are we are measuring and on what basis?
• What challenges that we are talking about?

If it is straight forward intervention which makes every bodies life easy. Such as employment and equality in selection process. Etc. Some we could quantifying easily with a qualitative explanation.   i.g. number of Female and male students enroll in internship at the multinational organization. Easley could take the numbers by age, sex, ethnicity and University enrollments wise etc.   But how ever complexity in the human behavior this sectors could transform in to a challenging complex platform. As below explain in challenges…

 

Challenges:

Individualism

How do you measured individual/s choices? As personal wise, as location wise. In other word My Rights.  Definition is a social theory favoring freedom of action by individuals over collective or state control intervention.

 Primary challenges will be collection of data and measuring  the  on individual choices as well as Risk Attitudes?   

• Data collection and measure Individual Risk Attitudes

Examples

In Health Care . i.g

•  Choice of  medical treatments   Sometimes most terminal ill or critical life threaten diseases ,  patients averting  to go to hospital for treatment thinking it’s  make no sense, even though it is affordable to  him or for her. 
• Choice of methodology: affordability on Treatment through other factors such enrolling better health care financial planning.  i.g Concept of not wanted to have it even they can afford or   incompetent to have it due to financial constrain.   

 

In Education

• In the certain circumstance collection of data and measuring is going to be a night mare in some south Asian countries.  I.g.  If someone decided not to make an effort to continue his or hers education saying I am comfortable with just learning to write, read and understand, will reflect as  morally an individual basis.

Likewise how do you Measure following    three individual choices which will have problem of measuring the equality is ….

• Did not to go to School:  under circumstance (Individual choice)
• Do not want to go to school:   Did but drop off? And why factor. How do you measure the why factor.  This data may not have very clear measurable qualitative or quantitative data however we need to analysis circumstance too.
• Did not get a chance to go to school:  perhaps it can measurable somewhat reasonable way.  Nevertheless it also based on circumstance such as lack of opportunity given to a girl child so on.

Other General Data Collection Measuring  

Challenges:

 Examples. In Health as General

How do you measures data on equality in assessment of diagnoses procedures conducted by an individual or institutional wise.

In Peace interventions

Process is intangibles, invisible in data and measurement is purely a long term our come and impact. 

Countries like Iran, Iraq, Syria or that matter Sri Lanka on Peace Reconciliation,  may not been able to find a room for equity due to prominence  given  on vulnerability suitability, priority,   necessity  as well based on human behavior .

 

• What have proved some of the most effective methods?

Institutionalize    data collection:

In order to eliminate poor and unmeasurable data.  All levels of Government institutions like   Department of Census and Statistics,  Non- Government organizations , academic institutions, Private sectors approach through chamber of commerce activist  institutions in sectors  from Health, educations to Social welfare.

 

New Matrix. Methods: Social Media, ATM, Mobile data collections and blogs and other form of print media. What we are looking for?  Usage in numbers of qualitative information’s on usage purpose by all ages .

Constant changes and fast moving uplifting in systems on above, questions will rise on data validation, perception and interpretation. Challenges will be

1. Data Collecting approach
2. Identifying the measurable and appropriate data
3. Understanding the collection method and matrix
4. Analysis of the data from quantitative to Qualitative
5. Validity 

 

[Rituu B Nanda]

Hi Isha,

Thanks for your detailed response. Just to pick up from the goal 5 indicator you mention on violence. Violence can be verbal and emotional as well. I am just wondering how can we set indicators which are relevant to all contexts and situations and countries and communities. Is it even possible? Just brainstorming and trying to learn from your experience and those of others.

Thanks

Rituu

[Isha Wedasinghe Miranda]

Dear rituu

Most importantly we need evidence based quality data. in that context how do you accounted these unaccounted  data . 

i am placing a another case study:

This was the survey done in later part of year 2014 in eastern part of Sri Lanka. Purpose was for a Government program, given  more focus on quantitative data however the team  priorities to qualitative data too. 

Case study: Evidence base data

As per the area police stations posted in the complaint record book:

22  entries based on GBV and Rape cases.

Data based on 5 Rape Cases:

• 1 out of 5 rape cases convicted at the court of Law
• 3 out of 5 still under the investigation, no identification of the convict ,  no arrest had been taken place. Explanations as per the records:   not enough credibility in the evidence from either side.

• 1 out of 5 (final) not taken action at all and it remained has an entry in the police stations.

Data on 17 GBV

• 5 out of 17   cases as per the police records  they have  transferred  to police family council unit.  No arrest has been taken.
• 6 out of 17 
  • Evidence was produced such as documentations of  hospitalization of the victim,
  • Arrest has made (according to the Police law and order) not taken to the courts.
  • Released on Bail after the strong advice given by the Police  

• 2 cases out of   17 complaint had withdrawn by the victim party.   
• Final 4 unattended by the police without any action 

Which  data relevant to the indicator. Almost every country it is not possible to get evidence based data on these indicators .

Just thought :  5th indicators could have sub indicators or proxy indicator to accommodate this?