E-discussion: Evaluating SDGs with an equity-focused & gender responsive lens (no one left behind)

[Rituu B Nanda]

The purpose of the consultations (18th Jan-18th Feb 2016)

Following the approval of the 17 Sustainable Development Goals (SDGs) by the UN General Assembly and by the international development community in 2015, EvalPartners (including EvalGender+) and United Nations Evaluation Group (UNEG)’s members have begun to form working groups to strengthen monitoring and evaluation systems to assess these different goals.  The purpose of the present consultations, organized by EvalGender+, UNEG and the Independent Evaluation Office (IEO) of UN Women, is to provide guidance to strengthen M&E systems to assess all SDGs with an equity-focused and gender-responsive lens, in addition to Goal 5 (Achieve gender equality and empower all women and girls) and Goal 10 (Reduce inequality within and among countries).  The above mentioned agencies and network, in collaboration with other strategic stakeholders, are planning to prepare a guidance note that will assist both actors directly involved in social equity and gender equality, as well as all actors involved with the evaluation of the SDGs, in ensuring that social equity and gender equality are adequately addressed in all of the SDG evaluations. It is intended to produce a first version of this Guidance note by June 2016.  The SDG strategy is to work through, and to help strengthen existing M&E systems at the national and local levels and consequently the focus of the present consultations is on indicators and approaches that can be implemented through existing M&E structures – many of which may have limited experience and resources to address social equity and gender equality issues.

In addition to their importance as stand-alone sustainable development goals, both of these are cross-cutting themes that must be integrated into the assessment of all of the other goals.  For example, the achievement of Goal 2 (end hunger), Goal 3 (ensure healthy lives and promote well-being for all at all ages), Goal 7 (ensure access to affordable, reliable, sustainable modern energy) and Goal 13 (take urgent action to combat climate change) – to mention only four, all have important gender dimensions that affect the achievement of these goals.  Similarly, there are social equity dimensions to all goals. To read more on SDGs visit the link "Transforming our world:  The 2030 agenda for sustainable development.https://sustainabledevelopment.un.org/post2015/transformingourworld

The consultations identify four important themes that must be addressed in the M&E systems to assess SDGs with an equity-focused and gender-equality lens, in addition to goals 5 and 10:

• Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.
• Theme 2:  Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”
• Theme 3:  Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities
• Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

Each of these themes has different implications in different countries, regions, sectors and according to the type of organization.  Consequently we invite you to share your experiences and perspectives to assist the EvalGender+, UNEG, UN Women IEO teams in ensuring that the Guidance Note will reflect the diversity of experiences and perspectives in different countries, regions and types of organization.

 

Theme 1:  The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective.

In recent years a number of “new metrics” have evolved which can potentially widen the range of indicators and measures available for the monitoring and evaluation of development results from equity and gender responsive perspectives. These include:

• Data that can now be collected through mobile phones, tablets, internet, GPS mapping and other new information technologies
• Big Data collected from satellites and drones, remote sensors, analysis of twitter and social media, mobile phone records, digital electronic transfers including purchase of mobile-phone air time and ATM withdrawals and crowdsourcing
• Participatory consultations (e.g. Most Significant Change, Outcome Harvesting, PRA)
• Concept mapping
• Mixed methods evaluations and
• Feminist research methods (e.g. oral history, feminist ethnography and content analysis, power relations, social justice and empowerment approaches)

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose)as they relate to equity-focused and gender responsive evaluation.

• In your experience what are some of the limitations of current data collection methods and the kinds of indicators they produce?
• What are the most difficult issues to measure with respect to social equity? and with respect to gender equality?
• What are the new challenges for assessing sustainable social equity and gender equality?
• What have proved some of the most effective methods?
• In addition to those mentioned above, what other new metrics are you familiar with?
• Which of the new metrics show the greatest promise?

Theme 2: Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”

As SDGs are interconnected, national policies and programme to implement them will be complex. As programs grow in size and scope, the number of partners and stakeholders and in terms of the kinds of social and behavioral changes they seek to produce, they become more complex – both in terms of how they are designed and implemented, but also in terms of how they must be evaluated.  Complexity is defined in terms of: (a) the nature of the programme, (b) the number of partners and stakeholders and the patterns of interaction among them (including the level of consensus or disagreement among them on the goals of the programs), (c) the number of external (contextual) factors that influence how the programme is implemented and its outcomes and (d) the complexity of the causal chains through which outcomes are to be achieved.  A number of additional factors are particularly important for the evaluation of social equity and gender equality, including: (i) social and cultural constraints and pressures, (ii) the power relationships and social definition of gender relations and social equity, (iii) multiple influences on processes of behavioral change, (iv) the role of social media, and (v) the long, non-linear causal chains through which changes are produced.

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity-focused and gender responsive evaluation.

• Which dimensions of complexity are most important in your work on social equity and gender equality?
• How does complexity affect our understanding of the effectiveness of different interventions on the production of changes in social equity and gender equality?
• What methods and approaches have you found most effective for understanding the outcomes of complex programs on social equity and gender equality?
• The processes of change are long, involving many actors and contextual factors.  Also the processes are not linear as advances on one front often involve set-backs on others.  What kinds of evaluation strategies have you found most effective in these complex scenarios?
• What are the special challenges for understanding the impacts of different interventions on the most vulnerable populations?  What evaluation methods are most effective for studying these very sensitive processes of change?

Theme 3: Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities

 

The SDGs pose challenges for national evaluation systems as the SDGs require the involvement of a broader range of stakeholders, a broadening of the range of indicators to be measured and the methodological and organizational problems required to assess sustainability which requires collecting data over a much longer period of time.  Many programmes are intended to produce benefits that continue over five or even ten years and the evaluation must (ideally) continue over all of this period.  So instead of conventional evaluations that often only cover the 3-5 years of project implementation, the SDG evaluations may be required to continue for twice as long.  The application of a social equity and gender equality lens will often present additional challenges for national evaluation systems, including the fact that the evaluation of gender outcomes and impacts is often the weakest part of many national evaluation systemsand the methodologies for evaluating social equity are also not well developed in many countries (or in the evaluation literature in general).  Given resource constraints of many evaluation agencies, it will often not be possible to consider specialized evaluations that focus exclusively on equity and gender, and it will be necessary to adapt standard M&E methodologies to address these issues.  It will be important to consider the extent to which some of the multi-shareholder partnerships can bring in agencies with expertise in these areas and with additional resources that may permit the selective application of gender and equity focused data collection and analysis methodologies

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what will be the main challenges that national evaluation systems will face when evaluating social equity and gender equality?
• In the countries with which you are familiar, how well established are the methodologies for evaluating these two areas.
• Which kinds of organization have the most experience in the evaluation of these two areas?  Are these organizations already part of the national evaluation systems?  If not, what will be required to ensure their active involvement?
• What kinds of evaluation capacity development will be required to strengthen the capacity of the national evaluation systems to address these issues?
• What are the example of successful partnership in your country or in your area of work to strengthen M&E systems in general, and equity-focused and gender-responsive systems in particular?
• What are the opportunities and challenges for such partnerships?

Theme 4:  Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

 

Experience from all regions and sectors shows that one of the biggest challenges facing evaluation systems is the very low rate of utilization of evaluations.  In many cases evaluation findings do not reach many of the key organizations and groups (including community and women’s organizations), in other cases they are not presented in a form which is easily accessible to some groups, particularly the most vulnerable.  Even when evaluations are reviewed, action is often not taken on many of the recommendations.  These challenges are likely to be even more serious for social equity and gender equality as these themes are less familiar to many organizations and the mechanisms to review and action are often less developed.

 

Participants are invited to share their thoughts and experiences on the following questions (as well as others they propose) as they relate to equity focused and gender-responsive evaluation.

• In your experience what are the factors affecting the demand for and use of evaluation?
• Are there additional factors affecting the demand and utilization of social equity and gender equality evaluations?
• How could the demand and utilization of these evaluations be increased?
• What types of evaluation presentations would you propose to make evaluation more accessible to stakeholders?

 

 

 

[Awuor Ponge]

Dear Preetei,

Thanks for your suggestion. I agree with you and Florence on the need to answer the questions as per the themes. It would make it easy to follow up the discussion. That would have been the ideal situation. But in the present circumstances where all the four themes have been bundled together, it will make sense if the one responding indicates which theme they are responding to.

Just at the prompting of Florence, I saw the need to align our responses to the indicators already developed. There was some consultation between 4 and 7 November on all these goals. I attach herein the document and I will be going through it to see how we can isolate the indicators under each of the targets and how our discussion can feed into this.

At least the discussion for now is free flowing and with no restrictions.

[Rabab Saab]

I am interested to join a working group on Theme 1.  I am a Monitoring and Evaluation Specialist with over 17 years of experience.  I am hoping to contribute to the SDGs in as many ways as I can. 

[Akanksha Babbar]

Hi everyone,

I am looking forward to being a part of the discussions around this post. I also went through the comments below. Some of them reflect a lot of evaluation level experience that can be great to be learn from.

I had some thoughts on the same- based on my limited experience of using community media in and around the suburbs of Mumbai- to challenge the normalization of sexual harassment  among adolescent 'victims' and 'perpetrators'. 

I specifically want to refer to a video bridging exercise I once conducted with a couple of other colleagues for an NGO that wanted to initiate some work around street sexual harassment in suburban Mumbai.  This exercise involved a participatory workshop with adolescent girls  ( a group of 10-12 girls, all in the age group of 12 and 15). Through participatory exercises we began discussions, theatre related activities on how gender roles can be challenged and how and why sexual harassment happens and specifically how it is not normal. Once we reached a certain comfort level with the girls we taught them how to use a basic camera and facilitated a video making exercise in which they interviewed each other on their views on street sexual harassment. The video was scripted and shot by the girls entirely. 

A few days after this, we designed a similar workshop for boys from the same area, who the girls had mentioned and identified as responsible for harassment. We started this workshop by screening the videos the girls had shot. The idea was to mobilize the guilt that it can potentially introduce to make the boys question themselves and also rectify their beliefs around what the girls feel about harassment (since most them believed that the girls asked for it, wanted it and in fact enjoyed such attention from them).

There are a few reflections I had which I feel can speak to the post on SDGs.

The first one is from programme point of view. This workshop was done by three of us- another female colleague (apart from me) and a male colleague as well. During the course of designing and implementing the workshop, there were several points at which I did not agree with the point of view of my colleagues. However, the issue I want to raise is not about difference of opinion.

I was wondering that if one was to evaluate this programme, how would the evaluation framework assess gendered perceptions of the implementing agents themselves. This is related to my second reflection- which is that the conceptions of what counts as feminist 'empowerment'  varies significantly across contexts. I think that the lack of a fit between the community level conceptions of empowerment and those of the implementing bodies can affects the kind of 'impact' that a certain project can or cannot have. In projects like the one I described above- how would an evaluation framework be able to assess 'gaps' or 'fits' between the programme and the community level conceptions of 'empowerment'? What if the kind of 'impact' that is intended is not 'wanted' by the community?

[Rituu B Nanda]

Thanks Akanksha. Referring to the question you pose, I worry that we outsiders should not create a rift in the community. Once a colleague shared with me that after a training on gender for women in certain areas in Bangladesh, it created a conflict between women and their spouses as the women started demanding their rights. I think trainings are not enough, a collective reflection leads to behaviour change. We ie outsiders should not go to teach and tell but to be facilitators to stimulate community to think on the issues around gender and equity and take subsequent action. 

In my experience using strength-based approaches like community life competence, appreciative inquiry etc can create a space where communities can engage in evaluation and the process itself can be transformative. I can refer you to evaluations we have conducted at ISST.

[Inem K. Chahal]

Hello All,

I response to the limitations to current data collection methods section, from my professional experience, two main limitations come to mind:

1) Ineffective/lack of data storage

I had evaluated projects in the past where data was collected years prior, but because of high staff turnover, the data was not stored and/or handled effectively. It is important to store it in an accessible database so that it can be found and used even years later, or else more time, money and resources will need to be committed in order to recollect data that would have been available had it been stored or stored in an accessible way.

2) Asking the right questions when drafting data collection tools

It is important to ensure that the questions you ask the stakeholders/participants directly answer the evaluation questions you seek answers to. It is very helpful to make a list of the types of information you need to answer the evaluation questions prior to drafting data collection tools (key informant interview guides, questionnaires, surveys, focus groups guides, etc.) and then draft the questions that will help you collect the required information.

Inem K. Chahal

Research Associate/Program Analyst, HCA Consulting

[Rituu B Nanda]

Dear Inem,

Thanks for your response. I sometimes work on very sensitive issues in which case participants do not open up even if ask the right questions. What is your experience of facilitation in this case?

Thanks

Rituu

[Inem K. Chahal]

Thanks for your reply Rittu.

I find myself in that situation as well from time to time. In those cases, it is best to have someone from that particular community (someone whom stakeholders respect and trust) on your data collection team, people are more likely to open up to someone local, which they trust, as opposed to 'outsiders' to the community.

Inem

[Rituu B Nanda]

I like that Inem! If data collection is participatory:-) I am facilitating participatory statistics right now!

Also I have used elements of community life competence approach to create an environment where people can share openly. So nice to discuss with you Inem. Thank you

[1wloq31p6ypda]

I agree with this approach. We are applying it in our programs. Unfortunately, and sometimes, even with the intervention of those deemed to be respectable stakeholders and who even know the respondent, the latter still refuses to provide the correct response/data. We keep working to elicit the information.

[Awuor Ponge]

Dear Inem,

I seriously identify with your two issues. They somehow limit our data collection methods, and with that, the data collected. I have been thinking about information technology as a way of probably solving this. What if we made our systems such that at all points of data collection, we have a system that collects and relays information real-time to multiple data points, so that even in cases where one staff at one point leaves the system, the data is already captured and shared with multiple actors.

I have been part of an evaluation Team, and at the end of each day, we shared our evaluation notes with the whole team on Dropbox, including the audio files. We came to realise that this was advantageous in a case where this information was required later and one of the researchers was not there. In the absence of this shared data, it would have meant going back to the field with all the inconvenience and unlikelihood of getting the same respondents and same data earlier collected. So we have to harness the power of information technology in data collection, storage and retrieval.

But from a gender perspective, it even becomes critical how information technology is accessed and utilised. Who has access to, say the mobile phone or other data collection gadget? And who controls the access to the same? We can have a situation where a woman has a mobile phone to report say on market dynamics in her village real-time. But given the power dynamics in the household, the man can come and get the phone for his own personal use, after losing his. This may therefore mean that the woman is not able to report on the market dynamics leading to missing data.

That aside, the analysis of the data presented can also be problematic. How do we capture the qualitative aspects of the information that is specific to men and women through information technology? How do we for example isolate cases of information specific to the marginalised or the socially excluded categories. These are some of the issues we need to interrogate. 

Thank you!

[Isha Wedasinghe Miranda]

Dear  Awour 

As i said before mobile phones  data is critical since it is very personal object and it will have a privacy issues. to obtain mobile phone data it has  covered with Rights, Mobile phone company ethics and personal privacy. 

I personally think we must go with evidence based data which has institualization (recognized institutions) methods in order to avoid above issues. 

[Awuor Ponge]

Dear Isha,

In agreeing with you about mobile phone data and privacy issues, let me throw spanner in the works, what happens with these projects on health targeting rural women and which rely on mobile phone services as the main source of data collection? What happens to the projects that have handheld devices for women in the rural areas say to report their temperatures, especially those targeting expectant mothers, so as to know whether everything is okay with them? I'm thinking trust can be one of the cornerstones in this kind of data collection. 

As you rightly say, the data should come from institutionalised methods, the examples I just gave above in most cases are coordinated from the hospitals or clinics serving women in the rural areas. So at all levels, as we work on trust, the next thing we should be asking ourselves is the efficiency or reliability of the data collected.