Gender transformative evaluations: Negotiating power, intersectionalities, contexts, and accountability

[Ranjani K.Murthy]

 

Gender transformative evaluations: Negotiating power, intersectionalities, contexts, and accountability

Ranjani K Murthy, Anweshaa Ghosh and Ayesha Dutta, 2021

 

Abstract

Development evaluations take place within social hierarchies and can both reflect and challenge these hierarchies. In this discussion paper the authors reflect on their experiences in facilitating gender transformative evaluations in the development sector from India and a few other developing countries, and argue that such evaluations entail weaving in principles of gender transformations in evaluation concept, evaluation conduct, evaluation relations and evaluation resources.  This is not a smooth process, and involves negotiating power, intersectionalities, contexts and accountability. This negotiation is difficult in the case of evaluations of projects/programs which do not have a gender transformative focus, pointing to the need for gender transformative policies and programs in the first place.

Introduction

Gender transformation entails transforming individual attitudes & behaviours, social norms, resources, structures and agency towards substantive gender equality; that is equality which goes beyond equal opportunities to address barriers which prevent most marginalised women from achieving equality in outcomes (UN Women Asia Pacific, 2014). The concept of gender transformation is particularly relevant in India where one finds huge gender gaps, evident from the Global Gender Gap Index 2020 wherein India ranked 112th out of 153 countries (World Economic Forum, 2019). One also finds marked inequalities in development outcomes amongst Indian women based on caste, class, ethnicity, minority status, disability, gender identity and sexual orientation.

Development evaluations in India take place within this larger hierarchical context, and can both reflect and challenge the inequalities. This article shares and analyses the authors’ experiences in facilitating evaluations and the challenges faced by the authors in weaving in principles of gender transformation in four elements of evaluation - evaluation concept, evaluation conduct, evaluation relations and evaluation resources (drawing on UN Women, 2015; Batliwala and Pittman, 2010; and Murthy, 2019). Based on this reflection, the authors pull together lessons on what facilitates and hinders gender transformative evaluations in the context of development evaluation.

The authors together have 35 years of experience in evaluation, which they draw upon in this article. The evaluations facilitated by the authors comprised of some projects/programmes which aimed at gender-transformation and others which did not.  Half of the fourteen evaluations of projects/programmes examined in this article focused on livelihoods and the rest on gender-based violence against women and girls. Most of the evaluations facilitated by the authors were summative evaluations, and were participatory in approach.   In this article, examples of good practices are supported by reference to relevant evaluation reports, while practices that need strengthening are shared in anonymity. Eighty percent of the examples used in this article pertain to   evaluations conducted in India. The rest of the examples cited in this are from other countries when illustrations were not available from India.

The strength of this narrative is that it highlights the real time issues and challenges that are currently being experienced in India in development evaluation through the collective experiences of the authors. This means that evidence is anecdotal as opposed to a formal and published evaluation. Recourse to personal narratives has been endorsed by the not-for profit BetterEvaluation which observes that personal stories provide qualitative information that is not easily classified, categorised, calculated or analysed, and that more value has been placed on narrative and anecdotal information in recent years (Salm and Stevens, 2020).The authors have drawn on some secondary literature mainly to bolster and elaborate the different elements of evaluation that connects with the issues that are encountered in the evaluation space, as well as to support their observations.

Few reports from a gender-transformative lens are in the public domain. On this basis, the collective voice plays an important role to encourage more evaluators to create more awareness of the gender- based issues in evaluations and provide a platform of change. This paper hopes to achieve this even though the authors are cognisant of the limitations of anecdotal style compared to a larger evidence based evaluative pursuit. This limitation points to research opportunities for future.

Elements of Evaluation

Evaluations can be seen as comprising of four essential elements: concept, conduct, relations and resources (drawing on UN Women, 2015; Batliwala and Pittman, 2010 and Murthy,2019). Chen (2012) sees evaluations as driven by a programme theory (or concept) of change. The purpose of theory-driven evaluation is not limited to assess whether an intervention leads to outcomes or not, but also to how and why it does so. At another level, evaluation concept can be seen as including evaluation objectives, evaluation criteria, evaluation questions and evaluation indicators (adapting from United Nations Evaluation Group (UNEG), 2018). 

The second element is evaluation “conduct”, which UN Women sees as including recruitment of the evaluation team, evaluation inception, data collection and data analysis (UN Women, 2015). Expanding further, evaluation conduct can be seen to additionally include sampling, identification of stakeholders, choosing and using evaluation methodologies and methods, identifying evaluation principles and analysis of data/information and report writing (see BetterEvaluation, 2014).

Murthy (2019) draws attention to a third element of evaluation, “relations of evaluation”. By this she refers to relationships between the evaluation team and implementing agency, donors and the implementing agency; evaluation team and the donors and the evaluation team members. A workshop on “Politics of Evaluation” by BetterEvaluation, organized in 2018, adds one more relationship to this list- the relationship between the evaluation team, commissioner and the marginalized (BetterEvaluation, 2018).

The last element of evaluation, “evaluation resources”, concerns the gender and North-South divide in terms of recruitment, contracts, financing and evaluation fees (Bruckner, 2015). Evaluation resources in this context refer to the availability of funds to visit most marginalised groups and appropriately compensate those that are marginalised for their involvement in an evaluation. It also includes equity in allocation of resources within the evaluation team.

These four elements do not operate in silos but actually influence each other. The conceptual underpinning of the evaluation influences its conduct, the relationship between different stakeholders and how resources are allocated. For example, a theory of change that sees working with men, boys and women on masculinities as essential to reduce violence on women and girls (evaluation concept) will involve discussions with women and men/boys on attitudes, incidence of and action on gender-based violence (evaluation conduct). It would also need non-hierarchical relations between the evaluation team, the implementing agency and groups of women and men (relationship), and resources (personnel and funding) for visiting women/girls and men/boys in the same village at a time which suits them (resources).

Integrating gender into evaluation concept

As discussed, evaluation concept refers to the conceptual underpinning of the evaluation comprising of evaluation objectives, theory of change, evaluation criteria, evaluation questions and evaluation indicators. The lead author has experienced that in the case of evaluation of programs and projects with gender-transformative objectives, evaluations too tend to have transformative objectives. For example, the Tejaswini Rural Women’s Empowerment Program in India (2005-2018) , which was sponsored by the state governments of Maharashtra and Madhya Pradesh, aimed to foster women’s empowerment and poverty reduction (Independent Office of Evaluation (IOE), 2020). As such, the theory of change of Tejaswini programme saw women’s mobilisation, strengthening of women’s livelihoods, reduction of their drudgery and sensitisation of men and policy advocacy as essential to expanding choices of women in the economic, social and political realms and empowering them. Thus, the evaluation questions and indicators explored impact of the program on expanding assets of women, intra household and community decision making of women and incomes of women (IOE, 2020).

It becomes problematic when the objective of the project/program is gender blind or where women are treated only as a target group. In such situations, we found that concerns around gender are added in evaluations as an afterthought as against being integrated at every level of the evaluation of such projects/programs. For example, a project in Rajasthan (India) in 2008, had an objective of mitigating poverty amongst the Adivasis[1], Dalits[2] and women. However, there was no reference to gender equality in the objective of the project.  Though the overreaching objective of the evaluation was to assess poverty reduction impact, the evaluation went on to also assess whether the gender specific dimensions of poverty, causes of poverty and adverse coping strategies of Adivasi and Dalit women were addressed and reduced by the project.

Evolving and testing a theory of change is integral part of evaluation. Mere inclusion of the term “women” or “gender” in theory of change does not ensure that complexity of the field is taken into account. To illustrate, the theory of change of a project in 2005 that aimed to end early marriage of girls in Tamil Nadu (18 years or below) in India, assumed that monitoring drop-out of girls from schools, educating mothers and women’s group members on harmful effects of early marriage, and helping parents access incentives from government to continue their girlchildren’s’ education in schools would be enough to arrest early marriage. Power relations and institutions which influence early marriage were not addressed in the program’s theory of change, like bringing changes in men’s attitude, attitudes of traditional community leaders (men) and safety in public spaces. Further, contingencies like repeated disasters and the need for families to move to insecure temporary camps were not anticipated (Murthy and Sagayam, 2006). As a result, early marriage of girls could not be eliminated, but only reduced. To maximise positive outcomes the interconnecting factors that create such interwoven complexities must be embedded and balanced in the project’s theory of change.

Coming to evaluation criteria, some evaluations use the Organisation for Economic Cooperation-Development Assistance Committee (OECD- DAC) criteria of relevance, coherence, effectiveness, efficiency, impact and sustainability (OECD, 2020). Some organisations have added gender equality and women’s empowerment under other performance criteria (IOE, 2015). For gender transformative evaluations to be mainstreamed, it would be useful to integrate gender in each OECD-DAC criterion and have a stand-alone impact on gender/social transformative outcomes and impact. Failure to integrate these may lead to incoherence. For example, in an evaluation of a nutrition sensitive agriculture programme, the nutrition specialist concluded that the intervention targeting mothers to further child nutrition was effective, while the gender specialist, in her part of the report, analysed that the intervention reinforced stereotypes.

Gender integrated and stand-alone gender equality criterion need supportive indicators which are gender-transformative.  These indicators need to go beyond what is mentioned in the project proposal, and should be linked to testing the evaluation’s theory of change. It is essential that gender transformative indicators go beyond participation of women in numbers to looking at changes in norms, resources and power as relevant to the project/program. In an evaluation in 2018 by the lead author of a project that worked with men and boys as parents, spouses and siblings in Jharkhand, a state in central India, found impressive gains in terms of men “helping” in house work, reduction in policing of girls’ mobility and interaction by brothers, and reduction in domestic violence. However, this was yet to become a norm in the areas covered, extending to those who did not participate in the project. Whilst men started to help with cooking, cleaning and looking after their children, they were rarely found to share the house work and care in totality (Murthy, 2018).

 An important task for gender transformative evaluation is to ask transformative questions pertaining to varied identities that women hold, and how they intersect with each other. A review of project reports may show that women from vulnerable communities such as Dalits, minorities, disabled, etc. are rarely seen as part of the committees that manage government schemes such as the Integrated Child Development Scheme (ICDS)[3], public distribution system (of food and non-food commodities) and government schools. It is important to probe reasons and further explore the kind of actions required by government and project managers to ensure the most vulnerable women be involved and included in these committees.

 Another example of the need for attention to intersectionality in evaluations is an evaluation of a program related to Muslim women’s empowerment in Tamil Nadu, India, which one of authors was conducted in 2018. The program had a positive impact on enhancing the education of girl children, reducing girl child marriages, reducing domestic violence and enhancing self-employment opportunities for Muslim women. Through repeated dialogue with religious leaders, it also reduced the incidence of instant Triple Talaq[4] in several villages. However, when the recent national legislation banning Triple Talaq was passed by the government, there were unjustified negative repercussions on the objectiveness of the project leader and her staff who were mistaken to be allies for right-wing nationalists(Legislative Department, 2019).This brought about a further evaluation-based question concerning the timing of gender transformative interventions and whether they should be initiated and continued when the ruling national party is not progressive.

 Yet another evaluation question is whether a human rights approach underpins the interventions; for example, is the approach to furthering women’s land ownership, “market purchase” oriented, or is it enabling women to fight for their legal rights under land reform and right to inheritance. Questions probing human rights should be backed by indicators (input, process, output, outcome and impact), which are gender transformative. A few example of such questions could be - what is the gender disaggregated data of the staff, who are aware of legal entitlements of marginalised women (input and output respectively), what is the percentage of meetings are conducted in marginalised settlements (process), what percentage of Adivasi women attend meetings with district officials to claim their right to forests (Outcome) and what percentage of those women who attend such meetings are able to secure forest titles (impact).

 

Integrating gender into evaluation conduct

In order to make evaluation conduct gender transformative, it is necessary to integrate gender and (substantive) equality concerns into sampling, identification of stakeholders, choice of evaluation methodology and methods, evaluation principles, analysis of data/information and report writing (adapted from UN Women, 2015, UNEG, 2018).

Gender transformative evaluation entails weaving gender concerns into sample selection. Given the scope of the evaluation, this would also entail visiting areas where gender interventions have been successful as well as those where it has failed or been less successful. Further, attention to gender and diversity is important while selecting the sample. As such, the sampling has to be purposive and include women from diverse and vulnerable sections of the society. These would include women from female headed households, single women, Dalit women, women from minority communities, Indigenous women, disabled women, elderly women, young women, sexual/gender minorities and other vulnerable women.  

To illustrate the above, one of the authors as part of an evaluation team assessing gender and caste equality impact of micro-finance self-help groups (SHGs) in 2019 in a district in Tamil Nadu in India, visited villages with defunct women’s SHGs, those with active ones, as well as women’s SHGs with different caste composition (Murthy et al., 2020). Individual discussions were held with women who were heading households, and a sample of women who had improved and women who had not (Murthy et al.,2020).

Further, in the case of projects or programs that involve men, it may be useful to meet their spouses and/or other women family members to understand how benefits get distributed within the household. A comparison with non-participants is also crucial, provided that they were at the same socio and economic level as the participants when the programme began and were not influenced by the project or program. In fact, one of the authors compared empowerment impact of an urban poverty reduction project in northern Sudan in 2002 on participating women, participating men’s spouses, and non-participant women. This exercise showed that women who participated in the project visited their parents more, undertook non- traditional activities, had greater access to income and assets, and took independent decisions on voting when compared to women non participants. The performance of spouses of male participants fell in between, suggesting that working with men on gender may have its own benefits (Murthy and Al-Showya, 2002).

In gender transformative evaluations, the stakeholders involved need to be from a wider cross-section in addition to program participants (Hillenbrand et al., 2015). It could include elected representatives, front line government workers, traditional midwives, field staff, leaders of NGOs, representatives of project implementation unit, police, block development officers, district collectors, people in market places, etc. This will bring different perspectives into nature and the extent of changes in participants, how changes varied with context, and reasons for observed changes (Hillenbrand et al., 2015).   For example, in an evaluation of the Women’s Empowerment Project in Badakhshan, Afghanistan in 2017, traders in marketplaces observed that more women were coming to the market than before, and these turned out to be mainly from women headed households and who were leaders of self-help groups (Murthy, 2017).

 

Evaluation methods, ideally, need to comprise of mixed methods; surveys along with participatory methods like gender-based violence mapping, body mapping, story-telling, gender and seasonality mapping, decision making matrix and gender division of labour mapping add value to the evaluation. As observed by Batliwala and Pittman (2010), quantitative methods such as surveys, help generate statistical evidence, while qualitative methods help generate more nuanced information at individual and institutional levels, and interpret contribution to change. It is important that questions asked in the surveys must be transformative. To give an example, a survey with adolescent girls asked open ended questions - has the road of your life changed during the life span of the project? If so how and why? (Gupta and Murthy, 2016). The evaluation team documented reasons related to project separately from the other reasons. An adolescent girl drew an ascending road map (indicating progress)[5], and explained that she was able to stand up against harassment by boys in her neighbourhood because of the training she received from the project and viewing a television program on addressing violence against women and girls,  which was not linked to the project (Gupta and Murthy, 2016).

Moving to the issue of feminist evaluation principles, care needs to be taken that the evaluation takes place at a time and place convenient to the most marginalised women. Holding the meeting at a time convenient to most marginalised women will increase the number of women being able to attend and strengthen quality of discussion. Often times, women who are labourers have to take leave from work to attend the evaluation meeting which results in loss of wages. Seasonality is also important to take note of before planning an evaluation. It would be difficult to speak to women around important festival or agricultural season when rural women tend to be extremely busy (Mosse, 1995).

Identification of an appropriate place to hold discussions during the evaluation process is also essential. The place should be safe and should provide participants with privacy as well as the confidence to speak freely. Discussions, when conducted in public or open spaces, often get interrupted by the presence of men. Further, as observed by Mosse (1995), in some communities in India it is not the norm that women speak in public spaces in front of men. In particular, discussions on sensitive topics such as sexual violence, reproductive and sexual choices and decisions cannot be held in public or open spaces. Similarly, within the cultural context, younger women will also not share openly if elder women (such as a mother-in-law) of their household are present (Mosse, 1995). Hence, a gender sensitive facilitator needs to take in to account all of these constraints in order to facilitate an immersive and open discussion during evaluations.

 Frameworks like the Gender at Work, social relations framework and power and empowerment framework can be used to give depth to the analysis of findings form evaluations (Chigateri and Saha, 2016). For example, the Gender at Work framework entails assessing changes of a project or program in four quadrants: formal-systemic (e.g. policies), formal-individual (e.g. resources, control over body), informal-systemic (e.g. norms on violence against women) and informal-individual (e.g. individual beliefs) (Gender at Work, 2018). Using the Gender at Work framework, the Tewa team (a women’s fund in Nepal) trained by one of the authors, assessed one of their partners’ work with women in entertainment sector in Kathmandu, Nepal and their employers. The women in entertainment reported that there was less sexual harassment at work, they were provided with transport facilities to return home in the night and salaries had increased (all three changes fell within the formal-systemic domain as per the Gender at Work framework discussed here). The respect accorded by employers to women employees, and the confidence of women involved in the programme increased, but it did not extend to those not involved in the program (both changes in the informal-individual quadrant). While the impact on the policy framework for women in entertainment was limited (formal-systemic quadrant), the stigma in society against women in the entertainment industry continued (informal-systemic quadrant) (Tewa, 2015).

 Finally, it is also vital for evaluation reports to capture complexities of gender transformative change and contribution of the project/program, and report lessons learnt and recommendations (Batliwala, 2011). Further, it would be of great benefit if the findings being reported included the comments of key stakeholders to provide comparative benchmarking for ongoing monitoring of the project/programme.

 Recognising power relations in evaluation

Power relations play-out between various actors in evaluations: evaluators and the marginalised community members; gate keepers of the community and the marginalised participants; evaluators and the implementing agency; donors and the implementing agency, and; donors and the evaluation team (Podems, 2018; Murthy, 2019).

In the 1990s, in an evaluation of a government project with Devadasi women[6] in Karnataka in India, the men belonging to privileged caste of the community would not allow the evaluation team to meet the Devadasi women. Ultimately government officials in charge of the programme had to intervene in order to allow the lead author to meet the women. This incidence highlighted the importance of dialoguing with gate keepers of the community, and not buckling to their unjust demands. The experience also brings to fore the importance of involving government officials who were seen to have greater power and authority and thus greater ability to dialogue with the gatekeepers of the community.

It is also critical to recognise the power that the evaluator holds vis-à-vis other actors in an evaluation such as the women in the community who do not know the evaluator. Including a staff of the implementing agency during the evaluation may reduce this hierarchy, although there is a risk that the women may express opinions which the implementing agency wants to hear or may be hesitant to speak clearly in their presence. This risk is less if the relationship of implementing agency with the women in the community is based on mutual trust and if the evaluation is delinked to future funding. Further, one must recognise the hierarchies amongst marginalised groups, which may come in the way of assessing outcomes during evaluations.; for example, Dalit women in South Asia rarely speak up and contradict the opinion of other caste women (some of the other caste women may be their employees). On a similar vein, Lahiri-Dutt and Samanta (2006) argue that upper-caste women and their husbands act as gate keepers to poor women’s access to programmes and their participation in evaluations 

Yet another aspect that reflects power relations is evaluators taking photographs and video recordings of meetings and interviews without consent, and sharing it without having control over their possible use. As observed by the Photography Center (2018), the use of photography in social research, this also raises issues of power hierarchies and inherent dignity of individuals. As such, one could avoid photographing faces of women and members of marginalised communities to maintain anonymity. While this is an ethical principle in general, the consequences can be more adverse for women than men in conservative settings.

Reporting findings is a crucial part of any evaluation and multiple players will influence how the report turns out to be and to what extent changes can be incorporated. There have been incidences where the head of the organisation tends to defend the organisation’s work while field staff is more open to share the inadequacies, unless the field staff’s remuneration is linked to the evaluation findings. Donors sometimes want to set agenda of evaluation, hold grantees accountable and at times showcase only positive achievements (Plotkin and Stagman, n.d.). As such, we find evaluation reports which are critical are less likely to be put in the public domain. To illustrate, in an evaluation facilitated by an evaluation team in which one of the authors was involved, a couple of participants in a larger group pointed to some areas for improvement. The donors, however wanted this to be removed, as it was expressed by a small number of participants. Nonetheless, the evaluation team retained the points as they felt that every voice should be counted. Likewise, Podems (2019) notes the paradox that when evaluations come up with critical and unanticipated findings the client questions, and when they come up with what is expected they are questioned on the value addition.

There are also divisions and hierarchies within the evaluation team. In the authors’ experience, in addition to this, gender and caste can also be added.  It is common to see that the responsibility for examining gender aspects often allocated to a woman team member, while evaluation of finance or procurement is allocated to men. For example, in a multi-country evaluation mission where one of the authors was involved, the gender/social inclusion and nutrition consultants were all women, while the specialists on policy/strategic positioning and data management systems were men.  As mentioned by WestEd (2021), when the evaluation team comes from different races, it is not uncommon for the implementing agency to give priority to those who are from the global North and especially who are white- irrespective of designation in the evaluation team. This is often due to the individuals’ internalised hierarchies based on race, caste and other prejudices. This is an important concern which must be addressed by the team leader and donors. Further, one also finds a bias towards appointing a cis-gender male evaluator as the evaluation lead. While, over the years more women have been entering the evaluation units and leading evaluations nevertheless, the situation is still far from equal.

The power relations between different stakeholders in evaluation do not operate in isolation, rather they overlap/interlock. To give a positive example, donors who are committed to long term change towards women’s empowerment do not normally see evaluation as a way of deciding whether to continue the project they are funding, and the terms of reference for evaluation is more oriented towards facilitating learning from what worked and what did not work. This is conductive to non-hierarchical relationship between evaluation teams and implementing agencies, and in-turn with marginalised women. As Podems (2018) rightly observes, evaluation entails strategically negotiating different interests of important players in evaluations.

Distribution of Evaluation resources

Some of the important resources-related issues in gender transformative evaluation are whether funds allocated for evaluation are adequate for visiting remote areas and meeting marginalised women. At the field level, it is important to compensate respondent’s time, just like evaluations compensate the evaluator’s time (Plotkin and Stagman, n.d.). Individual compensation to women for interviews may lead to exclusion of those not chosen or who missed the evaluation as they were busy with their livelihood. The authors suggest collective compensation may be more appropriate and this compensation can be paid directly to the group or to an apex body (collective of groups) if there are more than one group in the same village. However, one also needs to be mindful that the collective compensation is judiciously used and not co-opted by some powerful members of the community/collective.

Yet another issue is equitable payment to evaluators in the case of multiple country teams. A question is whether payment is gender and socially and economically equitable, as at times there are local and international consultants getting paid very different rates. For example, from the author’s own experiences, there are significant differences in remuneration of national and international consultants with similar qualification and years of experience. Further, it was also observed by the authors that there were differences of payment within the evaluation team based on thematic responsibility; the gender and social inclusion specialist got paid at a different and lower rate in comparison to the Institution or Marketing expert.  However, on a positive note, skill upgradation and capacities of development consultants from low and middle-income countries have been built over the last few years and many of them are now involved in international consultancies thereby levelling the field (Bruckner, 2015).

An additional issue around resource allocation is payment to implementing agencies for data collection costs such as transport, food and other logistical expenses which help make evaluations work. If this expense is not budgeted, it takes away money from program budget allocation which the authors believe is unethical (Plotkin and Stagman, n.d.).  If the evaluation includes a survey, an important question to note is whether the people who do the survey are compensated adequately, based on minimum wage of the place where the survey is being conducted. 

In recent years, there has also been a push by many evaluators to take back findings from the evaluations to the marginalised communities for their insights and recommendations in order to make the evaluation more robust and accountable (WestEd, 2021). In one such feedback session in January 2020, a Tamil Nadu level federation of women’s groups added several insightful recommendations which the evaluation team had not thought of, and also stated which recommendations of the evaluation team they would like to take up immediately (Murthy et al., 2020).  For example, they added the recommendation of holding a public hearing of violations of rules of the Reserve Bank of India by profit oriented micro finance institutions. These violations affected women in particular, as they were the primary borrowers. Hence, a good practice to follow would be to allocate budget for discussing findings and recommendations with forums of marginalised women.

WestEd, 2021 also points out the need of allocating budget for appropriate dissemination product which is language appropriate and bias free. The authors also feel that budget allocation for different products for dissemination to different audiences is a must and this will enable better uptake of the evaluation findings.  

Conclusion

Development evaluations take place within social-economic hierarchies and can both reflect and challenge these hierarchies. This article has pointed to the need for consciously weaving principles of gender transformation in evaluation concept, evaluation conduct, evaluation relations and evaluation resources. This is not a smooth process, and entails negotiating power, contexts and accountability in evaluations. This negotiation is difficult in the case of evaluation of projects/programs which do not have a gender-transformative focus, pointing to the need for gender transformative policies, programs and projects in the first place.

Gender-transformative evaluations need to take into account intersections of identities, and not treat women as a homogeneous category. Hierarchies exist amongst women based on race, caste, class, relation position (in family and community), marital status etc. These need to be taken into account while evolving theory of change, identifying questions and indicators, facilitating evaluations, interpreting findings, reporting and taking the findings and report back to women. It is important to keep in mind that gains for one group of women may be at the cost of another more vulnerable group of women. Social transformation is part of gender transformation, and this applies to gender-transformative evaluations.  

At the same time, evaluators in the field of gender-transformation need to be aware of how far, and how fast, gender and social relations can be pushed. While gender-transformative evaluations examine if men are sharing domestic work and child care responsibilities, at a practical level it is important for development evaluations to facilitate meetings with women taking into account such responsibilities. Further, at times compromises have to be made by evaluation teams like negotiating with male community gate keepers to meet women. That is, ‘what is possible’ has to be kept in mind, while pushing the boundaries of evaluation in a gender transformative manner. 

The cultural context, as well as socio-political matters; fostering gender transformative changes in some locations may be more difficult than others where gender norms are more rigid. The same milestones need not apply to all contexts, but progress matters.  Across cultures, changes at individual, family, community, market, and state levels need to be tracked, in ‘normal’ and special situations like disasters, conflict, and pandemics. 

The authors have pointed to existence of gender transformative frameworks (e.g. Gender at Work)  to capture change due to development policies and programmes. However, there are few gender transformative frameworks to assess underpinning politics of implementing agencies and donors that lead (or do not lead) to change. This gap is something that needs further research and work to develop suitable and appropriate frameworks for such complex social and political situations. It is hoped that this article has stimulated more thinking in this area.

Gender transformative evaluations have to be accountable to marginalised women. This includes sharing the findings and recommendation with the marginalised women, so as to make the evaluation more robust through their insights, observation and analysis as well as to create space for independent action on recommendations by marginalised women.

Some of the issues highlighted in the article  are from India and a few developing countries.. The bottom line is that these issues are critical to an equitable landscape for the purposes of evaluation as a field of practise and for improving the projects and programmes in the development space that are being evaluated.  Highlighting the collective experiences and views of the authors in this article is an opportunity to bring these issues to the fore to promote more awareness of the need for gender transformative evaluation to be acknowledged and acted upon for the betterment and empowerment of women, their livelihoods and their families.

Endnotes

[1]Adivasis, the original people, is a collective name for a group of approximately 100 million people identified in the Indian Constitution as Scheduled Tribes. They are also referred to as tribals (Oskarsson and Sareen(2020).

[2] Dalit means “oppressed” and are listed as the Scheduled Castes under the Articled 341 of the Indian constitution (Szczepanski, 2021).

[3] The Integrated Child Development Service (ICDS) Scheme was launched by the Indian government in 1975 and provides for supplementary nutrition, immunization and pre-school education to the children (Government of India, 2021)

[4] Triple talaq is a form of divorce that was practised in Islam, whereby a Muslim man could legally divorce his wife by pronouncing talaq (the Arabic word for divorce) three times. The pronouncement could be oral or written, or, in recent times, delivered by electronic means such as telephone, Short Messaging Service (SMS), email or social media. The Muslim Women (Protection of Rights on Marriage) Act, 2019 passed on 26 July 2019, after a very long discussion and opposition. It made triple talaq illegal in India on 1 August 2019 (Legislative Assembly, 2019).

[5] An ascending road map indicates that the adolescent girl who drew this road map was making progress towards empowerment (Gupta and Murthy, 2016).

[6] Devadasi system is a religious practice in some parts of southern India whereby parents marry a daughter to a deity or a temple. The marriage usually occurs before the girl reaches puberty. In recent decades, the practice has been used to push young girls into sex work.

---

References

Batliwala, S. (2011). Strengthening monitoring and evaluation for women’s rights: Twelve insights for donors. Association for Women’s Rights in Development. https://www.awid.org/sites/default/files/atoms/files/strenghteningm...

Batliwala, S. & Pittman, A. (2010). Capturing change in women’s realities: A critical overview of current monitoring and evaluation frameworks and approaches. Association for Women’s Rights in Development. https://www.awid.org/sites/default/files/atoms/files/capturing_chan...

BetterEvaluation. (2018). Managing the politics of evaluation. https://www.betterevaluation.org/en/events/elearning/TRUSS_Managing...

BetterEvaluation. (2014.). Rainbow framework. BetterEvaluation Evaluation Plan Template

Bruckner, T. (2015). International consultants: Overpaid today, extinct tomorrow? Devex. https://www.devex.com/news/international-consultants-overpaid-today...

Chen, H. (2012). Theory-driven evaluation: Conceptual framework, application and advancement. In R. Strobl, O. Lobermeier& W. Heitmeyer (Eds.) Evaluation von programmen und projektenfüreinedemokratische kultur. Springer. https://doi.org/10.1007/978-3-531-19009-9_2

Chigateri, S.& Saha, S. (2016). A resource pack on gender transformative evaluations. Institute of Social Studies Trust.

Gender at Work. (2018). Gender at Work Framework. Retrieved from https://genderatwork.org/analytical-framework/

Government of India. (2021.). Integrated child development scheme. https://vikaspedia.in/social-welfare/women-and-child-development/ch...

Gupta, P., and Murthy, R.K (2016, October 11). Making a difference: Empowering girls through sports, life skills and gender awareness. https://gendereval.ning.com/profiles/blogs/making-a-difference-empo...

Hillenbrand E, Karim N, Mohanraj P and Wu D. 2015. Measuring gender-transformative change: A review of literature and promising practices. CARE USA. Working Paper http://pubs.iclarm.net/resource_centre/AAS-Working-Paper-Measuring-...

Independent Office of Evaluation of IFAD (IOE). (2020). Tejaswini women’s empowerment programme. International Fund for Agricultural Development. Retrieved April 5, 2020, from https://www.ifad.org/en/web/ioe/evaluation/asset/41984457

Independent Office of Evaluation of IFAD (IOE), 2015, Evaluation Manual-Second Edition, IFAD, Rome. Retrieved April 10, 2020 from https://www.ifad.org/en/web/ioe/evaluation/asset/39984268

 Legislative Department. (2019.). The Muslim women (protection of rights on marriage) act, 2019. Ministry of Law and Justice, Government of India. Retrieved April 5, 2020, from https://legislative.gov.in/actsofparliamentfromtheyear/muslim-women...

 Mosse, D. (1995). Authority, Gender and Knowledge: Theoretical Reflections on Participatory Rural Appraisal. Economic and Political Weekly,30(11), 569-578.

 Murthy, R. K. (2019, November). Gender transformative evaluations [Presentation]. Building Alliances for Gender Transformative Evaluations, New Delhi, India.

Murthy, R. K. (2018, December). New frontiers in working towards gender equality and child rights?: Evaluation of the project ‘responsible partners and caring fathers . Centre for Health and Social Justice, New Delhi.  https://www.researchgate.net/publication/329707280_New_Frontiers_in...

 Murthy, R.K (2017). Review of Women’s Economic Empowerment Project Badakhshan, Afghanaid, Kabul.

Murthy, R. K., Arogyagam., Vasantham., & Iyakkam, T. M. P. (2020, July). Gender, caste, and SHGs in the context of neo-liberal MFIs. https://www.researchgate.net/publication/343305180_Gender_caste_and...

Murthy, R. K.,  and Sagayam, J. (2006). Non-discrimination, children and Tsunami. Save the Children. https://www.researchgate.net/publication/282335963_Non-Discriminati...

 Murthy, R. K., and Al-Showya, S. (2002). Lessons from the gender and poverty impact study of urban poverty alleviation program, Khartoum. United Nations Development Program. https://www.researchgate.net/publication/332014231_LESSONS_FROM_THE...

 Organisiation for Economic Cooperation and Development (2020). Better Criteria for Better Evaluation: Revised and Updated Evaluation Criteria. https://www.oecd.org/dac/evaluation/evaluation-criteria-flyer-2020.pdf

 Oskarsson, P. & Sareen, S. (2020). Adivasiness as Caste Expression and Land Rights Claim-Making in Central-Eastern India. Journal of Contemporary Asia, 50 (5), 831-847. DOI: 10.1080/00472336.2019.1656277

Photography Ethics Center (2018). Understanding photography ethics in social research. https://www.photoethics.org/content/2018/5/31/understanding

-photography-ethics-in-social-research

 Plotkin, G. & Stagman, S. (n.d.). Mitigating Power Dynamics in Learning and Evaluation. Greater Good Blog. https://www.arabellaadvisors.com/blog/mitigating-power-dynamics-in-...

Podems, D. (2019). Being an evaluator: Your practical guide to evaluation. The Guilford Press, New York.

Podems, D. R. (2018, September, 28). Evaluation for development: A focus on transformation and the global south Evaluation for Transformation. https://zendaofir.com/donna-podemss-top-tips-for-yees/

Salm, M. & Stevens, K. (2020). Stories. BetterEvaluation. https://www.betterevaluation.org/en/evaluation-options/stories

Szczepanski, K. (2020, February 4). Who are the Dalits? ThoughtCo. https://www.thoughtco.com/who-are-the-dalits-195320

 Tewa. (2015). Learning and evaluation training: 1st February 2015-5th February 2015. Tewa, Kathmandu.

 United Nations Evaluation Group (2018). Guidance Document-UN-SWAP Evaluation Performance Indicator Technical Note.

 UN Women. (2015). How to manage gender-responsive evaluation: Evaluation Handbook, Independent Office of Evaluation. https://genderevaluation.unwomen.org/en/evaluation-handbook

 

UN Women Asia Pacific. (2014). CEDAW-Principle of substantive equality (English).  https://www.youtube.com/watch?v=SZR0RJtghyY

 World Economic Forum. (2019). Global Gender Gap Report 2020, World Economic Forum, Geneva, Switzerland. https://www.weforum.org/reports/gender-gap-2020-report-100-years-pa...

WestEd. (2021). Anti-Racist Evaluation Strategies: A Guide for Evaluation Teams. https://www.wested.org/resources/anti-racist-evaluation-strategies/