Getaneh Gobezie Draft Publication
Economic Empowerment of Rural Women?Make Sure that She has Safe Control of Income: HARVEST TIME SAVINGS
Silberman School of Social Work/Hunter College MEAL Team Publication
Development in Practice - September, 2024
Women's Health Journal - September, 2024
Shama Dossa Publications
Gender & Development Journal Vol.31, Nos. 2-3, 2023
October 14, 2024 at 6pm to November 15, 2024 at 7pm – Ede - The Netherlands
0 Comments 0 LikesOn 19th July 2013, the Institute of Social Studies Trust (ISST), hosted a reflection session on ‘Randomized Control Trials and Gender’, at the National Foundation of India, India Habitat Centre. The session was organized with the primary aim of building an informal network of students and professionals, either working in or interested in the areas of evaluation and gender. Being first in the series of reflection sessions that have been planned under the project ‘Engendering Policy through Evaluation’ (funded by IDRC, Canada and the Ford Foundation), the session aimed at enhancing the participants’ knowledge and understanding of RCT’s while giving them the space to reflect upon the ways in which their implementation is effected by and in turn impacts gender relations.
Diva Dhar, Assistant Director of Capacity Building for CLEAR/J-PAL South Asia at IFMR facilitated the session by presenting J PAL’s work on RCT’s. The session was attended by participants from a variety of backgrounds- students interested in learning about the application of RCT’s for evaluation and research and mid-level and senior professionals from social development consultancies and non-profit organizations. The names of the participants have been mentioned at the end of the report.
After giving a brief introduction of J PAL’s mission to maintain a strong interdependence between evaluations, capacity building and policy outreach so that policy remains driven by evidence and research gets translated into action, Diva moved on to discuss several conceptual and related issues associated with RCT’s. These have been summarized below:
Defining ‘Impact’
Impact was defined as a comparison between the ‘outcome sometime after the program has been introduced’ and the ‘outcome at that same point in time had the program not been introduced’. This was explained as the best measure of assessing impact.
The ‘Counterfactual’
Defining ‘Impact’ in this way leads one to the ‘Counterfactual’, which ‘represents the state of the world that program participants would have experienced in the absence of the program ( i.e, had they not participated in the program)’.
This difference is often constructed by selecting a group not affected by the program. Thus, in randomized evaluations, individuals, villages or districts are randomly selected to receive the treatment, while other villages serve as a comparison. In this way, because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, any difference that subsequently arises between them can be attributed to the program rather than to other factors.
Advantages of RCT’s
When not to do Randomized Evaluations
J PAL evaluations in the gender space
Towards the end, Diva highlighted a few of J PAL evaluations in the gender space that employed RCT’s. One important example was that of J PAL’s evaluation of a school based intervention by Breakthrough. This intervention was based in Haryana and involved teacher training and capacity building for mobilization and education of youth on gender inequality, empowerment of girls, and SSE. It used innovative means of communication like audio-visual , digital media and online forums for youth discussion.
Some examples of the research questions through which J PAL sought to understand the programme’s impact on gender attitudes are as following:
The indicators of change were also designed in a gender responsive manner;
J PAL’s evaluation carried out RCT’s among 300 schools out of which 150 were Treatment schools ( that experienced Breakthrough’s intervention) and 150 were Control schools( that did not receive any intervention). Baseline, midline and endline surveys used quantitative and qualitative instruments like surveys, structured interviews, Implicit Association Test and Focus Group Discussions.
Diva then concluded by bringing out the ways in which the results drawn from the evaluation were used by Breakthrough;
Discussion
The presentation by J PAL team stimulated an interesting as well as intense discussion amongst the participants. It brought out the range of perspectives on RCT’s and the ways in which people’s opinion on them was based on their background and experience in the development sector. A few of the questions and issues raised during the discussion have been listed below and we invite you all to take this discussion further:
The presentation made at the session has been attached with this post.
Tags:
This seems to be an interesting session. My question is with regard to the definition of 'impact' and the assumption that all differences observed in the randomly selected villages with the program in place and those without the program are to be assumed as relating to the program.
Are all the villages included have everything else equal? How about education, gender ratio, per-capita income, governance etc.
Attending this reflection session was useful. For a person like me who wants to understand its relevance to the work one does this was basic and sufficient. J-Pal Delhi team did a good job of finally educating me in most simple manner. I learned that there are evaluation of programmes which resulted in decision of not to scale up as a desired impact was not seen. I need to learn more on such evaluations. Diva Dhar's advice on scope of RCT in the work I have in hand is very useful. Now what I need to explore about evaluation that has informed any policy and how the scaling up was decided in the context of India. I would also need to explore the not so resource intensive evaluation which helped in deciding what works and what does not? Which is an effective and cost effective programme? My understanding on evaluation and its implication on policy making at the level of government or at organisation needs to be sharpen. I also need to be aware if gender is part of my thinking and hence working or not? What is the connection between being child rights activists and having insight on gender?
I thank ISST for organising such session which promotes learning and sharing!
Warmly,
Krinna Shah
This is a great question! One of the main advantages of the RCT methodology is ensuring that both the treatment and control groups are similar to the population at whole, as well as each other. With randomization, we can ensure that the villages are not statistically different on factors such as education, per-capita income, etc. After researchers randomly select the treatment and control groups, they perform baseline data collection to ensure that these two groups are indeed similar on key indicators.
The J-PAL website provides a detailed explanation randomization:
http://www.povertyactionlab.org/methodology/what-randomization
http://www.povertyactionlab.org/methodology/why/why-randomize
Please let us know if you would like any further clarification. Thanks so much for contributing to the RCT and Gender discussion!
Best,
Urmy Shukla
CLEAR/J-PAL South Asia
Urvashi Soni-Sinha said:
This seems to be an interesting session. My question is with regard to the definition of 'impact' and the assumption that all differences observed in the randomly selected villages with the program in place and those without the program are to be assumed as relating to the program.
Are all the villages included have everything else equal? How about education, gender ratio, per-capita income, governance etc.
Thanks Urmy, for your response!
I am also very curious to know how one can introduce a gender lens in evaluation using RCT's, when the programme being evaluated is not specifically for women's development and empowerment.
Would you be able to share some examples from previous evaluations?
Thanks,
Akanksha
Thanks for your question, Akanksha.
There are many ways in which we can introduce a gender lens to RCTs that are not specifically focused on female empowerment. Many projects have gender-specific impacts, even if the project itself is not specifically focused on women. Key examples of this are in water and sanitation, health, and employment. While the project might not target women, it is possible to see female-specific outcomes.
The J-PAL website has a 'search' function that will allow you to browse summaries of previous evaluations: http://www.povertyactionlab.org/search/apachesolr_search?filters=ty...
Best,
Urmy
I have seen people using the word Impact in a very loose manner. Focus on RCT emphasises the need for measurement using scientific designs to know the impact of intervention. RCT being a rigorous quantitative exercise, it is important to identify gender related variables to measure even before any baseline is initiated. Therefore, we need think ahead how the intervention would impact gender and would we measure those indicators both at baseline and end-line evaluations.
© 2024 Created by Rituu B Nanda. Powered by