Randomized Control Trials through a Gender lens

[Akanksha Babbar]

On 19^th July 2013, the Institute of Social Studies Trust (ISST), hosted a reflection session on ‘Randomized Control Trials and Gender’, at the National Foundation of India, India Habitat Centre.  The session was organized with the primary aim of building an informal network of students and professionals, either working in or interested in the areas of evaluation and gender. Being first in the series of reflection sessions that have been planned under the project ‘Engendering Policy through Evaluation’ (funded by IDRC, Canada and the Ford Foundation), the session aimed at enhancing the participants’ knowledge and understanding of RCT’s while giving them the space to reflect upon the ways in which their implementation is effected by and in turn impacts gender relations.

Diva Dhar, Assistant Director of Capacity Building for CLEAR/J-PAL South Asia at IFMR facilitated the session by presenting J PAL’s work on RCT’s. The session was attended by participants from a variety of backgrounds- students interested in learning about the application of RCT’s for evaluation and research and mid-level and senior professionals from social development consultancies and non-profit organizations. The names of the participants have been mentioned at the end of the report.

After giving a brief introduction of J PAL’s mission to maintain a strong interdependence between evaluations, capacity building and policy outreach so that policy remains driven by evidence and research gets translated into action, Diva moved on to discuss several conceptual and related issues associated with RCT’s. These have been summarized below:

Defining ‘Impact’

Impact was defined as a comparison between the ‘outcome sometime after the program has been introduced’ and the ‘outcome at that same point in time had the program not been introduced’. This was explained as the best measure of assessing impact.

The ‘Counterfactual’

Defining ‘Impact’ in this way leads one to the ‘Counterfactual’, which ‘represents the state of the world that program participants would have experienced in the absence of the program ( i.e, had they not participated in the program)’.

This difference is often constructed by selecting a group not affected by the program. Thus, in randomized evaluations, individuals, villages or districts are randomly selected to receive the treatment, while other villages serve as a comparison. In this way, because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, any difference that subsequently arises between them can be attributed to the program rather than to other factors.

Advantages of RCT’s

• Conclusions drawn from RCT’s can be used as ‘hard evidence’ of what works and the evidence can then be used to attract further funds and make decisions regarding expansion and scaling up.
• RCT’s prove to be helpful when budgets are limited; the program is in its pilot stage, and being phased in overtime.
• Lastly, randomization has often been considered the fairest way to select who receives the benefits of the program.

When not to do Randomized Evaluations

•  When the program is at a premature stage or on too small a scale to randomize into two “representative groups”
• If positive impact has been proven using rigorous methodology and resources are sufficient to cover everyone
• If the program has already begun and there are no plans of expanding further

J PAL evaluations in the gender space

Towards the end, Diva highlighted a few of J PAL evaluations in the gender space that employed RCT’s. One important example was that of J PAL’s evaluation of a school based intervention by Breakthrough. This intervention was based in Haryana and involved teacher training and capacity building for mobilization and education of youth on gender inequality, empowerment of girls, and SSE. It used innovative means of communication like audio-visual , digital media and online forums for youth discussion.

Some examples of the research questions through which J PAL sought to understand the programme’s impact on gender attitudes are as following:

• What is the difference in the programme’s impact on male and female students?
• Does the intervention differentially affect students whose parents have less progressive gender attitudes or are more educated?
• Does the intervention succeed in reducing support for sex-selective elimination among the participating students in the long run?

The indicators of change were also designed in a gender responsive manner;

• Long term: fertility preferences and outcomes
• Girls mobility, decision-making, time use
• Awareness of gender issues
• Parents’ gender perception/aspirations

J PAL’s evaluation carried out RCT’s among 300 schools out of which 150 were Treatment schools ( that experienced Breakthrough’s intervention) and 150 were Control schools( that did not receive any intervention). Baseline, midline and endline surveys used quantitative and qualitative instruments like surveys, structured interviews, Implicit Association Test and Focus Group Discussions.

Diva then concluded by bringing out the ways in which the results drawn from the evaluation were used by Breakthrough;

• To inform policy/program creation and expansion
• Augment the support from external stakeholders like government bodies ,donors and other civil society organizations
• And contribute to an evidence base of what works and what is cost effective

Discussion

The presentation by J PAL team stimulated an interesting as well as intense discussion amongst the participants. It brought out the range of perspectives on RCT’s and the ways in which people’s opinion on them was based on their background and experience in the development sector. A few of the questions and issues raised during the discussion have been listed below and we invite you all to take this discussion further:

• The question of ethics- randomly selecting population to give treatment means that those who did not receive it, might question the basis of allocating and distributing the benefits of the program. How does one deal with these situations?
• What are the ways in which a ‘gender lens’ can be introduced in RCT’s, when the evaluation is not specifically of a programme aiming at women’s development?
• To what extent and how are the indicators of ‘success’, ‘progress’ defined in RCT’s and evaluations, in general ?
• Since carrying out RCT’s is an investment itself, how does one allocate funds between programme implementation and expansion and evaluation, in case of funding constraints? How cost effective are RCT’s?

 The presentation made at the session has been attached with this post.

[Urvashi Soni-Sinha]

This seems to be an interesting session. My question is with regard to the definition of 'impact' and the assumption that all differences observed in the randomly selected villages with the program in place and those without the program are to be assumed as relating to the program.

 

Are all the villages included have everything else equal? How about education, gender ratio, per-capita income, governance etc.

[Krinna Shah]

Attending this reflection session was useful. For  a person like me who wants to understand its relevance to the work one does this was basic and sufficient. J-Pal Delhi team  did a good job of finally educating me in most simple manner. I learned that there are evaluation of  programmes which resulted in decision of not to scale up  as a desired impact was not seen. I need to learn more on such evaluations. Diva Dhar's advice on scope of RCT in the work I have in hand is very useful. Now what I need to explore about evaluation that has informed any policy and how the scaling up was decided in the context of India. I would also need to explore the not so resource intensive evaluation which helped in deciding what works and what does not? Which is an  effective and cost effective programme? My understanding on evaluation and its implication on policy making at the level of government or at organisation needs to be sharpen. I also need to be aware  if gender is part of my thinking and hence working or not? What is the connection between being child rights activists and having insight on gender?

I thank ISST for organising such session which promotes learning and sharing! 

Warmly,

Krinna Shah 

[3d79ktyspfofn]

This is a great question! One of the main advantages of the RCT methodology is ensuring that both the treatment and control groups are similar to the population at whole, as well as each other.  With randomization, we can ensure that the villages are not statistically different on factors such as education, per-capita income, etc.  After researchers randomly select the treatment and control groups, they perform baseline data collection to ensure that these two groups are indeed similar on key indicators.   

The J-PAL website provides a detailed explanation randomization: 

http://www.povertyactionlab.org/methodology/what-randomization

http://www.povertyactionlab.org/methodology/why/why-randomize

Please let us know if you would like any further clarification.  Thanks so much for contributing to the RCT and Gender discussion! 

Best,

Urmy Shukla

CLEAR/J-PAL South Asia 

Urvashi Soni-Sinha said:

This seems to be an interesting session. My question is with regard to the definition of 'impact' and the assumption that all differences observed in the randomly selected villages with the program in place and those without the program are to be assumed as relating to the program.

 

Are all the villages included have everything else equal? How about education, gender ratio, per-capita income, governance etc.

[Akanksha Babbar]

Thanks Urmy, for your response!

I am also very curious to know how one can introduce a gender lens in evaluation using RCT's, when the programme being evaluated is not specifically for women's development and empowerment.

Would you be able to share some examples from previous evaluations?

Thanks,

Akanksha

[3d79ktyspfofn]

Thanks for your question, Akanksha. 

There are many ways in which we can introduce a gender lens to RCTs that are not specifically focused on female empowerment.  Many projects have gender-specific impacts, even if the project itself is not specifically focused on women.  Key examples of this are in water and sanitation, health, and employment.  While the project might not target women, it is possible to see female-specific outcomes.  

The J-PAL website has a 'search' function that will allow you to browse summaries of previous evaluations: http://www.povertyactionlab.org/search/apachesolr_search?filters=ty... 

Best, 

Urmy 

[Shankar Talwar]

I have seen people using the word Impact in a very loose manner. Focus on RCT emphasises the need for measurement using scientific designs to know the impact of intervention. RCT being a rigorous quantitative exercise, it is important to identify gender related variables to measure even before any baseline is initiated. Therefore, we need think ahead how the intervention would impact gender and would we measure those indicators both at baseline and end-line evaluations.