Impact Assessment without a baseline

[Priya Anand]

Hi 

Am planning to do an impact assessment that will assess the impact of a teaching tool used by teachers Unfortunately a baseline wasn't done and its over three years now.  Was wondering if comparing to a control group of teachers who haven't used the tool would work. Are there any other methods that can be utilized to assess impact without bias.

Thanks

Priya Anand

[049poui46alay]

The most simplistic and authentic method of doing an impact evaluation with one point survey is by using cross-sectional methods. You have to have a rigor of sample selection to ensure that the sample who have exposed are similar in certain key characteristics with those who have not been exposed to teaching tools. You can use propensity score matching (PSM) to select individuals.

I hope this is useful. 

[SHANTHI PERIASAMY]

I guess there are two things here:

1. If you had the baseline data, you could speak about the difference this particular tool has made (if data provides this) to the respondents who may have a special interest and engagement in areas that warrant the tool. You could then go on to elaborate the impact of the tool.
2. Now that you do not have the baseline data, you can only describe the impact based on a comparison of achievement between the group that used the tool and the control group.  It may not be able to tap on in-depth details and elaborate data but however may still be able to reveal a substantial justification on the use of the tool and its impact.

My two cents!!

[3eojs5yj0d18m]

Dear Priya,

Capturing impact on student learning through a baseline and endline would have been ideal.

1. In the absence of a baseline you could attempt a Participatory Impact Assessment among the teachers and the students.

2. Technically, putting together a group of teachers who have not used the teaching tool does not qualify as a Control Group but if you craft your areas of enquiry for FGDs very carefully you will be able to draw out what you need from this group of teachers and compare it with your intervention teachers.

Regards

Manju

[Afsana Wahab]

Comparing the assessment findings to the base line would be perfect but life isn't perfect!. Development Partners accept this fact.

In the absence of a baseline I use a control group. In the 'limitations' section of the report I mention that due to the lack of a baseline I have used a control group and that this should be used as an illustration only. 

Afsana

[madhumita sarkar]

Hi! Priya,

you have received some very interesting responses regarding the use of control group. i would concur with Dr. manju that just a control group would not necessarily give you the right results.

i do think it is important you look at what were the objectives of your tool. accordingly design qualitative and quantitative tools for assessing the impact. use this tool to make inquiries with the control group too and compare the results.

would look forward to knowing how the assessment went,

madhumita

 

[Priya Anand]

Thanx to both Madhumitas, Shanti, Dr Manju, Afsana for your valuable inputs. I am also of the opinion that comparisons to a control group may not be effective in the absence of a baseline. I do plan to use participatory techniques but feel that quantitative data if gathered and analysed properly, will highlight impact better 

[Isha Wedasinghe Miranda]

Dear Priya,

This is a very common to most in present days. All above advises are correct. you can used Participatory Impact Assessment tool, Control and treatment tools etc.

 Also you can do this methodology.It is very orthodox. What is important,  that you to do somewhat desk reading, or get some KII informations on history of the either community , areas, or particular segment of the community where this program has been done. and do the In-depth analysis with  program data with present changes. you able to meet past teachers and students, parents of past and present students , school officials of past and presents, informations on similar programs  and your common sense and experiences in to it. 

It is very Orthodox. Sometimes it works better than all above tools. 

 

[ESTEBAN TAPELLA]

Dear Priya Anand, try to get and read the book Real World Evaluation http://www.realworldevaluation.org/

There you will have different alternatives to assess impact when not having a base line. 

Best, Esteban (from Argentina)

[39bhdmhxyix1w]

Do you have access to student data?  In a lot of cases, schools records tons of data about students anyway, regardless of existing projects, so maybe you can look at general student indicators from three years ago that the school may still have available, and compare them with student indicators now.  You can also reconstruct a baseline by asking teachers to think back to the time period before the project.  

A control group is good, but it doesn't really solve your time problem. It just tells you whether there's a difference in your groups now.  They difference could have nothing to do with your intervention. 

[Will Allen]

Hi Priya,

There is a good paper on how to reconstruct baseline data for impact evaluation and results measurement up at  http://siteresources.worldbank.org/INTPOVERTY/Resources/335642-1276...  

This is written by Michael Bamberger, the lead author of the Real World Evaluation book - http://www.realworldevaluation.org/ - that Esteban recommended.

kind regards

Will

[Fanaye Gebrehiwot Feleke]

In the food security through women's empowerment project I am working with we have a psycho-social well-being aspect and even though we had a base line it was done as part of another bigger project. We are now preparing to do an end-line and we wanted to measure the impact of the psycho-social interventions and we are using both comparing with groups who did not receive the psycho-social well-being intervention as well as making comparisons among the different groups depending on when they started to receive the intervention, to check if involvement with the psycho-social interventions had different impacts. Another innovative approach that has been proposed to be used disregarding the baseline has been to use a regression analysis controlling for some elements. This might be even more useful in your case.

[Priya Anand]

 Thanx. I particularly like the idea of making comparisons between different groups based on when the intervention started.