Tools for impact assessment of training and awareness interventions for citizens

[Tina Mathur]

Dear friends,

In partnership with UNDP India, the Department of Justice, Government of India is implementing the project ‘Access to Justice to Marginalized Communities’. The project focuses on strengthening access to justice for the poor by developing strategies that address barriers to accessing justice in legal, social, economic and political domains.

One of the core components of the project is the development of legal literacy content in the form of training material/manuals and mainstreaming legal literacy through the National Literacy Mission Authority programmes, Common Service Centres (CSC), National Legal Services Authority (NALSA), State Legal Services Authorities (SALSA) and civil society organisations.    

 A set of actors such as para legal workers, empaneled lawyers with legal services authorities, law students and community members in select districts of Orissa, Madhya Pradesh, Maharashtra, Jharkhand in India have been trained. These groups are expected to reach out to the marginalized communities to make them aware of their legal rights and entitlements, provide free legal aid and empower them to seek speedy, affordable and quality justice.  ICT tools such as information kiosks have been used at 50 locations in district legal services authorities towards this objective.   

The project now intends to assess the impact of these training programmes on the increased awareness of these marginalized communities- whether there is now a change in their lives as a result of legal awareness training, are they able to better access their entitlements? 

In this context, we seek inputs from members on:

What examples can you share of simple tools to get feedback/assessment from communities/citizens on the following aspects?

• Changes perceived as a result of awareness training provided to them- in this case legal awareness training and training on their rights and entitlements. The examples of tools could be from other types of community awareness trainings that may work in this case.
•  Changes perceived by citizens in behavior/ citizen centricity of frontline government functionaries as a result of their trainings on behavior change.

This query has been posted on the Decentralization Community, Solution Exchange of the United Nations in India. Your responses will be gratefully acknowledged and shared back with the project team and the Decentralization Community.

Thank you and regards, Tina

[Ian C Davies]

Hi Tina,

Most Significant Change would appear to be well suited to this type of assessment http://betterevaluation.org/resources/guides/most_significant_change 

I also wish to take this opportunity to suggest you drop the word "impact" from your assessment objective or define the term clearly. Impact has become a loaded term which has become to all extent and purposes meaningless, although it is popular and seems to generate support from donors (read funding). The word also conveys a one-off conception of "success", e.g. we've implemented the program, people have responded positively, we can now go home and claim success.

The reach challenge is sustainability, as in sustainable processes. "Results", including "impact", tend to be one-offs. Assessing those tells you little about the reality of the processes, including, and I would argue more importantly, those not related to "intended results". So I encourage you to focus your assessment on processes, those that matter most to the individuals who are affected by them. 

Cheers,

Ians

[Vedhna Lalla]

That is helpful and I agree. May I ask how what would you consider changing "impact" to?

Vedhna

[Ian C Davies]

Hi Vedhna,

I would change it to "value".

Cheers,

Ian

[Tina Mathur]

Hi Ian, thanks for your suggestion on focusing the assessment on processes and for sharing the evaluation guidelines.

[Rituu B Nanda]

Hi Tina,

I would suggest two processes if you want to measure perception and behaviour change.

One is using Constellation's self assessment framework. Here the trainees can define their dream for the issue on which they were trained and where are they in terms of the response to the dream after having attended the training. As it is self assessment the participants will be more honest. This process also creates a space where participants can share openly, excellent for capturing from gender and equity lens. Here is the link http://communitylifecompetence.org/en/pages/8-community-life-compet...

Second process I will suggest is in which all primary stakeholders can engage in  is participatory action research.

Both the above processes have sytematic reflection and action learning cycle. I am happy to respond to further queries if any.

Thanks

Rituu

Constellation/ISST

[Fanaye Gebrehiwot Feleke]

Hi Tina,

How about using the community score cards, it is a well used tool in India.

Selam,

Fanaye

[Tina Mathur]

Thank you Fanaye, yes we have some good examples of community score cards and will use that, we are looking at more innovative ways also.

regards,

Tina

[Rukmini Panda]

Dear Tina

Thanks for your posting. Based on my experience of evaluation of a project on violence against women, I have some suggestions which you may take into consideration.

• what were the perceived changes. Using those indicators, a semi-structured questionnaire may be administered with the people with whom legal awareness training had been done. In that case sampling of the population plays a vital role.The semi structured questionnaire may also be administered with the PLVs who have linked the people to justice system.
• KII may be undertaken with the government functionaries having both quantitative and qualitative questions. To capture their behavioural change, they may be asked to share stories of their intervention with the people on desired issues after training has been imparted to them.
• Case studies of men and women who have used the legal structures and Laws after training.
• Empowerment matrix may be a powerful qualitative tool to capture changes. It captures changes in behaviours such as:
• Whether they have been able to go to different government offices
• Whether they ask questions in the meetings.
• Whether they are able to bring the application and deposit them
• Whether they encourages others to use the law
• Whether they have use the Law without help of others

The questions may be so many as per your need. To each question, the respondents would be asked to give mark on scale of 1 to 10. How do they evaluate themselves. Based on the score, the analysis may be done.

Thanks & Regards

Rukmini

[Tina Mathur]

Dear Rukmini,

Thanks for your suggestions. Can you also suggest some resources/ reports that we can refer to where the KII and Empowerment Matrix have been used?

Regards,

Tina

[Rukmini Panda]

Dear Tina

It was a Oxfam project's in-house evaluation,which was done by Praxis and I was a part of it. But I do not know whether it is available in public domain being their in-house evaluation. But I may help you in designing these tools.

Regards,

Rukmini 

[Theo Nabben]

dear Tina 

Most Significant Change is a useful story based tool to get information on changes and the perspectives about what was important from eh respondents perspective (which can be different from the program staff etc) - I you wish further information email me at [e-mail] or contact me via this group. 

Skills needed - basically how to ask questions and listen  and record. A qualitative approach vs quantitative mindset.This will work in both cases. 

For community use above - also Ten Seed technique perhaps- depends. Again contact time if you wish to explore further.  Kind regards Theo