Cultural Competence and Language in Evaluation: Share your Experience!

[Michele Tarsilla]

Dear Gender and Evaluation Colleagues,

I hope all is well. A few days ago I wrote you mentioning that I am currently conducting a research on the use of language in evaluation and I asked you to kindly share some valuable resources on this topic. The response has been pretty good so far and I am in the process of compiling a list of the recommended resources. Today, I would like to follow up and hear more about your own experience and thoughts on this subject.  In particular, I have three quick questions dealing not so much with your use of technical evaluation language (e.g., I won’t ask you about the definitions of output, outcomes or impact) but rather with your use of language that has either strengthened or weakened your interactions with partners and clients in the course of your evaluation work in the past. 

THREE QUICK QUESTIONS (please write your responses under each question and e-mail them back by Tuesday September 16. You could either reply to all or e-mail your responses to me at [e-mail]. I will send a summary of all your contributions at the end of my analysis)

 1) What are words or expression that you used in the past but that you no longer use since you or your partners/clients perceived them to be inadequate or offensive? (e.g., in terms of cultural competence and equity)?

 2) What are words or expressions that that you have started using more often in the course of your evaluation assignments over the last few years since they appear to be particularly well received by your evaluation partners and/or clients?

3) How could you explain other evaluation colleagues how to promote a more respectful use of language in evaluation (e.g., equity-focused and culturally competent language) in their profession? How could you “detect” that some expressions are no relevant to the success of an evaluation? Also, what could you do to promote the use of more culturally competent language within the evaluation community?

Thank you so much for your time and interest.

 Best,

 Michele

 

Michele Tarsilla, Ph.D.

Evaluation Capacity Development Group, Vice-President

Chair of the International and Cross-Cultural Evaluation TIG at AEA

[Michele Tarsilla]

Dear Gendereval Colleagues,

I hope all is well. As promised two weeks ago, I am sharing with all of you a quick update on the preliminary findings of my current research on the use of language in evaluation. I am glad to see that my research was very well received and what I would like to do today is to share with you five of the words or expressions that seem to be the most contested in our profession. I hopethat, by reading the list of words listed below, you will be encouraged to share your own experience and perspectives on either these very same or different words. As a reminder, my research
aims to identify those words and expressions used in our field that have either greatly hindered or promoted sound interactions between evaluators and all the different individuals, partners and clients involved whom the earlier interact with in the course of their evaluation endeavours.

Below is a small sample of terms that I purposely selected to give you a flavour of my research preliminary findings:

1. According to some respondents, “Gender-sensitive” is a vague term that almost conveys the
idea of gender being a “soft” and not an essential program feature that deserves closer attention. In order to address this limitation, some respondents suggested the use of alternative expressions as “gender intelligent”, “gender responsive”, “gender fairness”, “gender optimized” (on a related note, please read the interesting blog on the difference between “equality” and “equity” that was posted on this website a few days ago: http://gendereval.ning.com/?xg_source=msg_mes_network);

2. Another contested term is “beneficiary” along with all other related terms (e.g. beneficiary analysis). In particular, this term seems to perpetuate the old logic of “aid” whereby individuals and communities in the field are merely passive recipients of “help” and assistance delivered by foreign experts. Also, the concept of beneficiary implies that whoever is affected by a program or project is automatically benefiting from it;

3. “Institutional Change” and “change agents” are also criticized terms for being associated with rebels’ attempts to overthrow current governments in a variety of countries. Hence, such words as “institutional strengthening” or “capacity building” are increasingly being used as they appear to be less threatening;

4. The limitations of close-ended questions on race/identity were also criticized for not providing respondents with the freedom to define their gender and identity the way that best applies to their specific case and individual perception;

5. Target (cible en francais) is a term that also makes people cringe quite a bit. In particular, the term seems to convey the idea that a project/program is a unilateral intervention (a sort of shooting gun) where dialogue and participation are dismissed and where fairness and equality of status among all the different parties affected (more or less directly) by a program/project evaluation are not guaranteed.

Overall, the conversation I had these past few weeks on the thirty terms that I have collected so far, suggests the two trends are driving the use of language in evaluation. First, the tendency to reify constructs (this is the case of evaluation capacity development or evaluation capacity building often associated with training activities but not with clearly articulated goals). Second, the use of vague and not easily operationalized/operationalizable (and therefore measurable) concepts (such as, all the evaluation questions and criteria containing the terms sustainability and empowerment).

FYI, I will be presenting some of the preliminary findings of my study at the upcoming European and American Evaluation Conferences. It would be great if I could incorporate your own comments on these or some other contested words and expressions within a week or two. However, I am planning to keep working on this for quite a while. Therefore, I look forward to exchanging with all of you before and after the envisaged deadline. Please remember that you could either respond publicly or send me your contribution in private (in which case, I will guarantee the anonymity of your responses).

Below is also a list of relevant resources on the use of language in evaluation among those shared by some of the respondents so far:

http://betterevaluation.org/resources/guide/sourcebook_results_base...

https://www.climate-eval.org/blog/whats-name-indicators-measures-an...

I look forward to receiving your contributions this week,

Michele

Michele Tarsilla, Ph.D.
Independent Evaluation Capacity Development Specialist
International and Cross-Cultural Evaluation TIG at AEA, Chair
YouTube Video: http://www.youtube.com/watch?v=ZsY0OV0Elfs

[Michele Tarsilla]

Dear Minal,

Thank you very much for your contribution. You raised a very important topic indeed, which I have been spending some time reflecting on over the last few years. That is, finding a creative way to introduce the term "evaluation" to clients and/or partners who might be either unfamiliar with it or even terrified by it. This ability to convey messages on the rationale and uses of evaluation beyond the traditional definition, is what I refer to in my Evaluation Capacity Development (ECD) workshops as "ECD opportunism": evaluation practitioners need to be able to adapt the traditional definitions of evaluation to their clients/partners' background/interest in a way that would make sense to the latter (on a related note, I will refrain from using the term "stakeholders" here, as this is one of those terms that many of us are increasingly having a hard time with because of its generic and vague use in many of the today's evaluation proposals and reports). Overall, your idea of evaluation as a tool to celebrate an organization's work is quite effective and I am very glad that, in promoting this idea, you did not only concentrate on the "good" but also on both what went wrong with the program/project being evaluated AND what happened unexpectedly during its implementation. In my opinion, only looking at the positives and the negatives/unexpected you could make a fully evaluative judgment and "celebrate" this time the value added by evaluation (as opposed to monitoring or performance management).

As far as your suggestion on the attitude that practitioners should have in order to better promote cultural competence in their own work and that of others, I agree with you: we are not perfect but we are resourceful and have the skills to improve our own practice day by day. That also allows to clarify that the whole purpose of my research is not to become a sort of "language police" whose objective is to ban language and "punish" current practices but rather to provide an opportunity to those among us who are interested in this topic to think about their own use of language in evaluation more deeply with the possibility (I will leave that to each of us) of modifying or not our own current vocabulary in evaluation. The ultimate goal of this effort is to enhance the cultural competence of what we do by creating a more truthful and constructive rapport with all the parties whom we engage with in the course of our evaluation work. 

Once again, thank you very much for sharing your thoughts and experience. I look forward to hearing from other members of our community on their use of language in evaluation, as well.

[Minal Mehta]

pleasure  and thank you for giving food for thinking.

[Minal Mehta]

 1) What are words or expression that you used in the past but that you no longer use since you or your partners/clients perceived them to be inadequate or offensive? (e.g., in terms of cultural competence and equity)?

Earlier I used  word evaluation  and clients would find it threatening and fearful to share. I shifted to using  detail explanation of evaluation as celebration of their work and to learn what worked so others could learn and what did not work so the same  methods, approaches are not repeated elsewhere instead of saying mistakes are not repeated.

 2) What are words or expressions that that you have started using more often in the course of your evaluation assignments over the last few years since they appear to be particularly well received by your evaluation partners and/or clients?

As mentioned above.

3) How could you explain other evaluation colleagues how to promote a more respectful use of language in evaluation (e.g., equity-focused and culturally competent language) in their profession? How could you “detect” that some expressions are no relevant to the success of an evaluation? Also, what could you do to promote the use of more culturally competent language within the evaluation community?

we cannot act good  all the time so we must grow over a period of time by reflecting and learning from others and seeing the results of  our own change and then repeating those changes or innovating new as and when required.

[Michele Tarsilla]

Hi Kylie,

Thank you very much for your contribution to the ongoing discussion. I actually meant to write you later this week as I was already well aware that you share my same interest in this topic. Back to your comment, what you wrote about the term "stakeholder" is something that I could easily relate to as it is one of those terms that I have been trying to use less and less in my own professional practice. While some might rightly differ among "primary", "secondary" and "tertiary" stakeholders, I have found that the use -or rather abuse- of this term in evaluation proposals and other evaluation reports has turned "stakeholder" into a catch-all expression that has discouraged many practitioners from focusing on what that really meant at design, implementation and evaluation phase. Some other colleagues, too, mentioned that this is a term that they are struggling with and suggested that replacing the term "stakeholder" with a more precise list of the different groups affected and/or with a vested interest in the evaluation at stake is a more effective strategy. Put simply, some of our colleagues are suggesting that, thanks to this "retro" approach (yes, it feels like going to the basics of our language and professional practice with no frills or technical jargon to play with), you would benefit from doing the following:

If you were evaluating a school program, you would precisely talk about teachers, students, headmasters and so on - rather than some more vaguely defined "stakeholders".  What do you all think about this? How viable is this suggestion?

With respect to the other question on more "adequate" or "culturally competent" expressions or words that you are using now, I would be really interested in learning more about what are some other words/expressions whose use you have started appreciating, mainly as a reflection of your partners'/clients' satisfaction/comfort with them. As already mentioned, I am planning to share the first batch of my research emerging findings on this site as well as at AEA and EES. I see myself engaged in this effort for quite some time, though. Therefore, I am confident that I will be engaging in productive exchanges on language with you and others within our community for many more months to come. Once again, thanks so much to you and all the others for your great ideas: I really look forward to the continuation of our discussion!

[KylieHutchinson]

 1) What are words or expression that you used in the past but that you no longer use since you or your partners/clients perceived them to be inadequate or offensive? (e.g., in terms of cultural competence and equity)?

I remember years ago catching a lift from a conference with a local community activist who lectured me at great length about my use of the term "stakeholder".  According to him, he felt it negatively symbolized a group of people desperately grasping onto a overtly phallic symbol which was a grossly inappropriate representation of community members.  Right or wrongly, I'm afraid I still use the term today.  :-)

The other questions you have put to us are fascinating so I do plan to respond, but I want a bit more time to fully consider them.  

Thanks for starting this discussion!

P. S.  I assume you know about the informal Evaluation Typology map I curate (https://bubbl.us/?h=b9bc4/1a7bc4/769hviFngrfnI&r=1531766846).  If you have any suggestions for it out of this research let me know.

 

[Michele Tarsilla]

Dear All, 

I have already received some quite interesting feedback on the use of language in evaluation from our community of practice members these past few days. Yet, I would encourage more members to take some time to share their thoughts and experiences on this topic. In particular, I hope that some more colleagues among the seventy who have viewed the blog this week will be able to let me hear back from them. The results of this research on the use of language of evaluation (what words or expressions are perceived to be more or less appropriate to use in contemporary evaluation practice) will be widely disseminated. To this end, my intention is to collect as many perspectives on this topic as possible and provide a humble platform for some more productive discussion on equity and cultural competence in evaluation in the near future. Thanks so much for espousing the cause and many thanks to those among you who have expressed appreciation for my current effort!

[Michele Tarsilla]

Hi Joanne, 

Thank you very much for your prompt response. I really appreciate your contribution to the ongoing discussion. An an evaluation capacity development specialist, I have often been creative when it comes to using acronyms in the course of workshops and mentoring sessions. Distinguishing between Monitoring and Evaluation is certainly needed but I would suggest to go even further: that is, to first build upon the language and terminology that your partners and colleagues are already familiar with (e.g., vocabulary borrowed from the Result-Based Management arena) and then to introduce the terms of Monitoring and Evaluation once their respective meanings have been understood by the groups or individuals whom one works with. This approach is what I have call the "normalization of the evaluation function" (see my recent piece on the OECD/DAC Newsletter: http://www.oecd.org/dac/evaluation/ecdnewsletter.htm#Rbmization)

As mentioned in my today's blog, I am particularly interested in words and expressions that are not necessarily technical but that are still widely used in the course of evaluations but have often lost their meaning because of their "jargonesque" and unintentional use (e.g.,I am thinking of such words as accountability, aid, empowerment; or expressions such as "I am here to build your capacity" ).

I encourage you and the rest of your colleagues to think about some of such words and expressions AND share your thoughts and reflections this week. Once again, thank you very much for your time and interest!

[Joanne Roberts]

Hi Michele

1. I am trying hard to not use the term M & E with anyone and endeavour to instead say Monitoring and evaluation in full. In the world I work in these two terms have become conflated and in the process lose their relevance.

2. Depending on the audience I will say Performance Framework rather than Program Logic. This term works better with senior level bureaucrats.

3. This is not really an answer to 3. However, I do a lot of work in PNG where there are in the order of 800 different indigenous languages. Recently a group of Papua New Guinean researchers told me that there is no equivalent word for gender in any of those languages, it is indeed an external construct. So the researchers have to talk around the concept of gender to undertake evaluation research in the gender space.

  Good luck with your research